Disk quota exceeded on ppc64le

We are seeing disk quota exceeded errors on ppc64 builds for the past few days.
Cannot mkdir: Disk quota exceeded

The build has been restarted multiple times, but is failing with the same error.
The failed build log is available here

3 Likes

This is severely affecting Facebook RocksDB builds also, seeing ‘disk quota exceeded’

https://travis-ci.org/github/facebook/rocksdb/jobs/671496406
https://travis-ci.org/github/facebook/rocksdb/jobs/671424293

Or ‘collect2: error: ld returned 1 exit status’ with no prior error message (likely due to disk space https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=38222)

https://travis-ci.org/github/facebook/rocksdb/jobs/671926305
https://travis-ci.org/github/facebook/rocksdb/jobs/671460860

Or 100x slowdown (or more) in some of our tests (https://github.com/facebook/rocksdb/issues/6653)

https://travis-ci.org/github/facebook/rocksdb/builds/672176972
[==========] 92 tests from 5 test cases ran. (2005962 ms total)

vs. baseline two weeks ago
https://travis-ci.org/github/facebook/rocksdb/jobs/664565410
[==========] 92 tests from 5 test cases ran. (12911 ms total)

155x slower on the same code

Our build history has dozens and dozens of these PPC failures.

I’m seeing the same problem with travis-ci.org[0] builds. the same commit builds ok on ppc64le on travis-ci.com[1].

[0] https://travis-ci.org/github/PrincetonUniversity/PsyNeuLink/jobs/677014825
[1] https://travis-ci.com/github/jvesely/PsyNeuLink/jobs/320959233

I am also seeing this, with https://travis-ci.org/github/gap-system/gap – I can’t link to a concrete log right now, as I restarted the job. Because so far, these failures (all indeed on .org) were transient: the builds usually work, and if they fail, restarting them fixes the issue.

But that’s of course not a good modus operandi moving forward.

This problem is also occured on ARM64, I am trying to build and test Storm on ARM platform with Travis infrastructure, it is also easy to raise “Disk quota exceeded” error, here is an example:
https://travis-ci.com/github/liusheng/storm/jobs/323797338#L2409

On mariadb and iccurring very early in the build at a cmake configure stage

git-depth 2 to reduce size. Submodule (including rocksdb) probably don’t have git-depth 2 applied which would save more space.

4.5G/5G of ccache explictly set.

https://travis-ci.org/github/MariaDB/server/jobs/680933484#L1710

Anyone found a way to mitigate? Is it a timing thing? What is the actually quota limit?

The docs mention that arm64/ppc64le/s390x disk space is limited to ~18GB [0] compared to 40+GB on other platforms.
However, the error is intermittent and tends to happen on temporary files; (the mysql log above proceeds to build object files despite quota error in configure).
It doesn’t happen in the exact same place (at least for my builds), and I also occasionally see No usable temporary directory found error.[1]

My guess would be that the error indicates running out of tmpfs space (OOM). so any workaround would be ‘hope there are no other memory hungry jobs on the same machine’.

EDIT: I tried redirecting TMP and TMPDIR, but to no effect so the above is probably incorrect

[0] https://docs.travis-ci.com/user/reference/overview/
[1] https://travis-ci.org/github/PrincetonUniversity/PsyNeuLink/jobs/683672848

I have encountered this issue many times during on Arm64 building, but this issue has not occurred before (perhaps a month or two ago)

https://travis-ci.com/github/iecedge/cilium/jobs/330875210#L1900

Lots of similar reports on this (I tried to collect links to all here: S390x Adding APT Sources - No usable temporary directory found - /tmp ENOSPC).

I happens for my ppc64le build too and it does happen intermittently.

Hello everybody

There was a major refresh in August, can you check if disk quota issues still occur on your ppc64le/s390x builds?

Imprint