We are seeing disk quota exceeded errors on ppc64 builds for the past few days.
Cannot mkdir: Disk quota exceeded
The build has been restarted multiple times, but is failing with the same error.
The failed build log is available here
2 Likes
This is severely affecting Facebook RocksDB builds also, seeing âdisk quota exceededâ
https://travis-ci.org/github/facebook/rocksdb/jobs/671496406
https://travis-ci.org/github/facebook/rocksdb/jobs/671424293
Or âcollect2: error: ld returned 1 exit statusâ with no prior error message (likely due to disk space https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=38222)
https://travis-ci.org/github/facebook/rocksdb/jobs/671926305
https://travis-ci.org/github/facebook/rocksdb/jobs/671460860
Or 100x slowdown (or more) in some of our tests (https://github.com/facebook/rocksdb/issues/6653)
https://travis-ci.org/github/facebook/rocksdb/builds/672176972
[==========] 92 tests from 5 test cases ran. (2005962 ms total)
vs. baseline two weeks ago
https://travis-ci.org/github/facebook/rocksdb/jobs/664565410
[==========] 92 tests from 5 test cases ran. (12911 ms total)
155x slower on the same code
Our build history has dozens and dozens of these PPC failures.
Iâm seeing the same problem with travis-ci.org[0] builds. the same commit builds ok on ppc64le on travis-ci.com[1].
[0] https://travis-ci.org/github/PrincetonUniversity/PsyNeuLink/jobs/677014825
[1] https://travis-ci.com/github/jvesely/PsyNeuLink/jobs/320959233
I am also seeing this, with https://travis-ci.org/github/gap-system/gap â I canât link to a concrete log right now, as I restarted the job. Because so far, these failures (all indeed on .org) were transient: the builds usually work, and if they fail, restarting them fixes the issue.
But thatâs of course not a good modus operandi moving forward.
This problem is also occured on ARM64, I am trying to build and test Storm on ARM platform with Travis infrastructure, it is also easy to raise âDisk quota exceededâ error, here is an example:
https://travis-ci.com/github/liusheng/storm/jobs/323797338#L2409
On mariadb and iccurring very early in the build at a cmake configure stage
git-depth 2 to reduce size. Submodule (including rocksdb) probably donât have git-depth 2 applied which would save more space.
4.5G/5G of ccache explictly set.
https://travis-ci.org/github/MariaDB/server/jobs/680933484#L1710
Anyone found a way to mitigate? Is it a timing thing? What is the actually quota limit?
The docs mention that arm64/ppc64le/s390x disk space is limited to ~18GB [0] compared to 40+GB on other platforms.
However, the error is intermittent and tends to happen on temporary files; (the mysql log above proceeds to build object files despite quota error in configure).
It doesnât happen in the exact same place (at least for my builds), and I also occasionally see No usable temporary directory found error.[1]
My guess would be that the error indicates running out of tmpfs space (OOM). so any workaround would be âhope there are no other memory hungry jobs on the same machineâ.
EDIT: I tried redirecting TMP and TMPDIR, but to no effect so the above is probably incorrect
[0] https://docs.travis-ci.com/user/reference/overview/
[1] https://travis-ci.org/github/PrincetonUniversity/PsyNeuLink/jobs/683672848
I have encountered this issue many times during on Arm64 building, but this issue has not occurred before (perhaps a month or two ago)
https://travis-ci.com/github/iecedge/cilium/jobs/330875210#L1900
Lots of similar reports on this (I tried to collect links to all here: S390x Adding APT Sources - No usable temporary directory found - /tmp ENOSPC).
I happens for my ppc64le build too and it does happen intermittently.
Hello everybody
There was a major refresh in August, can you check if disk quota issues still occur on your ppc64le/s390x builds?