No space left on device for System Z

Worker information:

hostname: 48be7aa6-189e-4dce-be1e-db8186466c1f@1241.lxd-s390x-travis-ci-production-1-worker5-com
version: ? ?
instance: travis-job-radareorg-radare2-256856420 a5448d3ad7ae521a8c2821af8b0f9c020dda03de8393b95fd6081ad948bac7a3 (via amqp)
startup: 2.519641297s

Build system information

Build language: generic
Build group: stable
Build dist: trusty
Build id: 136642533
Job id: 256856420
Runtime kernel version: 5.3.0-19-generic
travis-build version: 119feb2eb

See https://travis-ci.com/radareorg/radare2/jobs/256856420#L158 for the full log

ar: lib/libgdbr.a: No space left on device
make[2]: *** [lib/libgdbr.a] Error 1
cc1: fatal error: can’t open ‘/tmp/ccpypDQn.s’ for writing: No space left on device
compilation terminated.
Cannot create temporary file in /tmp/: No space left on device
make[2]: *** [src/packet.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [src/sigutil.o] Aborted (core dumped)
ar: creating libr_ar.a
kern/env.c:257:1: fatal error: closing dependency file kern/env.d: No space left on device
 }
 ^
compilation terminated.
cc1: fatal error: can’t open ‘/tmp/ccr6rkP5.s’ for writing: No space left on device
compilation terminated.
make[2]: *** [kern/env.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [kern/disk.o] Error 1
Cannot create temporary file in /tmp/: No space left on device
make[2]: *** [src/ptrace_wrap.o] Aborted (core dumped)
make[1]: *** [preall] Error 2
make: *** [all] Error 2
+exit 1

Hi @XVilka, glad to see that you’re using s390x.
I updated the storage pool.
Apologies for the inconvenience and please let us know if you do run into any issues.
Happy building!

2 Likes

Thank you for the prompt answer! I confirm, it was fixed.

I am seeing this issue again:

The disk space is low on s390x: https://travis-ci.org/MarcoFalke/bitcoin-core/jobs/616893865#L77

There is plenty on arm64 or amd64: https://travis-ci.org/MarcoFalke/bitcoin-core/jobs/616893862#L117 or https://travis-ci.org/MarcoFalke/bitcoin-core/jobs/616893866#L182

@Damian I am still seeing this error. See for example here:

https://travis-ci.org/bitcoin/bitcoin/jobs/625880747#L8181

which shows that only 9% disk was left (9GB):

https://travis-ci.org/bitcoin/bitcoin/jobs/625880747#L96

I presume this 9GB is shared among travis users running on that machine?

Hi @Marco
Do you still see “Error: Disk space is too low”?
in df please look at instance storage e.g “/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-bitcoin-bitcoin-625880747/rootfs”

1 Like

Hi @Damian

thanks for following up. I can confirm that the instance storage is large enough to run the full build. However, we are running the build inside of docker. Our ci scripts are designed to run (also) locally and for safety mark the build folder read-only. Modifications only happen in the docker container.

So I believe the issue here is that docker does not put its data files in the s390x travis instance storage. According to https://beta.docs.docker.com/engine/reference/commandline/dockerd/#miscellaneous-options this can be changed on startup of the docker daemon.

Sometimes it is not even possible to start docker, because

write /var/lib/docker/tmp/GetImageBlob036386674: no space left on device

https://travis-ci.org/bitcoin/bitcoin/jobs/641563767#L138

Yep, s390x is broken also on our LTP tests:
https://travis-ci.org/linux-test-project/ltp/jobs/642893307

Unpacking libllvm7:s390x (1:7.0.1-8) ...
dpkg: error processing archive /tmp/apt-dpkg-install-0kziPQ/049-libllvm7_1%3a7.0.1-8_s390x.deb (--unpack):
 cannot copy extracted data for './usr/lib/s390x-linux-gnu/libLLVM-7.so.1' to '/usr/lib/s390x-linux-gnu/libLLVM-7.so.1.dpkg-new': failed to write (No space left on device)
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
dpkg: unrecoverable fatal error, aborting:
 unable to flush /var/lib/dpkg/updates/tmp.i after padding: No space left on device
E: Sub-process /usr/bin/dpkg returned an error code (2)
The command "docker run -t ltp /bin/sh -c "cd travis && ./$INSTALL.sh && if [ \"$VARIANT\" ]; then ./$INSTALL.$VARIANT.sh; fi && ../build.sh -o $TREE -t $BUILD -c $CC"" exited with 100.
1 Like

Same for python manylinux2014 docker image builds:
https://travis-ci.org/pypa/manylinux/jobs/639363918?utm_medium=notification&utm_source=github_status

/usr/bin/install: error writing ‘/opt/_internal/cpython-3.8.1/lib/python3.8/pydoc_data/_pydoc.css’: No space left on device
/usr/bin/install: failed to extend ‘/opt/_internal/cpython-3.8.1/lib/python3.8/pydoc_data/_pydoc.css’: No space left on device
/usr/bin/install: error writing ‘/opt/_internal/cpython-3.8.1/lib/python3.8/pydoc_data/topics.py’: No space left on device
/usr/bin/install: failed to extend ‘/opt/_internal/cpython-3.8.1/lib/python3.8/pydoc_data/topics.py’: No space left on device
make: *** [libinstall] Error 1
1 Like

Ditto for the auditwheel project: https://travis-ci.org/pypa/auditwheel/jobs/643561176 shows errors like:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/create

which are due to low disk space on /var/lib/docker according to this issue.

Running df -h in the build shows this volume at 99% full:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/var/snap/.../rootfs   19G  2.1G   17G  11% /
none                  492K  4.0K  488K   1% /dev
udev                  7.9G     0  7.9G   0% /dev/tty
/dev/vda1              97G   96G  1.6G  99% /var/lib/docker
tmpfs                 100K     0  100K   0% /dev/.lxd-mounts
tmpfs                 7.9G   28M  7.9G   1% /dev/shm
tmpfs                 7.9G  8.2M  7.9G   1% /run
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 7.9G     0  7.9G   0% /sys/fs/cgroup

@Damian are you able to take another look?

This is still blocking manylinux builds: https://travis-ci.org/pypa/manylinux/jobs/646636343. Any news on a fix?

Seeing this in our initial attempts to build images on s390x as well (https://travis-ci.org/OpenLiberty/ci.docker/jobs/653640672). Any updates on the issue would be appreciated.

@Damian

This is still an issue for at least manylinux, auditwheel, cibuildwheel projects (urls already listed above).
Are there any update on that issue being fixed you can provide ?

Thanks

This is also something that requires substantial manual intervention from maintainers for the OpenBLAS binaries built for the Scientific Python ecosystem. I have an issue open: https://github.com/MacPython/openblas-libs/issues/27

I don’t normally allow the error to remain (I manually restart failures because we need the build artifacts for the community), but here’s an example fail log: https://travis-ci.org/github/MacPython/openblas-libs/jobs/654967335?utm_medium=notification&utm_source=github_status

We switched away from docker and this is still an issue. https://travis-ci.org/github/bitcoin/bitcoin/jobs/682858666#L336

It doesn’t even boot up and run the travis init scripts at this point: https://travis-ci.org/github/bitcoin/bitcoin/jobs/682963231#L58

There are at least four other threads here on this forum which also discuss this. Unfortunately Travis staff has been silent about this :-(. I collect links to all posts here: S390x Adding APT Sources - No usable temporary directory found - /tmp ENOSPC

Still an issue as of today: https://travis-ci.org/github/bitcoin/bitcoin/jobs/719916458#L697

A check shows that an s390x worker has 18G of space, in accordance with documentation.