There is not enough space on the disk

Environments Windows
1

On of my build jobs just randomly gave this error.

C:\Users\travis\.gimme\versions\go1.11.1.windows.amd64\pkg\tool\windows_amd64\link.exe: flushing $WORK\b001\exe\a.out.exe: write $WORK\b001\exe\a.out.exe: There is not enough space on the disk.

When I tried restarting the job, it completed without errors.

The build: Travis CI - Test and Deploy with Confidence

2

Hi @ameer1234567890

We have just released a new Windows image which has more space, so hopefully you shouldnā€™t see this issue again.

If you do, please reply with a link to the Job, and do not restart it.

3

Hey, for reference, I have jobs failing today with ā€˜free spaceā€™ errors (not sure if theyā€™d be running the new image or not):

https://travis-ci.org/joshuef/safe_browser/builds/441430405

4

Got same error: https://travis-ci.org/kubernetes-sigs/cri-tools/jobs/441646725

5

Same error here: https://travis-ci.org/simonpasquier/tsdb/jobs/441655899#L434

6

Iā€™m sorry for the disk issues everyone, we will update our config to include more free disk space today.

7

We have rolled out new configs which include 70gb disks.

Let us know if this is still an issue or resolved?

8

Is there specific worker version to look for? Iā€™m seeing this problem with:

Worker information
hostname: ab6e2d8a-b0e0-42bf-9e76-3386bfc53eb2@1.production-1-worker-com-gce-spww
version: v4.5.0 https://github.com/travis-ci/worker/tree/c4a9dd6cd328917d6d0e03f767514af24755bf94
instance: travis-job-02ce38fe-e588-4661-98b4-fa8bd2a1f79c travis-ci-onion-1803-containers-1539446698-a44782e (via amqp)
startup: 9.127095432s

which looks like an image from 3 days ago, but I donā€™t know how the images are actually set up.

9

@josh Still no luck: System.IO.IOException: There is not enough space on the disk.

10

Also still having issues, might be partly caused because we need LLVM for bindgen.

hostname: 8cf1b350-5831-4338-af79-c29ae427231b@1.production-1-worker-org-gce-vkm5
version: v4.5.0 https://github.com/travis-ci/worker/tree/c4a9dd6cd328917d6d0e03f767514af24755bf94
instance: travis-job-eb024579-6eac-4cdf-a5de-03e9e0b1e223 travis-ci-onion-1803-containers-1539446698-a44782e (via amqp)
startup: 7.396604862s

Build link

11

I had some luck with building my rust project as release build instead of a debug build.

This greatly increases the build time, but seems to reduce the disk usage.

hostname: 58e622bf-79a8-4fe0-9cb3-21e6ee7b751b@1.production-2-worker-org-gce-b0kc
version: v4.5.0 https://github.com/travis-ci/worker/tree/c4a9dd6cd328917d6d0e03f767514af24755bf94
instance: travis-job-3a269fe4-ce90-4ea7-86fc-6fb6dccc16b5 travis-ci-onion-1803-containers-1539446698-a44782e (via amqp)
startup: 7.65476767s

The build still failed for me, Iā€™m currently trying to work around this by splitting up the build.

It would be interesting how much free space is currently available, I think 70gb is the total disk size, not whatā€™s available for the build. It seems some people had luck with forcefully deleting some of the preinstalled programs.

12

Hi all

70gb is the total, but the free space at the start of the job should be around 20-25gb.

Could you please add a df -h at the start and end of the job to help figure out what is going on?

13

df -h reports the filesystem is almost full even at the start of the build:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
C:/program files/git   50G   49G  1.1G  98% /

It seems I managed to squeeze my build into that, but I would appreciate more disk space, I donā€™t feel confident that my windows build is stable enough to merge it into master.

My linux build seems to have 55GB available:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.7G  4.0K  3.7G   1% /dev
tmpfs           748M  292K  748M   1% /run
/dev/sda1        69G   12G   55G  17% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
none            768M  179M  590M  24% /var/ramfs

And my osx build has 118GB available:

$ df -h
Filesystem      Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk0s2   249Gi  131Gi  118Gi    53% 3847396 4291119883    0%   /
devfs          177Ki  177Ki    0Bi   100%     612          0  100%   /dev
map -hosts       0Bi    0Bi    0Bi   100%       0          0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%       0          0  100%   /home
14

@josh Iā€™m not sure how to do that as my job is failing in the checkout stage:

<snipped some proprietary info>

travis_time:start:158f14b0
e[0K$ GIMME_OUTPUT="$(gimme 1.11.1 | tee -a ${TRAVIS_HOME}/.bashrc)" && eval "$GIMME_OUTPUT"
go version go1.11.1 windows/amd64

travis_time:end:158f14b0:start=1539702272439658100,finish=1539702378307174200,duration=105867516100
e[0K$ export GOPATH=${TRAVIS_HOME}/gopath
$ export PATH=${TRAVIS_HOME}/gopath/bin:$PATH
$ mkdir -p ${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
$ tar -Pczf ${TRAVIS_TMPDIR}/src_archive.tar.gz -C ${TRAVIS_BUILD_DIR} . && tar -Pxzf ${TRAVIS_TMPDIR}/src_archive.tar.gz -C ${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
tar: ./.git/modules/openssl/objects/pack/pack-a94f1d9e35c3b7808032d7d95eade0dfff3ecf26.pack: Cannot write: No space left on device
tar: ./.git/modules/openssl/packed-refs: Cannot write: No space left on device

<snipped remainder of failed writes>

tar: Exiting with failure status due to previous errors
$ export TRAVIS_BUILD_DIR=${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
$ cd ${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
travis_time:start:1d36a4a4
e[0Kerror: failed to write new configuration file C:/Users/travis/gopath/src/github.com/saucelabs/project/.git/config.lock

travis_time:end:1d36a4a4:start=1539702448144363100,finish=1539702448274291600,duration=129928500
e[0K
e[31;1mThe command "git config remote.origin.url "${_old_remote%.git}"" failed and exited with 4 during .e[0m

Your build has been stopped.
15

Howdy,

got the same issue a bunch with Packer on pr 6858 https://travis-ci.org/hashicorp/packer/pull_requests.

It started happening after I got mingw and make installed with chocolatey. May be itā€™s an in-between builds remaining cache ?

Or may be removing the cache folder of chocolatey should free some space ?

In any case I believe having mingw by default to run gcc should help.

A lot of our golang tests use gcc.

Thanks !

Edit: itā€™s working now.

see https://travis-ci.org/hashicorp/packer/jobs/442219949

16

We were getting out of space errors yesterday, but our test PR is working now.

# before_install:
Filesystem            Size  Used Avail Use% Mounted on
C:/program files/git   70G   55G   16G  79% /

# after script:
Filesystem            Size  Used Avail Use% Mounted on
C:/program files/git   70G   56G   15G  80% /
17

Hi All

We have updated the image to include a larger disk of 70gb.

Please let me know if the problem is not resolved for you.

1 Like
18

Hi @josh !

Iā€™m a member of the Apache Knox community and we are using Travis CI for building our project for a long time; thanks for your efforts to make it happen.
For more than a year, Knox is being built on ARM64 too (see [KNOX-2535] Build and test Apache Knox on ARM64 CPU architecture - ASF JIRA for more information). However, our ARM64 build is failing for more than a month now due to the same issue as described in this topic:
write /var/lib/docker/tmp/GetImageBlob2121060940: no space left on device
Sample run: KNOX-2778 - Enforce concurrent session limit in KnoxSSO by MrtnBalazs Ā· Pull Request #615 Ā· apache/knox Ā· GitHub

I added df -h into our .travis.yaml to see disk consumption and found this:

Filesystem                                                                                           Size  Used Avail Use% Mounted on
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-apache-knox-579804005/rootfs   47G  2.3G   45G   5% /
none                                                                                                 492K  4.0K  488K   1% /dev
udev                                                                                                 7.6G     0  7.6G   0% /dev/tty
/dev/nvme0n1p1                                                                                        97G   97G  116M 100% /var/lib/docker
tmpfs                                                                                                100K     0  100K   0% /dev/.lxd-mounts
tmpfs                                                                                                7.7G     0  7.7G   0% /dev/shm
tmpfs                                                                                                1.6G  8.2M  1.6G   1% /run
tmpfs                                                                                                5.0M     0  5.0M   0% /run/lock
tmpfs                                                                                                7.7G     0  7.7G   0% /sys/fs/cgroup
travis_time:end:085825ad:start=1660573350031338546,finish=1660573350038305268,duration=6966722,event=before_install

As you can see the /var/lib/docker mount is full indeed. Could you please advise how to make some free space for us?

Thanks in advance,
Sandor

19

Hi Travis Team. Weā€™re facing ā€œno space left on deviceā€ errors when trying to build using arm64-graviton2 arch. It was working but from 2-3 days ago on itā€™s constantly failing.

I tested other options as well, trying ā€œvirt: vmā€ or changing to ā€œarch: arm64ā€ with no success.

20

Have you added a df-h command to your .travis.yml (would suggest this to be under the script hook) to see where itā€™s exactly running out of space at?