There is not enough space on the disk


#1

On of my build jobs just randomly gave this error.

C:\Users\travis\.gimme\versions\go1.11.1.windows.amd64\pkg\tool\windows_amd64\link.exe: flushing $WORK\b001\exe\a.out.exe: write $WORK\b001\exe\a.out.exe: There is not enough space on the disk.

When I tried restarting the job, it completed without errors.

The build: https://travis-ci.org/ameer1234567890/tinifier-go/jobs/441043741


#2

Hi @ameer1234567890

We have just released a new Windows image which has more space, so hopefully you shouldn’t see this issue again.

If you do, please reply with a link to the Job, and do not restart it.


#3

Hey, for reference, I have jobs failing today with ‘free space’ errors (not sure if they’d be running the new image or not):

https://travis-ci.org/joshuef/safe_browser/builds/441430405


#4

Got same error: https://travis-ci.org/kubernetes-sigs/cri-tools/jobs/441646725


#5

Same error here: https://travis-ci.org/simonpasquier/tsdb/jobs/441655899#L434


#6

I’m sorry for the disk issues everyone, we will update our config to include more free disk space today.


#7

We have rolled out new configs which include 70gb disks.

Let us know if this is still an issue or resolved?


#8

Is there specific worker version to look for? I’m seeing this problem with:

Worker information
hostname: ab6e2d8a-b0e0-42bf-9e76-3386bfc53eb2@1.production-1-worker-com-gce-spww
version: v4.5.0 https://github.com/travis-ci/worker/tree/c4a9dd6cd328917d6d0e03f767514af24755bf94
instance: travis-job-02ce38fe-e588-4661-98b4-fa8bd2a1f79c travis-ci-onion-1803-containers-1539446698-a44782e (via amqp)
startup: 9.127095432s

which looks like an image from 3 days ago, but I don’t know how the images are actually set up.


#9

@josh Still no luck: System.IO.IOException: There is not enough space on the disk.


#10

Also still having issues, might be partly caused because we need LLVM for bindgen.

hostname: 8cf1b350-5831-4338-af79-c29ae427231b@1.production-1-worker-org-gce-vkm5
version: v4.5.0 https://github.com/travis-ci/worker/tree/c4a9dd6cd328917d6d0e03f767514af24755bf94
instance: travis-job-eb024579-6eac-4cdf-a5de-03e9e0b1e223 travis-ci-onion-1803-containers-1539446698-a44782e (via amqp)
startup: 7.396604862s

Build link


#11

I had some luck with building my rust project as release build instead of a debug build.

This greatly increases the build time, but seems to reduce the disk usage.

hostname: 58e622bf-79a8-4fe0-9cb3-21e6ee7b751b@1.production-2-worker-org-gce-b0kc
version: v4.5.0 https://github.com/travis-ci/worker/tree/c4a9dd6cd328917d6d0e03f767514af24755bf94
instance: travis-job-3a269fe4-ce90-4ea7-86fc-6fb6dccc16b5 travis-ci-onion-1803-containers-1539446698-a44782e (via amqp)
startup: 7.65476767s

The build still failed for me, I’m currently trying to work around this by splitting up the build.

It would be interesting how much free space is currently available, I think 70gb is the total disk size, not what’s available for the build. It seems some people had luck with forcefully deleting some of the preinstalled programs.


#12

Hi all

70gb is the total, but the free space at the start of the job should be around 20-25gb.

Could you please add a df -h at the start and end of the job to help figure out what is going on?


#13

df -h reports the filesystem is almost full even at the start of the build:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
C:/program files/git   50G   49G  1.1G  98% /

It seems I managed to squeeze my build into that, but I would appreciate more disk space, I don’t feel confident that my windows build is stable enough to merge it into master.

My linux build seems to have 55GB available:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.7G  4.0K  3.7G   1% /dev
tmpfs           748M  292K  748M   1% /run
/dev/sda1        69G   12G   55G  17% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
none            768M  179M  590M  24% /var/ramfs

And my osx build has 118GB available:

$ df -h
Filesystem      Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk0s2   249Gi  131Gi  118Gi    53% 3847396 4291119883    0%   /
devfs          177Ki  177Ki    0Bi   100%     612          0  100%   /dev
map -hosts       0Bi    0Bi    0Bi   100%       0          0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%       0          0  100%   /home

#14

@josh I’m not sure how to do that as my job is failing in the checkout stage:

<snipped some proprietary info>

travis_time:start:158f14b0
e[0K$ GIMME_OUTPUT="$(gimme 1.11.1 | tee -a ${TRAVIS_HOME}/.bashrc)" && eval "$GIMME_OUTPUT"
go version go1.11.1 windows/amd64

travis_time:end:158f14b0:start=1539702272439658100,finish=1539702378307174200,duration=105867516100
e[0K$ export GOPATH=${TRAVIS_HOME}/gopath
$ export PATH=${TRAVIS_HOME}/gopath/bin:$PATH
$ mkdir -p ${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
$ tar -Pczf ${TRAVIS_TMPDIR}/src_archive.tar.gz -C ${TRAVIS_BUILD_DIR} . && tar -Pxzf ${TRAVIS_TMPDIR}/src_archive.tar.gz -C ${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
tar: ./.git/modules/openssl/objects/pack/pack-a94f1d9e35c3b7808032d7d95eade0dfff3ecf26.pack: Cannot write: No space left on device
tar: ./.git/modules/openssl/packed-refs: Cannot write: No space left on device

<snipped remainder of failed writes>

tar: Exiting with failure status due to previous errors
$ export TRAVIS_BUILD_DIR=${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
$ cd ${TRAVIS_HOME}/gopath/src/github.com/saucelabs/project
travis_time:start:1d36a4a4
e[0Kerror: failed to write new configuration file C:/Users/travis/gopath/src/github.com/saucelabs/project/.git/config.lock

travis_time:end:1d36a4a4:start=1539702448144363100,finish=1539702448274291600,duration=129928500
e[0K
e[31;1mThe command "git config remote.origin.url "${_old_remote%.git}"" failed and exited with 4 during .e[0m

Your build has been stopped.

#15

Howdy,

got the same issue a bunch with Packer on pr 6858 https://travis-ci.org/hashicorp/packer/pull_requests.

It started happening after I got mingw and make installed with chocolatey. May be it’s an in-between builds remaining cache ?

Or may be removing the cache folder of chocolatey should free some space ?

In any case I believe having mingw by default to run gcc should help.

A lot of our golang tests use gcc.

Thanks !

Edit: it’s working now.

see https://travis-ci.org/hashicorp/packer/jobs/442219949


#16

We were getting out of space errors yesterday, but our test PR is working now.

# before_install:
Filesystem            Size  Used Avail Use% Mounted on
C:/program files/git   70G   55G   16G  79% /
# after script:
Filesystem            Size  Used Avail Use% Mounted on
C:/program files/git   70G   56G   15G  80% /

#17

Hi All

We have updated the image to include a larger disk of 70gb.

Please let me know if the problem is not resolved for you.