Running out of disk space/quota when using docker on ppc64le

Environments Multi CPU Architecture
1

According to the docs, we should get around 18GB worth of disk space available on ppc64le.
Running/Building containers using the included docker service fails with “disk quota exceeded” well before the 18GB limit (in fact 9GB).
This can be seen this pypa/cibuildwheel job or this pypa/auditwheel job. This pypa/cibuildwheel job has df -h logging to help understand the issue.

The fact that the tests ran in these job would require 9GB worth of disk was also kind of worrying. I did an experiment to run all of those tests using a rootless docker service instead:

  • This pypa/cibuildwheel job runs perfectly showing a disk usage which increased by ~1.5GB which is what I would expect (the peak usage is certainly higher, but not 7.5GB higher).
  • This pypa/auditwheel job also runs perfectly showing a disk usage which increased by ~1.6GB (the peak usage is certainly higher, and higher than for the cibuildwheel job, but not as high as to reach the limit)

IMHO, Travis CI should offer the option to run rootless docker or podman in LXD builders. It certainly helps with some permission issues (c.f. Ppc64le docker buildx permission issues depending on which group of workers the job runs on - #3 by mayeut) and would have helped here.
Wether we’re talking about podman or rootless docker, it’s challenging to set those up in a generic way on all those “exotic” platforms and incurs a maintenance cost on projects working around the “native docker service” limitations offered by Travis CI which should thus be fixed by providing alternatives.

What’s Travis CI opinion to these observations ?

2

Podman is not available in the default Ubuntu repositories and a newer version of Ubuntu than the default Travis one is needed. Podman assumes a Fedora/CentOS/RHEL container configuration (/etc/containers/registries.conf).

The problem with rootless containers is, when doing so it emulates a TCP/IP stack in userland and allows to use a network namespace from a container and let it access the outside world (with some limitations). These limitations can be a garden variety of different things, rate limit, all the way from people try nefarious things.

Flannel VXLAN seems to work via how it encapsulates packets, all in all there’s a lot of runtime vulnerabilities. You can certainly replace Docker with Podman via:

dist: xenial
before_install:
- sudo add-apt-repository -y ppa:projectatomic/ppa
- sudo apt-get update -qq
- sudo apt-get install -qq -y software-properties-common podman
- sudo mkdir -p /etc/containers
- sudo sh -c 'echo -e "[registries.search]\nregistries = [\"docker.io\"]" > /etc/containers/registries.conf'

I will bring this topic up to some others and see what they ultimately think. I don’t think it’s a bad idea at all. It obviously takes a bit more knowledge on the end user side, but again I’m not opposed to bringing this up, and I will bring it up and keep you posted.

3

Thanks for the sample config, I will try that on the pypa/cibuildwheel project & get back with my findings.
Thanks also for the caveats about networking.

Just a few remarks:

  • xenial has reached “End of Standard Support” so will try with bionic instead (using focal but the ppa does not provide packages for focal)
  • The ppa doesn’t seem to be supported anymore and the recommendation for installing podman on focal is to use the Kubic project packages (which lack support for ppc64el for now)
  • pypa/auditwheel can’t use podman for now because it uses the docker python package which relies on a docker daemon running (I should have seen this before considering podman as a viable direct alternative).
4

I will try that on the pypa/cibuildwheel project & get back with my findings.

Couldn’t run podman on ppc64le/aarch64/s390x with the proposed ppa. Hiting the following issue: Failed to install podman with download.opensuse.org repo on Travis Ubuntu bionic non x86_64 · Issue #5447 · containers/podman · GitHub, e.g. s390x build

I tried to setup the new OBS repo (except on ppc64le where the package is not available). I could install it but I’ve never been able to run anything, followed multiple pointers to resolve this but couldn’t.
It always ended up with some error. I’m clearly no expert on podman (nor docker) & couldn’t find my way around those errors.

The rootless docker daemon seems to be working fine for all my use cases on ppc64le but it seems not to work well on arm64… Couldn’t test s390x because it’s unfortunately not available from docker on ppc64le/s390x… (for ppc64le, using the build from https://oplab9.parqtec.unicamp.br/pub/ppc64el/docker/version-20.10.6/). Relevant issues for availability from the docker teams are docker v19.03 not available for s390x · Issue #921 · docker/for-linux · GitHub & docker 18.09.x missing packages for power platform · Issue #629 · docker/for-linux · GitHub