Docker Hub: You have reached your pull rate limit

Travis CI Discussions & Feedback
1

DDEV is an open source product which relies heavily on Docker images.

Due to the new Docker pull rate limit, we are now running into the following errors in Travis CI:

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

This is the problematic build:

The problem is that we have a docker login command in the Travis setup script, but this only works for commits pushed to the master branch. It doesn’t work for pull requests, as those don’t have access to the environment variables DOCKERHUB_PULL_USERNAME and DOCKERHUB_PULL_PASSWORD.

It looks like the GitHub Actions team is working together with Docker Hub to create a scenario where open source pulls aren’t rate limited for pull requests:

Is the Travis team also planning to build a scenario similar to that of GitHub Actions? We currently can’t think of a good scenario to facilitate Traivs builds in pull requests against drud/ddev.

Thanks in advance! :rocket:

2 Likes
2
3

Thanks for your reply. Even though it makes sense, I’m especially curious if the Travis team will come up with a solution like GitHub Actions is currently setting up (as mentioned in my post), which will allow Travis to run on pull requests as well without needing any encrypted variables :blush:

2 Likes
4

Just to be sure @dennisameling,

Have you read this?

1 Like
5

@Montana Yes, that’s exactly what my question is about :blush: we’ve already configured a username and password for Docker Hub in the Travis CI pipeline (see my opening post), but that only applies to commits that are done directly in the repo (drud/ddev in this case). When a user opens a Pull Request against our repo (so they fork the repo first), the username and password environment variables aren’t set by Travis due to your security policy:

Similarly, we do not provide these values to untrusted builds, triggered by pull requests from another repository.

We need Travis to run on pull requests as well to see if users’ contributions don’t break things, so we can’t simply disable this part for Pull Requests. Though your security policy makes sense, I saw that the team at GitHub Actions is currently working together with the Docker team to come up with a solution where Open Source projects in their CI pipeline don’t need to authenticate against Docker Hub anymore for publicly available images:

So my question is whether the Travis team is considering a similar approach to what the GitHub Actions team is doing. Thanks in advance :rocket:

2 Likes
6

I don’t know about the Travis team working on it or not… but I lately also ran into this issue. And found a simple workaround - just switch over to another registry mirror like https://mirror.gcr.io. To do that, add the following to your .travis.yml:

before_install:
  - echo 'DOCKER_OPTS="$DOCKER_OPTS --registry-mirror=https://mirror.gcr.io"'
  - sudo service docker restart

I have a full example here: travis-ci-docker-mirror/.travis.yml at main · jonashackt/travis-ci-docker-mirror · GitHub Hope that helps you until Travis and Docker prepared a solution.

3 Likes
7

I think one solution could be to provide environment variables to “trusted” builds, triggered by pull requests from a repository of whitelisted users.

@jonashackt have you found any downside of adding a registry mirror?

Thanks

8

Right now the mirror works great for us - all the latest builds work as expected (GitHub - codecentric/spring-boot-admin: Admin UI for administration of spring boot applications) :slight_smile:

1 Like
9

It just didn’t worked right in a PR of mine :face_with_monocle:

image
image1528Ă—397 42.6 KB

10

That’s strange, try to dig a bit deeper with: docker --log-level debug pull mongo:latest - so we can examine whats happening.

11

I added it but it obviously happens only sometimes!
Meantime, thank you, I’ll update this topic if I found out something else.

12

Any further info or update on this? If travis-ci can’t build pull requests from forks because of this dockerhub rate limit issue, and if github actions can, I guess we’ll have to switch to github actions. :frowning:

13

This didn’t seem to work for me, at least according to docker system info. The only registry mirror listed is https://registry-1.travis-ci.com/. I’m using bionic though, instead of xenial.

Here’s my workaround:

before_install:
  - tmpdaemon=$(mktemp)
  - sudo jq '."registry-mirrors" += ["https://mirror.gcr.io"]' /etc/docker/daemon.json > $tmpdaemon
  - sudo mv $tmpdaemon /etc/docker/daemon.json
  - sudo systemctl daemon-reload
  - sudo systemctl restart docker
  - docker system info

And then docker system info outputs:

Registry Mirrors:
 https://registry-1.travis-ci.com/
 https://mirror.gcr.io/