Docker Hub: You have reached your pull rate limit

dennisameling · November 10, 2020, 2:23pm

DDEV is an open source product which relies heavily on Docker images.

Due to the new Docker pull rate limit, we are now running into the following errors in Travis CI:

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

This is the problematic build:

The problem is that we have a docker login command in the Travis setup script, but this only works for commits pushed to the master branch. It doesn’t work for pull requests, as those don’t have access to the environment variables DOCKERHUB_PULL_USERNAME and DOCKERHUB_PULL_PASSWORD.

github.com

drud/ddev/blob/af32898bd2bee5742e155793489ac7beba669cea/.travis/linux_travis_arm64_setup.sh#L9-L13


if [ ! -z "${DOCKERHUB_PULL_USERNAME:-}" ]; then
  set +x
  echo "${DOCKERHUB_PULL_PASSWORD}" | docker login --username "${DOCKERHUB_PULL_USERNAME}" --password-stdin
  set -x
fi

It looks like the GitHub Actions team is working together with Docker Hub to create a scenario where open source pulls aren’t rate limited for pull requests:

Is the Travis team also planning to build a scenario similar to that of GitHub Actions? We currently can’t think of a good scenario to facilitate Traivs builds in pull requests against drud/ddev.

Thanks in advance!

native-api · November 10, 2020, 2:28pm

dennisameling · November 10, 2020, 2:51pm

Thanks for your reply. Even though it makes sense, I’m especially curious if the Travis team will come up with a solution like GitHub Actions is currently setting up (as mentioned in my post), which will allow Travis to run on pull requests as well without needing any encrypted variables

Montana · November 11, 2020, 8:32am

Just to be sure @dennisameling,

Have you read this?

dennisameling · November 11, 2020, 10:16am

@Montana Yes, that’s exactly what my question is about we’ve already configured a username and password for Docker Hub in the Travis CI pipeline (see my opening post), but that only applies to commits that are done directly in the repo (drud/ddev in this case). When a user opens a Pull Request against our repo (so they fork the repo first), the username and password environment variables aren’t set by Travis due to your security policy:

Similarly, we do not provide these values to untrusted builds, triggered by pull requests from another repository.

We need Travis to run on pull requests as well to see if users’ contributions don’t break things, so we can’t simply disable this part for Pull Requests. Though your security policy makes sense, I saw that the team at GitHub Actions is currently working together with the Docker team to come up with a solution where Open Source projects in their CI pipeline don’t need to authenticate against Docker Hub anymore for publicly available images:

So my question is whether the Travis team is considering a similar approach to what the GitHub Actions team is doing. Thanks in advance