Continuous maven repo 403

Hi,

We keep running into http 403 from maven repo for https://travis-ci.org/pantsbuild/pants. E.g.

                     e[1me[31mE   	22:41:48 00:02     [ivy]e[0m
                     e[1me[31mE   	22:41:48 00:02     [coursier]e[0m
                     e[1me[31mE   	                   Invalidated 4 targets.e[0m
                     e[1me[31mE   	22:41:48 00:02       [coursier]e[0m
                     e[1me[31mE   	                       Result:e[0m
                     e[1me[31mE   	                     ├─ org.scala-lang:scala-library:2.12.8e[0m
                     e[1me[31mE   	                     ├─ junit:junit:4.12e[0m
                     e[1me[31mE   	                     │  └─ org.hamcrest:hamcrest-core:1.3e[0m
                     e[1me[31mE   	                     └─ org.scalatest:scalatest_2.12:3.0.0e[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	                     Error:e[0m
                     e[1me[31mE   	                       org.scalatest:scalatest_2.12:3.0.0e[0m
                     e[1me[31mE   	                         No fallback URL founde[0m
                     e[1me[31mE   	                         download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.0/scalatest_2.12-3.0.0.pom (Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.0/scalatest_2.12-3.0.0.pom) while downloading https://repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.0/scalatest_2.12-3.0.0.pome[0m
                     e[1me[31mE   	                       org.scala-lang:scala-library:2.12.8e[0m
                     e[1me[31mE   	                         No fallback URL founde[0m
                     e[1me[31mE   	                         download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.pom (Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.pom) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.pome[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	FAILURE: The coursier process exited non-zero: 1e[0m
                     e[1me[31mE   	e[0m

There was a similar but resolved issue: https://github.com/travis-ci/travis-ci/issues/10053. Would it make sense for travis-ci to check again whether the throttling is back on?

2 Likes

I’m running into similar issues with this PR: https://github.com/jwtk/jjwt/pull/458

These 2 builds for the same PR failed with the same error message, but for different jobs:
https://travis-ci.org/jwtk/jjwt/builds/550657001
https://travis-ci.org/jwtk/jjwt/builds/550665129

Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:3.3.0 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.felix:maven-bundle-plugin:jar:3.3.0, biz.aQute.bnd:biz.aQute.bndlib:jar:3.3.0, org.easymock:easymock:jar:3.4, org.objenesis:objenesis:jar:2.2, org.apache.felix:org.apache.felix.utils:jar:1.6.0, org.apache.maven:maven-plugin-parameter-documenter:jar:2.2.0: Could not transfer artifact org.apache.felix:maven-bundle-plugin:jar:3.3.0 from/to central (https://repo.maven.apache.org/maven2): Access denied to: https://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/3.3.0/maven-bundle-plugin-3.3.0.jar , ReasonPhrase:Forbidden.

So in some cases the download is successful, but in some cases it fails with a 403. Not 100% sure if it is Travis though, as in alles cases it is complaining about the same artifact.

Me too: https://github.com/sky-uk/cassandra-operator/issues/141

I found a couple of previous issues and see that others have been recently referencing these in their failed PRs

We’re trying to work around it by caching some of the dependencies:

I created a Maven Central issue at https://issues.sonatype.org/browse/MVNCENTRAL-4985

also just hit this, on https://github.com/vorburger/MariaDB4j builds.

Wondering why TravisCI doesn’t just run a Proxy Server, like Nexus, for Maven Central…

Long ago there was such a Maven mirror at Travis, I believe - no longer?

NB: A real mirror server is NOT the same as local ~/.m2 dir caching; depending on the project, that can be problematic - if, for whatever reason, you need to be able to “mvn install” and not “mvn package”, then you can’t use cache - e.g. https://github.com/vorburger/MariaDB4j/pull/263. (Otherwise it’s a great option!)

I notice that https://issues.sonatype.org/browse/MVNCENTRAL-4985 is currently labelled Security Level: Reporter and Sonatype so you may not be able to view it. I’ve asked them to make it public.

Thanks @wallrj!

Still seeing the issue just now:

                     e[1me[31mE   	21:52:13 00:02     [coursier]e[0m
                     e[1me[31mE   	                   Invalidated 3 targets.e[0m
                     e[1me[31mE   	21:52:13 00:02       [coursier]e[0m
                     e[1me[31mE   	                     ==== stdout ====e[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	                     ==== stderr ====e[0m
                     e[1me[31mE   	                       Result:e[0m
                     e[1me[31mE   	                     ├─ junit:junit:4.12e[0m
                     e[1me[31mE   	                     └─ org.hamcrest:hamcrest-library:1.3e[0m
                     e[1me[31mE   	                        └─ org.hamcrest:hamcrest-core:1.3e[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	                     Error:e[0m
                     e[1me[31mE   	                       junit:junit:4.12e[0m
                     e[1me[31mE   	                         No fallback URL founde[0m
                     e[1me[31mE   	                         download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.pom (Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.pom) while downloading https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.pome[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	FAILURE: The coursier process exited non-zero: 1e[0m
                     e[1me[31mE   	e[0m
                     e[1me[31mE   	e[0m
                     e[1me[31mE   	               Waiting for background workers to finish.e[0m
                     e[1me[31mE   	21:52:16 00:05   [complete]e[0m
                     e[1me[31mE   	               FAILUREe[0m

I saw more IP’s were white-listed, so I thought to give the build another spin. Unfortunately again 3 out of 4 jobs are failing with this issue: https://travis-ci.org/jwtk/jjwt/builds/551170023.
I saw the request for public IP’s for the failed builds in the Maven Central issue. I’m not that familiar with Travis yet, how can I determine the public IP for my build?

When I dig the IP addresses for the different OS types, I get a different set than for nat.travisci.net:

diff -u <(dig +short nat.travisci.net | sort | uniq) <(dig +short nat.gce-us-central1.travisci.net nat.macstadium-us-se-1.travisci.net nat.gce-us-central1.travisci.net | sort | uniq)
--- /dev/fd/63	2019-06-27 10:09:13.628876175 +0100
+++ /dev/fd/62	2019-06-27 10:09:13.629876180 +0100
@@ -1,12 +1,13 @@
 104.154.113.151
 104.154.120.187
+104.198.131.58
 207.254.16.35
 207.254.16.36
 207.254.16.37
 207.254.16.38
 207.254.16.39
-34.233.56.198
-34.234.4.53
+34.66.178.120
+34.68.144.114
 35.184.226.236
 35.184.96.71
 35.188.1.99
@@ -14,12 +15,7 @@
 35.192.136.167
 35.192.187.174
 35.192.85.2
+35.193.14.140
 35.193.7.13
 35.202.145.110
 35.224.112.202
-52.3.55.28
-52.45.185.117
-52.45.220.64
-52.54.31.11
-52.54.40.118
-54.208.31.17

So perhaps the correct set of NAT IP addresses is this:

dig +short nat.gce-us-central1.travisci.net nat.macstadium-us-se-1.travisci.net nat.gce-us-central1.travisci.net | sort | uniq
104.154.113.151
104.154.120.187
104.198.131.58
207.254.16.35
207.254.16.36
207.254.16.37
207.254.16.38
207.254.16.39
34.66.178.120
34.68.144.114
35.184.226.236
35.184.96.71
35.188.1.99
35.188.73.34
35.192.136.167
35.192.187.174
35.192.85.2
35.193.14.140
35.193.7.13
35.202.145.110
35.224.112.202

Good find, it looks like we still had some DNS records from when we were running on AWS.

I’ve removed these records from nat.travis-ci.net, they will soon be propagated around the globe.

My apologies for the confusion.

Thanks. And MVNCENTRAL report that those extra IPs have now been whitelisted: https://issues.sonatype.org/browse/MVNCENTRAL-4985?focusedCommentId=732346&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-732346

1 Like

Hi, everyone!

We have deployed the change to use Google-managed Maven Central mirror (https://maven-central.storage-download.googleapis.com/repos/central/data/). We hope that this improves the future stability and availability of Maven Central repositories.

If you were experiencing the issue before, please run another build. (If you are so kind, please push a new commit or trigger a new build so that we can compare the before and after times for downloading artifacts.)

If you have questions, do let us know!

1 Like

Seen several “Connection reset” errors using google-maven-central (https://maven-central.storage-download.googleapis.com/) today. How sure are you that your IP addresses are whitelisted with that service?

Notably this seems to affect some of our projects more than others – could be a max concurrent connection limit? If Sonatype have whitelisted us then I’d recommend switching back to their service for the time being.

Could you point to the build log that shows this error?

Unfortunately all my runs where I had this error have since been restarted, but substance of it was a long series of requests to https://maven-central.storage-download.googleapis.com/ followed by a Connection reset error (i.e. it’s not the first request that failed). The build had two different Maven builds running concurrently, so there would have been many concurrent connections active at the time.

Hi there,

I also have checksum issues since moving to google central.
For instance in this project, we have

I could reproduce locally playing with local maven settings (https://stackoverflow.com/a/4997553). When I use the default central repository, the build runs fine.

Best.

Could you open a new topic to specifically track this issue? Thanks.

I’m trying to look into the connection reset to the maven-mirror on GCS, but I need a time and maybe a file path that might help us look up what might be going on.

1 Like