Continuous maven repo 403

wisechengyi · June 25, 2019, 11:18pm

Hi,

We keep running into http 403 from maven repo for https://travis-ci.org/pantsbuild/pants. E.g.

                     e[1me[31mE   	22:41:48 00:02     [ivy]e[0m
                     e[1me[31mE   	22:41:48 00:02     [coursier]e[0m
                     e[1me[31mE   	                   Invalidated 4 targets.e[0m
                     e[1me[31mE   	22:41:48 00:02       [coursier]e[0m
                     e[1me[31mE   	                       Result:e[0m
                     e[1me[31mE   	                     â”œâ”€ org.scala-lang:scala-library:2.12.8e[0m
                     e[1me[31mE   	                     â”œâ”€ junit:junit:4.12e[0m
                     e[1me[31mE   	                     â”‚  â””â”€ org.hamcrest:hamcrest-core:1.3e[0m
                     e[1me[31mE   	                     â””â”€ org.scalatest:scalatest_2.12:3.0.0e[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	                     Error:e[0m
                     e[1me[31mE   	                       org.scalatest:scalatest_2.12:3.0.0e[0m
                     e[1me[31mE   	                         No fallback URL founde[0m
                     e[1me[31mE   	                         download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.0/scalatest_2.12-3.0.0.pom (Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.0/scalatest_2.12-3.0.0.pom) while downloading https://repo1.maven.org/maven2/org/scalatest/scalatest_2.12/3.0.0/scalatest_2.12-3.0.0.pome[0m
                     e[1me[31mE   	                       org.scala-lang:scala-library:2.12.8e[0m
                     e[1me[31mE   	                         No fallback URL founde[0m
                     e[1me[31mE   	                         download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.pom (Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.pom) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.pome[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	FAILURE: The coursier process exited non-zero: 1e[0m
                     e[1me[31mE   	e[0m

There was a similar but resolved issue: https://github.com/travis-ci/travis-ci/issues/10053. Would it make sense for travis-ci to check again whether the throttling is back on?

jaapcoomans · June 26, 2019, 8:58am

I’m running into similar issues with this PR: https://github.com/jwtk/jjwt/pull/458

These 2 builds for the same PR failed with the same error message, but for different jobs:
https://travis-ci.org/jwtk/jjwt/builds/550657001
https://travis-ci.org/jwtk/jjwt/builds/550665129

Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:3.3.0 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.felix:maven-bundle-plugin:jar:3.3.0, biz.aQute.bnd:biz.aQute.bndlib:jar:3.3.0, org.easymock:easymock:jar:3.4, org.objenesis:objenesis:jar:2.2, org.apache.felix:org.apache.felix.utils:jar:1.6.0, org.apache.maven:maven-plugin-parameter-documenter:jar:2.2.0: Could not transfer artifact org.apache.felix:maven-bundle-plugin:jar:3.3.0 from/to central (https://repo.maven.apache.org/maven2): Access denied to: https://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/3.3.0/maven-bundle-plugin-3.3.0.jar , ReasonPhrase:Forbidden.

So in some cases the download is successful, but in some cases it fails with a 403. Not 100% sure if it is Travis though, as in alles cases it is complaining about the same artifact.

wallrj · June 26, 2019, 11:15am

Me too: https://github.com/sky-uk/cassandra-operator/issues/141

I found a couple of previous issues and see that others have been recently referencing these in their failed PRs

We’re trying to work around it by caching some of the dependencies:

https://github.com/sky-uk/cassandra-operator/pull/142

wallrj · June 26, 2019, 12:01pm

I created a Maven Central issue at https://issues.sonatype.org/browse/MVNCENTRAL-4985

vorburger · June 26, 2019, 1:33pm

also just hit this, on https://github.com/vorburger/MariaDB4j builds.

Wondering why TravisCI doesn’t just run a Proxy Server, like Nexus, for Maven Central…

Long ago there was such a Maven mirror at Travis, I believe - no longer?

NB: A real mirror server is NOT the same as local ~/.m2 dir caching; depending on the project, that can be problematic - if, for whatever reason, you need to be able to “mvn install” and not “mvn package”, then you can’t use cache - e.g. https://github.com/vorburger/MariaDB4j/pull/263. (Otherwise it’s a great option!)

wallrj · June 26, 2019, 1:57pm

I notice that https://issues.sonatype.org/browse/MVNCENTRAL-4985 is currently labelled Security Level: Reporter and Sonatype so you may not be able to view it. I’ve asked them to make it public.

wisechengyi · June 26, 2019, 4:42pm

Thanks @wallrj!

wisechengyi · June 26, 2019, 9:54pm

Still seeing the issue just now:

                     e[1me[31mE   	21:52:13 00:02     [coursier]e[0m
                     e[1me[31mE   	                   Invalidated 3 targets.e[0m
                     e[1me[31mE   	21:52:13 00:02       [coursier]e[0m
                     e[1me[31mE   	                     ==== stdout ====e[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	                     ==== stderr ====e[0m
                     e[1me[31mE   	                       Result:e[0m
                     e[1me[31mE   	                     â”œâ”€ junit:junit:4.12e[0m
                     e[1me[31mE   	                     â””â”€ org.hamcrest:hamcrest-library:1.3e[0m
                     e[1me[31mE   	                        â””â”€ org.hamcrest:hamcrest-core:1.3e[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	                     Error:e[0m
                     e[1me[31mE   	                       junit:junit:4.12e[0m
                     e[1me[31mE   	                         No fallback URL founde[0m
                     e[1me[31mE   	                         download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.pom (Server returned HTTP response code: 403 for URL: https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.pom) while downloading https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.pome[0m
                     e[1me[31mE   	                     e[0m
                     e[1me[31mE   	FAILURE: The coursier process exited non-zero: 1e[0m
                     e[1me[31mE   	e[0m
                     e[1me[31mE   	e[0m
                     e[1me[31mE   	               Waiting for background workers to finish.e[0m
                     e[1me[31mE   	21:52:16 00:05   [complete]e[0m
                     e[1me[31mE   	               FAILUREe[0m

jaapcoomans · June 27, 2019, 7:58am

I saw more IP’s were white-listed, so I thought to give the build another spin. Unfortunately again 3 out of 4 jobs are failing with this issue: https://travis-ci.org/jwtk/jjwt/builds/551170023.
I saw the request for public IP’s for the failed builds in the Maven Central issue. I’m not that familiar with Travis yet, how can I determine the public IP for my build?

wallrj · June 27, 2019, 9:12am

When I dig the IP addresses for the different OS types, I get a different set than for nat.travisci.net:

diff -u <(dig +short nat.travisci.net | sort | uniq) <(dig +short nat.gce-us-central1.travisci.net nat.macstadium-us-se-1.travisci.net nat.gce-us-central1.travisci.net | sort | uniq)
--- /dev/fd/63	2019-06-27 10:09:13.628876175 +0100
+++ /dev/fd/62	2019-06-27 10:09:13.629876180 +0100
@@ -1,12 +1,13 @@
 104.154.113.151
 104.154.120.187
+104.198.131.58
 207.254.16.35
 207.254.16.36
 207.254.16.37
 207.254.16.38
 207.254.16.39
-34.233.56.198
-34.234.4.53
+34.66.178.120
+34.68.144.114
 35.184.226.236
 35.184.96.71
 35.188.1.99
@@ -14,12 +15,7 @@
 35.192.136.167
 35.192.187.174
 35.192.85.2
+35.193.14.140
 35.193.7.13
 35.202.145.110
 35.224.112.202
-52.3.55.28
-52.45.185.117
-52.45.220.64
-52.54.31.11
-52.54.40.118
-54.208.31.17

So perhaps the correct set of NAT IP addresses is this:

dig +short nat.gce-us-central1.travisci.net nat.macstadium-us-se-1.travisci.net nat.gce-us-central1.travisci.net | sort | uniq
104.154.113.151
104.154.120.187
104.198.131.58
207.254.16.35
207.254.16.36
207.254.16.37
207.254.16.38
207.254.16.39
34.66.178.120
34.68.144.114
35.184.226.236
35.184.96.71
35.188.1.99
35.188.73.34
35.192.136.167
35.192.187.174
35.192.85.2
35.193.14.140
35.193.7.13
35.202.145.110
35.224.112.202

Duologic · June 27, 2019, 3:19pm

Good find, it looks like we still had some DNS records from when we were running on AWS.

I’ve removed these records from nat.travis-ci.net, they will soon be propagated around the globe.

My apologies for the confusion.

wallrj · June 27, 2019, 3:32pm

Thanks. And MVNCENTRAL report that those extra IPs have now been whitelisted: https://issues.sonatype.org/browse/MVNCENTRAL-4985?focusedCommentId=732346&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-732346

BanzaiMan · June 27, 2019, 8:59pm

Hi, everyone!

We have deployed the change to use Google-managed Maven Central mirror (https://maven-central.storage-download.googleapis.com/repos/central/data/). We hope that this improves the future stability and availability of Maven Central repositories.

If you were experiencing the issue before, please run another build. (If you are so kind, please push a new commit or trigger a new build so that we can compare the before and after times for downloading artifacts.)

If you have questions, do let us know!

smowton · June 28, 2019, 8:53am

Seen several “Connection reset” errors using google-maven-central (https://maven-central.storage-download.googleapis.com/) today. How sure are you that your IP addresses are whitelisted with that service?

smowton · June 28, 2019, 10:01am

Notably this seems to affect some of our projects more than others – could be a max concurrent connection limit? If Sonatype have whitelisted us then I’d recommend switching back to their service for the time being.

BanzaiMan · June 28, 2019, 12:37pm

Could you point to the build log that shows this error?

smowton · June 28, 2019, 2:08pm

Unfortunately all my runs where I had this error have since been restarted, but substance of it was a long series of requests to https://maven-central.storage-download.googleapis.com/ followed by a Connection reset error (i.e. it’s not the first request that failed). The build had two different Maven builds running concurrently, so there would have been many concurrent connections active at the time.

antoine-morvan · July 1, 2019, 2:17pm

Hi there,

I also have checksum issues since moving to google central.
For instance in this project, we have

last working build https://travis-ci.org/preesm/graphiti/builds/550544536
first failing due to maven central access denied https://travis-ci.org/preesm/graphiti/builds/551052166
first failing due to checksum issue https://travis-ci.org/preesm/graphiti/builds/551545973

I could reproduce locally playing with local maven settings (https://stackoverflow.com/a/4997553). When I use the default central repository, the build runs fine.

Best.

BanzaiMan · July 1, 2019, 3:35pm

Could you open a new topic to specifically track this issue? Thanks.

lesv · July 1, 2019, 9:34pm

I’m trying to look into the connection reset to the maven-mirror on GCS, but I need a time and maybe a file path that might help us look up what might be going on.

Topic		Replies	Views
Maven repo 403 again Linux build-env	1	804	November 14, 2019
Maven - Received status code 403 from server: Forbidden Travis CI Discussions & Feedback java , bug	0	4005	August 8, 2019
Maven builds are failing in only one repository Environments java	1	878	June 26, 2019
Intermittent issues downloading maven central artifacts Travis CI Discussions & Feedback	1	948	June 26, 2019
[Maven] Maven Central repository restricted communication via HTTPS only which breaks the stock settings.xml Travis CI Discussions & Feedback build-env , java , travis-build	7	1175	January 21, 2020

Continuous maven repo 403

Related topics