Travis Network Timeout Issue

Currently experiencing build issues with sbt (https://travis-ci.org/vinyldns/vinyldns/jobs/568365799)

For some reason, the build hands trying to load the definition.

We have seen output where we are having connection timeouts downloading artifacts needed for the build.

Here is one of the artifacts that timed out - https://dl.bintray.com/typesafe/ivy-releases/javax.annotation/jsr250-api/1.0/jars/jsr250-api.jar

Due to the network timeouts, it seems like the build just hangs and then Travis kills it as it is not seeing any output.

I was able to clear my local workstation of all cached artifacts and do a build locally, confirming that the artifact repositories are reachable.

Any known networking issues?

Seems that a number of people are experiencing the same issue for various network operations (including our project) - see few threads in https://travis-ci.community/latest

FYI, you can see a number of timeouts that happen, this works fine from my own workstation

[error] Server access Error: Connection timed out (Connection timed out) url=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.planet42/laika-pdf_2.12/0.7.5/jars/laika-pdf_2.12.jar

596[error] Server access Error: Connection timed out (Connection timed out) url=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.github.mpilquist/simulacrum_2.12/0.10.0/jars/simulacrum_2.12.jar

597[error] Server access Error: Connection timed out (Connection timed out) url=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.apache.xmlgraphics/batik-xml/1.8/jars/batik-xml.jar

598[error] Server access Error: Connection timed out (Connection timed out) url=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.thesamet.scalapb/protoc-bridge_2.12/0.7.3/jars/protoc-bridge_2.12.jar

599[error] Server access Error: Connection timed out (Connection timed out) url=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.dwijnand/sbt-compat/scala_2.12/sbt_1.0/1.2.6/jars/sbt-compat.jar

It’s definitely not sbt related, as I get it also via Maven and Gradle, and not even Java related, as I use a peculiar setup. I’m not sure if it affects MacOS, but for sure it affects Linux and Windows builds.

Agreed, I updated the title. Definitely network issue

we’ve had persistent networking issues all day today. including contacting travis’s own servers:

E: Failed to fetch http://dl.yarnpkg.com/debian/pool/main/y/yarn/yarn_1.17.3_all.deb Could not connect to apt.cache.travis-ci.com:80 (34.96.81.152), connection timed out

I am experiencing the same issue. Our builds are looking for the presence of files on a http address and many/most are failing.

Same here, for the past few days all the builds face one more network issues, and it seems like even retries don’t help https://travis-ci.org/aptly-dev/aptly/builds/568448363?utm_source=github_status&utm_medium=notification

1 Like

another

curl: (56) GnuTLS recv error (-54): Error in the pull function.

Well, a quarter of my jobs randomly fail because of network issues. Travis seems to be less and less reliable.

I have been investigating this issue since last week, and I have found a few things about the failures that occur:

  • Failures occur in bursts, often blocking retries.
  • The destination never receives the request sent.
  • The size of the request is a significant factor. In my testing, requests with a size of 2200 bytes would reproduce this issue, while 1400 would not (headers & body).
  • Concurrent requests seem to be an aggravating factor.

I came up with a fairly minimal test case to reproduce. I sent 400 POST requests sequentially to https://example.com with a body of 2200 characters (all characters were the letter “a”).

If there is anything further I can provide to help, let me know. This problem has been causing nearly a 100% failure rate for our builds for nearly a week.

A link to a contrived build test I did: https://travis-ci.com/SpeedlineSolutions/SL-OLO/builds/122164359 (sorry, it’s a private repo, but I provide it to help any Travis CI staff)

1 Like

Thanks for the detailed analysis. I would like to second what you said.

My build is doing a lot of downloads, including concurrent ones (with concurrency of four), and even 5 retries (without pause between them) doesn’t seem to solve the problem.

I tried different download locations, nothing seems to help.

Uggh, this has been happening to me all day. So it’s not on my end?

Looks like they are well aware of the issue: https://www.traviscistatus.com/

Response from Travis:

Thanks for writing in and please accept our apologies for the network issues you have been experiencing.

Background
We observed some abnormal behavior in our network which has manifested itself in different forms across our customer builds. In addition, we received reports from customers like you who have experienced some form of interruption in their builds

Current Status
We have given visibility about this to our Engineering Team and the various ways it has affected customers. This is to ensure we focus our resources on fixing the root cause rather than the side-effects.

Next Steps
Our Engineering Team will start looking into it shortly. We might reach out to some of you to help us validate the fixes once they will be pushed to our production systems for the general public to use. Please subscribe to the incident get updates: Travis CI Status - Unreliable or intermittent network connectivity for Linux and Windows builds.

Once again, we apologize for the disruption this is causing and we are working to get things back to normal for you and all other customers.

Sincerely,

Sabri

–
Sabri Helal
Support Engineer

Thank you for your detailed analysis, Wes. I’ve referenced your analysis and build to our Engineering team to help as they investigate this.

I also noticed you just joined the forum. Welcome and thank you for your significant contribution!

Please follow the incident for further updates!

Even though incident says it’s resolved, I still see timeouts in the builds: https://travis-ci.org/aptly-dev/aptly/jobs/568448367

Certainly not as bad as it was before, but I can’t say it became normal.

https://travis-ci.org/jayvdb/tri.declarative/jobs/569157206 - just now, while Travis CI status website says “All Systems Operational” and looks very green.

Worker information

curl: (7) Failed to connect to whereami.travis-ci.com port 443: Connection timed out
system_info

Build system information

Timeout waiting for network availability.

curl: (6) Could not resolve host: cli-assets.heroku.com

xz: (stdin): File format not recognized

tar: Child returned status 1

tar: Error is not recoverable: exiting now

Failed to update Heroku CLI
docker_mtu
resolvconf

pypy3 is not installed; attempting download

Downloading archive: https://storage.googleapis.com/travis-ci-language-archives/python/binaries/ubuntu/16.04/x86_64/pypy3.tar.bz2

131.03s$ curl -sSf -o pypy3.tar.bz2 ${archive_url}

curl: (7) Failed to connect to storage.googleapis.com port 443: Connection timed out

Unable to download pypy3 archive. The archive may not exist. Please consider a different version.
2 Likes

Seeing similar issues here: https://travis-ci.com/bryangingechen/lean-web-editor/jobs/223431128

Worker information
hostname: fea70b76-4707-4964-8248-1a6846329e87@1.worker-com-69c59c6d6b-kldc4.gce-production-1
version: v6.2.1 https://github.com/travis-ci/worker/tree/4e3246c044eb4915c2378ffacd0b3d3ed0136bba
instance: travis-job-47ca3e78-96b3-4ef7-8ebf-03116030e02e travis-ci-sardonyx-xenial-1553530528-f909ac5 (via amqp)
startup: 6.344691549s

curl: (7) Failed to connect to whereami.travis-ci.com port 443: Connection timed out
Download from https://build.travis-ci.com/filter/redirect_io.rb failed. Trying https://build.travis-ci.com/filter/redirect_io.rb ...

Already tried restarting the build once.

Edit: Second restart and it seems to be working for now.

1 Like

I’m also seeing this error and cannot create any builds. This is now two days this week with failed builds.

curl: (7) Failed to connect to whereami.travis-ci.com port 443: Connection timed out

1 Like
curl: (7) Failed to connect to whereami.travis-ci.com port 443: Connection timed out