Stalled builds on `opensuse` over the last 12 hours

Query on behalf of SaltStack Formulas.

We’ve starting having stalled runs on all of our opensuse instances over the past 12 hours or so. All of them terminating with the following:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

All of these are unchanged from their working configurations. This includes both opensuse-leap-15 and opensuse-leap-42.

Some examples:

Since we rely upon a successful build to run semantic-release, the only workaround appears to be to disable opensuse in the meantime.

Is there any advice you can give us?

I’ve seen similar issues on Ubuntu

https://travis-ci.org/ska-sa/xarray-ms/jobs/527687744
https://travis-ci.org/ska-sa/xarray-ms/jobs/527837908

I also seen pip timeouts:

https://travis-ci.org/ska-sa/xarray-ms/jobs/527837906

Note that the test runs in these cases are running inside docker containers.

We have very similar case in our case with pip installs:

https://travis-ci.org/apache/airflow/jobs/528104607 or https://travis-ci.org/apache/airflow/jobs/528104609 - also increased number of failures (20%/30% of our jobs) over the course of last few days.

It looks like a routing issues from the Travis build workers… I raised an issue with Travis support

After doing some debugging, we’ve been able to identify that our issue is do with accessing the SUSE mirror for downloading packages. Our instances always get stuck at those points. Via. the debugging, we’re able to keep restarting the process until we finally force it through.

It still works locally though, although the mirrors are quite slow. So wondering whether there is specific issue between Travis and the SUSE mirror(s).

Been having the same issue on a python 3.6 build - switched to xenial with no luck. Randomly started happening yesterday around 10am EDT.

Sometimes hangs installing certain pip dependencies, sometimes during our test suite.

Tried tweeting at @travisci and @traviscistatus with no luck. Others are having the same problem as well: https://github.com/robotology/ycm/issues/254

Another member of our organisation has done further debugging to look at how to access other SUSE mirrors during the build. We appear to be making some progress to a workaround. Any interested parties can see the report at the following link: