Intermittent network outages on macOS builders

Hi, I’m a MacPorts developer and a member of our infrastructure team. We use Travis CI to test pull requests to our ports collection. For months or maybe years now, these builds have failed intermittently due to network issues. This typically manifested with Could not resolve host errors. To test whether this was just a DNS outage or a full network outage, I modified our build startup script to resolve some commonly-used hostnames ahead of time and cache the results in /etc/hosts; I’ve now seen builds fail with Operation timed out errors when trying to get data from those hosts using curl or git which points to a full network failure.

There’s a similar post in this community from 2 weeks ago but I didn’t want to hijack that thread. There, the user was using a build image for an older Xcode version, and using a newer image fixed the problem. In our case, we deliberately want to build on several different versions of macOS and Xcode. In a recent build where I observed the problem (linked above), the builds succeeded on the 10.14.4/Xcode 10.3 and 10.13/Xcode 9.4 builders while failing on the 10.12/Xcode 8.3 and 10.11/Xcode 7.3 builders, but the issue is intermittent and I haven’t paid attention to whether certain OS/Xcode versions are more likely to experience the problem.

Imprint