macOS builders outage?

Travis CI Discussions & Feedback
1

Twice in a row now, the macOS build failed just saying

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

This is without any CI config change on our end, and the Linux build works fine. Looks like a macOS outage on the Travis side to me, but https://www.traviscistatus.com/ reports “All Systems Operational”.

1 Like
2

same. we have been seeing bad behavior from mac builds for the past few days. many timeouts.

3

My macOS builds don’t even start, even ~30 minutes after the other platforms finished. There’s a sizable backlog of OSX builds listed on the status page but half an hour? Seems odd.

1 Like
4

Same. These past three days we’ve had almost all Travis builds fail due to MacOS jobs timing out. Is the behavior in the graph typical? (from traviscistatus.com). There is a marked rise in the macOS backlog over a month period.

image
image.png888Ă—247 5.93 KB

1 Like
5

Things worked for us after waiting for 3h, but maybe that was just luck.

I only see the graphs there that span the last few hours; how did you get that long-term graph?

1 Like
6

Hi Ralf, thanks for the update. Ours are looking a little better as well. To get to the month graph, click Month to the right of the System Metrics title.

1 Like
7

thanks as well @BlackSwan2019 for the graph. we have noticed as well that the mac executors have grown to be routinely saturated during US working hours. outside those bounds, we have had little to no problems.

edit: we just got a timeout error, on a weekend afternoon. sigh.

edit: 2 more timeouts since last report.

8

@starpit What part of the job is timing out? Ours is occurring during a yarn install phase. To accommodate this new misbehavior, we, for now, are using travis_wait yarn install --network-timeout 1000000 and almost all our builds were then succeeding. Maybe you should use travis_wait or, if applicable, --network-timeout 1000000. The first will allow 20 minutes of no Travis log activity before considering it a timeout. The second option allows processes within the yarn_install command to take up to 16 minutes to perform before considered timing out. It is a hacky, brute force of a “fix” but maybe try that if you haven’t already.

9

thanks for the info. more often than not, the logs have nothing. e.g. https://api.travis-ci.org/v3/job/599996335/log.txt

i’m not sure what to make of that.

10

the backlog is over 625 jobs today. every day it increases by another 30-40%. hello travis? i no longer see any way to report a bug to travis, and they refuse to acknowledge any issues here. time to move away from travis?

11

We are currently working on this issue and we are sorry about this trouble. As soon as our Infra Team solves this issue, we will be posting you on this issue.

Thank you for your patience and again we are sorry about the hassle.

1 Like
13

Hello @RalfJung,
Could you please check if you are still having the same issue?
Best,

14

We had one more timeout yesterday (or was it the day before?) but other than that things worked fine. However, we also typically only have 2-3 builds a day so that doesn’t say much.

1 Like
15

Most of our mac OS builds today have been stalling out. Status shows that the job was received but never moves to queued.

1 Like