Builds timing out in last several days

We have a Python project with 13 jobs. Generally these jobs take <10 minutes (often around 5-6) each, as seen here:

About 2 days ago, our jobs suddenly started failing with “The job exceeded the maximum time limit for jobs, and has been terminated.” after 50 minutes:

Note that these were cron jobs, so no code changes occurred between the two jobs.

I assumed this was an individual test timing out, possibly due to a recently release dependency, but I was unable to find a candidate. I also restarted the failing job today and watched it on and off, and it was simply going extremely slowly.

Has there been a change in resource allocations? Is it possible to get some advice on things to try to get our build times back into a reasonable range?

1 Like

I’m seeing this as well, it seems like jobs are being queued for longer than was previously normal. 3min 40s 30min 50s

I updated dist from xenial to bionic. After one slow loop, things started speeding up again, so I don’t know if the issue coincidentally cleared up, or if xenial VMs are getting deprecated and I just needed to fill out some cache, but at least it looks resolved from my end.

Hmm, ok. I was on xenial as well, but switching doesn’t seem to help as my builds are all stuck in the queued state. Thanks for the idea though, glad it seems to be resolved for you.

You can see here that the current linux build backlog has spiked massively and is not being processed at the moment. I guess everyone is affected by this.

It’s strange however that there hasn’t been any official announcement about it.

1 Like

Hello folks,

Please read my response here: Builds hang in queued state.

Thank you,
Montana Mendy
Travis CI Staff