Build delays for open source project

Hi Travis CI Community,

I experience critical build delays for my open source project. Something that took minutes now takes hours or even days.

Here is a build example that takes 2h 20m and not yet over - https://travis-ci.org/github/ets-labs/python-dependency-injector/builds/738348896. There are two more in a queue that even not yet started. Before that it took 15 minutes or less. I’ve started observing delays yesterday.

Is it possible to get rid of these delays and get back to 15-minute builds?

3 Likes

See

2 Likes

I noticed that for the last 24 hours or so the Linux build jobs have been capped at just under 600 from 13:30 on Thursday (whereas it has peaked over the last week in the 1200 range) which is a change from previously. This is with no announcement and NO reports of any incidents. As a result the backlog is ramping up and is in excess of 6,000 (it reached a local minimum of ~900 in the middle of the Thursdays-Friday night whereas it has always cleared in the past)!

https://www.traviscistatus.com/#week

3 Likes

Thanks

Yep ( See @BanzaiMan link. Travis guys say that it’s related to moving from .org to .com.

That isn’t immediately clear as an explanation - as far as I can tell the repositories for the project I code from have already made the switch - it is more as if they have said, "oh we do not have to provision quite as many build workers for the free projects, let’s take off a third and see if they squeal, if not we can save some Euros or use them elsewhere… " :worried:

… OTOH I am a glass half empty guy!

"oh we do not have to provision quite as many build workers for the free projects, let’s take off a third and see if they squeal, if not we can save some Euros or use them elsewhere… " :worried:

Right. I hope Travis will fix it or get the workers back. I’ve been using it for last 3 years and it did a great job for my open source project. As of now, the project development is in collapse cause I have build and publishing process automated as well :slight_smile:

1 Like

It is a way better now. Not sure if it’s related to a fix itself or it’s just cause people don’t work on a weekend :slight_smile: Hope for the first one


No the queue is still long.
It is better because during the weekends there is less build

see here for the graph https://www.traviscistatus.com/

1 Like

Yeah, the cap has now dropped to around 560 jobs - which was only briefly hit on Saturday but we have been hard at that since 15:00 UTC on Sunday though it might just be descending from that peak now it is 23:00 UTC Sunday night.

Of course the situation is not going to improve tomorrow without action by the Travis people - and as things stand I wouldn’t be surprised if the backlog exceeds 10,000 (10K) by Wednesday and does not fall to less than 3K for the rest of the week-days after that point…

2 Likes

Yes, we are hitting this problem as well. It’s going to cause major problems very soon if it’s not resolved. Or, I suppose I can migrate over to travis-ci.com.

Will it really fix the issue? The ‘Travis CI Status’ link from from *.org and *.com leads me to the same page. If *.com projects builds on different pool of machines, how can I look for a status of the queue? Just to be sure that it’s reasonable to move to *.com now and that it will solve the issue we’re all facing.

We switched to travis-ci dot com this morning and are no longer experiencing the delays.

The project I code for had made the switch some time ago and that is being impacted as far as I can tell.

The Linux builds maxed out again at 07:00 UTC and have been hard against that limit (558) ever since - the back log ramped up from zero at that time and plateaued today at around 8K at 15:45 UTC - and has not significantly dropped from that since then…

I’m trying to figure out what exactly is involved in migrating—I know a year or two ago the process wasn’t super straightforward, but it looks like the documentation is here: https://docs.travis-ci.com/user/migrate/open-source-repository-migration

I’ll try this with a few of my repos soon—the ~535 worker cap is getting really annoying, as every weekday now, my builds basically don’t start until a day later!

1 Like

I remember running into this last time too… I am unable to migrate any of my repositories (I have some set up in .com, some older ones in .org.

On the migrate tab, for all the repositories that aren’t already in .com, I see:

And after a few minutes trying everything under the sun, I finally found there’s a ‘Sign up for the beta’ button on my travis-ci.org that I had to click to enroll my “organization” (which was actually my personal account, which already had many projects enrolled on travis-ci.com… so it was unintuitive, but now I can migrate repos.

2 Likes

So does move from *.org to *.com really helps? I see comments that contradict each other.

@AenBleidd My builds at travis-ci.com run fine.

Yes, I am experiencing delays too! Let me try travis-ci.com now.