Builds hang in queued state

Hi,

Recently I’ve found that our builds became slower and slower after each iteration. Our build is a Scala build on Xenial (default env). Here is an example: https://travis-ci.org/github/rusexpertiza-llc/yupana/jobs/738084928

Previously it takes 1-2 minutes before VM became booting, but today it takes really long time.

Also I’ve found that build fails without any reproducible reason.

Is it known issue (maybe some infrastructure changing is going or something like that), so any chance to see it fixed?


Regards,
Mike.

9 Likes

Agreed. Looking at the status page, it appears that, for the past two days, there has been a huge spike in the Linux job backlog. Whereas normally there are only a few hundred in the backlog, yesterday the backlog spiked at 10 thousand! And today it’s already up to 6500!

3 Likes

the active linux builds status is stuck at 596 since 6 hours and my job hasn’t started since an hour, did travis scale down?

my build is running now :slight_smile: , maybe things are fixed :woman_shrugging:

For me it still broken.

We are in the process of migrating workers from travis-ci.org to travis-ci.com in preparation to fully shut down .org (or rather, make it read-only) on December, 31st.

Unfortunately, this might happen as a result — if we see a job stuck in a queue for more than 16 hours, we will review this on a case by case basis and possibly cancel the job to ensure resources are not being reduced across the platform.

Here’s a FAQ to clear things up: https://docs.travis-ci.com/user/migrate/open-source-repository-migration#frequently-asked-questions.

You may want to consider migrating across to .com sooner rather than later, or consider building at a quieter time of the day if remaining on .org for a while longer.

5 Likes

A build that usually takes 30 minutes took 3.5 hours yesterday.

I’ve been seeing fewer than 5 workers on .org for at least 2 weeks, and on every account I’ve checked. The most I’ve seen is 4.

Right now, I can see one account had 2 workers, now it’s stalled. Two other accounts are stalled.

You may want to consider migrating across to .com sooner rather than later, or consider building at a quieter time of the day if remaining on .org for a while longer.

@Montana A blocker: I would migrate more repos but history isn’t yet migrated, please could you answer the questions here? Thank you!

3 posts were split to a new topic: Extremely poor official communication of the .org shutdown

Thanks for that information and thanks for the service to the open source community in all these years!

Can you clarify in which way moving over to travis-ci.com will help? It seems the capacity problems are affecting travis-ci.com in a similar way? Is the queue shown at traviscistatus.com solely for travis-ci.org builds? From our own experience travis-ci.com recently has been similarly affected by high queuing times.

For many usages moving to travis-ci.com is not possible or beneficial:

  • the problem that @dfawley mentioned about permissions
  • travis-ci.com has big open issues with broken caches and a broken cache cleanup page (see other threads on this forum but we also experienced it ourselves), which needs manual and awkward steps to clear caches. Before I found this page we were at the point of considering to move some projects back from travis-ci.com to travis-ci.org for that reason alone
  • inconsistent communication about the overall migration to travis-ci.com and what to expect and when things happen (the email with the migration deadline announcement wasn’t send to everyone, so what does that mean?)

Altogether, for bigger open source projects (which admittedly should probably have been set up on paid plans or a different kind CI in the first place), the situation points rather to a move away from Travis altogether.

Currently, it’s unclear if OS projects are going to be supported in the same way as they used to, whether the situation is going to improve any time soon, or if Travis would like to cut back on OS project support (which would be fair enough, you don’t owe us anything). It would just be nice to know what to expect :slight_smile:

Johannes

8 Likes

Looking at the monthly stats (https://www.traviscistatus.com/#month), it seems in previous spikes the number of “Active Linux Builds for Open Source projects” rose to meet demand while now it is capped at 555.

1 Like

9 posts were split to a new topic: Migrating from .org to .com in the light of .org shutting down

Finally, I’ve migrated several my repos to travis-ci.com. Everything works as expected. Thanks everybody for help.

1 Like

6 posts were split to a new topic: .org -> .com migration unexpectedly comes with a plan change for OSS. What exactly is the new deal?

I am having the same problem. Build stuck, no progress, repeated attempts to remove and re-add travis integration on github did not help. Looks like this years-old problem is not fixed yet.

This shows “build cancelled”, this does not add my repo despite repeated attempts, while this shows successful builds.

@kagalenko-m-b See this for caveats associated with migration:

I have moved to .com and I regret it for the moment, as no builds have started for 15 days:

This is because Travis CI has put a cap on the number of builds you can do (AFAIR 10k or 100k builds). I’ve apparently reached that cap. Up until that point, indeed, builds started fast!

I am a heavy user of Travis CI for all my FOSS and I predict/hope this cap will be lifted when the migration is done. Travis has always been awesome like that!

I just want to share my experience: do not move to .com yet if you are a heavy Travis CI user, as you cannot move back to .org (which has no such cap) after moving to .com.

What? :confused:
It rather says you’ve run out of credits. See

Ah, incorrect wording, my bad! Indeed, instead of describing it as an experience (‘put a cap on the number of builds’) it is indeed closer to ‘You get 10k credits for free. In that time, builds go fast, until the credits run out’.

Imprint