Output is truncated heavily in ARM64 when a command hangs

In https://travis-ci.org/github/native-api/test_travis/jobs/663728118, I accidentally read stdin with cat -n "${FF[@]}" when $FF happened to be empty.

It caused the build to hang (understandably), but output got truncated mid-line and long before the offending code.
(A sibling job in AMD64 environment, with exact same settings, correctly displayed all output up to the offending line.)

In another build with this defect, https://travis-ci.org/github/native-api/test_travis/jobs/663783018, the output got truncated at exactly the same place – which suggests that the output defect is connected to stream buffering.

2 Likes

The stream buffering issue also seems to cause timeouts for Arm64 jobs.

And since the log is incomplete (due to buffering) it looks like the job hung in the middle of an earlier command, sending the customer down a false trail.

1 Like

Came here to add a +1 to the buffering issue. I’ve even had lines get cut off in the middle due to buffering, and then the build fails because, I suspect, not enough text gets put in the buffer in time.

https://travis-ci.org/github/Kansattica/msync/jobs/675085943
https://travis-ci.org/github/Kansattica/msync/jobs/675423402

Same problem here: https://travis-ci.org/github/solvespace/solvespace/jobs/679218467

It’s failing all the time with the last line in the log being truncated. The build times out then due to no new output for ten minutes.

Any ideas? I’ve fiddled way too much with output rate limiting, different flags for stdout & stderr and whatnot. Nothing helps.

Maybe the buffering issue should be filed separately.

Hi,
Any updates about this issue ?
I have met same problem, I am trying to build and running tests of Storm on ARM platform with Travis, the CI job easily hangs and no explicit reason, even the job has already run sucessfully. here are 2 examples:
https://travis-ci.com/github/liusheng/storm/jobs/323797329
https://travis-ci.com/github/liusheng/storm/jobs/323797330

The same CI jobs run OK on x86 platform.

1 Like

I wish there was… this issue has been reported may times since October 2019; I tried to list all posts doing so here Multiarch builds (lxd) always time out on failure instead of exit

See also Travis Build gets stalled for ARM64 job in pandas package here for the workaround that I use (but it’s horrible)

Imprint