Sometimes build fails when apt is updating postgresql apt repository

I should note that as a workaround I changed my travis config to look like:

- sudo apt-get update || true

While it is not proper solution, it prevents build from fail

The upstream reports that they cannot migrate to the by-hash mirror scheme because the repo generating software that they use doesn’t support it. It appears that a grant is available to implement that feature. See https://redmine.postgresql.org/issues/4679 for details.


So, the options currently are:

  • Disable Travis’ caching (this would only mitigate the problem)
  • Implement the by-hash scheme at the Travis’ mirror privately
    • since the upstream repo is impossible to sync reliably, we’d have to integrity-check the result before making it available (though a mirroring software that supports syncing from “unreliable” repos should be able to do this automatically)
  • Implement the above-mentioned feature so the upstream can use it – then the transparent mirror can be left unchanged

We’ve deployed


to mitigate this problem.

Please report if you continue to encounter.

Thanks.

2 Likes

We got the problem today …

travis_fold:start:before_install.1
e[0Ktravis_time:start:12a2148a
e[0K$ sudo apt-get update -q
Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:5 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease [46.3 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main amd64 Packages [262 kB]
Get:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages [261 kB]
Err:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages
  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
  Hashes of expected file:
   - Filesize:260866 [weak]
   - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
   - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
   - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
  Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
Fetched 308 kB in 1s (529 kB/s)
Reading package lists...
E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/bionic-pgdg/main/binary-i386/Packages.gz  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
   Hashes of expected file:
    - Filesize:260866 [weak]
    - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
    - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
    - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
   Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
travis_time:end:12a2148a:start=1567602318172404383,finish=1567602320793469877,duration=2621065494,event=before_install
e[0Ke[31;1mThe command "sudo apt-get update -q" failed and exited with 100 during .e[0m

Your build has been stopped.

We are still seeing this issue on ppc64le environments:

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-ppc64el/Packages.gz Hash Sum mismatch

196E: Some index files failed to download. They have been ignored, or old ones used instead.

197Failed at 15: sudo -E apt update

198The command ".ci/setup.sh" failed and exited with 100 during .

@j0k3r Sorry to hear this happened again. Do you have a link to that build where this happened today? If it’s from a private build, could you please send it over to support [AT] travis-ci [DOT] com. Thanks!

It got fixed after around ten retries.

Could you still provide a link to that build so we can check things?

I’ve sent the build link to the travis ci support email!

1 Like

This problem has resurfaced. Things that buildt af few days ago, now get this error, without anything new being pushed. VERY Anoying

@Lars-m Sorry to hear this is happening again.

Could you share with us an example of a build that shows the issue?

If it’s a private build, you can send it over to support [AT] travis-ci [DOT] com .

Thanks!

This is the build that failed: https://travis-ci.org/dat3startcode/rest-jpa-devops-startcode/builds/581202674
Right now it’s OK, because I added the hack given ealier in this thread.
sudo apt-get update || true

Hi all,

I’m getting this too here:

https://travis-ci.org/arupiot/daq/builds/587553195?utm_source=github_status&utm_medium=notification

Is there any indication as to why the error is so intermittent?

On option on the Travis side that has not yet been mentioned here yet is to implement a proper mirror using debmirror. Running that hourly while avoiding the xx:17 “cron.hourly” sync window would fix the problem.

1 Like

I’m recieving the very same error for one of my projects on bionic today.

hitting this today https://travis-ci.com/kind-ci/containerd-nightlies/builds/130446473

All of my projects are running into this issue today.

https://travis-ci.com/gvsucis263/lists-PerpetualLink/builds/130449109

This issue caused a lot of excitement: I use TravisCI to auto grade coding assignments for one of my classes. There was a lot of panic when everybody’s builds started failing at once :slight_smile:

1 Like

The problem was back again around lunchtime today: https://travis-ci.com/gvsucis263/lists-blueberrycola/builds/130580622

The problem seems to persist, even after the patch!

e.g. https://travis-ci.org/libtom/libtomcrypt/jobs/595984944 or https://travis-ci.org/libtom/libtomcrypt/jobs/596013723

But as a special side fact, https://travis-ci.org/sjaeckel/travis-pg-test/jobs/596007087 succeeded at the same time …

We’ve made another try to fix this issue yesterday by deploying

Could you tell us if you are encountering this issue again from now on?

Thanks again!

1 Like
Imprint