Sometimes build fails when apt is updating postgresql apt repository

This looks like an upstream problem. Traditional apt mirror format is inherently prone to race conditions. This is fixed in Bug #1430011 “support apt by-hash mirrors” : Bugs : Launchpad by providing a mirror format with hash-based file names, support for which is available since Ubuntu 16.04. But http://apt.postgresql.org/pub/repos/apt/ does not use this new format.

I reported this at https://redmine.postgresql.org/issues/4679.

3 Likes

Thanks for the clarification.

This appears to be an issue again today.

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-amd64/Packages.bz2 Hash Sum mismatch
225E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-i386/Packages.bz2 Writing more data than expected (209052 > 209044)
226E: Some index files failed to download. They have been ignored, or old ones used instead.

The problem here seems to be that Travis is adding a caching layer on top of apt.postgresql.org that we (PostgreSQL.org) don’t control. The mentioned IP 34.96.81.152 is not part of the PostgreSQL infrastructure. So whoever added that layer on the Travis side should make sure they got the mirror sync correct.

1 Like

Yes it’s definitely a problem today! Hope this will be fixed soon since it’s a serious “show stopper”

But NOW it seems to be fixed :slight_smile:

This is also a problem right now for us. Still not fixed it seems…

It is still not resolved, we are kind of stuck because of this. Can someone please step in and help out?

I can confirm that we’re hitting someone else’s server (cache proxy) inside TravisCI, and not the source apt.postgresql.org (apt.postgresql.org resolves to 72.32.157.246 on travis, and to 204.145.124.244 on my computer)

so, if you really really need to, you could add this in the before_install phase, at the expense of potentially slower transfers: “echo ‘apt.postgresql.org 204.145.124.244’ | sudo tee -a /etc/hosts; sudo apt-get clean”

I should note that as a workaround I changed my travis config to look like:

- sudo apt-get update || true

While it is not proper solution, it prevents build from fail

The upstream reports that they cannot migrate to the by-hash mirror scheme because the repo generating software that they use doesn’t support it. It appears that a grant is available to implement that feature. See https://redmine.postgresql.org/issues/4679 for details.


So, the options currently are:

  • Disable Travis’ caching (this would only mitigate the problem)
  • Implement the by-hash scheme at the Travis’ mirror privately
    • since the upstream repo is impossible to sync reliably, we’d have to integrity-check the result before making it available (though a mirroring software that supports syncing from “unreliable” repos should be able to do this automatically)
  • Implement the above-mentioned feature so the upstream can use it – then the transparent mirror can be left unchanged

We’ve deployed


to mitigate this problem.

Please report if you continue to encounter.

Thanks.

2 Likes

We got the problem today …

travis_fold:start:before_install.1
e[0Ktravis_time:start:12a2148a
e[0K$ sudo apt-get update -q
Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:5 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease [46.3 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main amd64 Packages [262 kB]
Get:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages [261 kB]
Err:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages
  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
  Hashes of expected file:
   - Filesize:260866 [weak]
   - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
   - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
   - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
  Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
Fetched 308 kB in 1s (529 kB/s)
Reading package lists...
E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/bionic-pgdg/main/binary-i386/Packages.gz  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
   Hashes of expected file:
    - Filesize:260866 [weak]
    - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
    - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
    - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
   Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
travis_time:end:12a2148a:start=1567602318172404383,finish=1567602320793469877,duration=2621065494,event=before_install
e[0Ke[31;1mThe command "sudo apt-get update -q" failed and exited with 100 during .e[0m

Your build has been stopped.

We are still seeing this issue on ppc64le environments:

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-ppc64el/Packages.gz Hash Sum mismatch

196E: Some index files failed to download. They have been ignored, or old ones used instead.

197Failed at 15: sudo -E apt update

198The command ".ci/setup.sh" failed and exited with 100 during .

@j0k3r Sorry to hear this happened again. Do you have a link to that build where this happened today? If it’s from a private build, could you please send it over to support [AT] travis-ci [DOT] com. Thanks!

It got fixed after around ten retries.

Could you still provide a link to that build so we can check things?

I’ve sent the build link to the travis ci support email!

1 Like

This problem has resurfaced. Things that buildt af few days ago, now get this error, without anything new being pushed. VERY Anoying

@Lars-m Sorry to hear this is happening again.

Could you share with us an example of a build that shows the issue?

If it’s a private build, you can send it over to support [AT] travis-ci [DOT] com .

Thanks!

This is the build that failed: https://travis-ci.org/dat3startcode/rest-jpa-devops-startcode/builds/581202674
Right now it’s OK, because I added the hack given ealier in this thread.
sudo apt-get update || true

Imprint