Sometimes build fails when apt is updating postgresql apt repository

[MikailBag]

For example, you can see this build log.

Let me highlight important part:

Err:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main amd64 Packages

File has unexpected size (261012 != 261548). Mirror sync in progress? [IP:34.96.81.152 80]

Hashes of expected file:

- Filesize:261548 [weak]

- SHA256:37a20f862843d0ea2ca9178efdd7f46cd8fcbe85926304893f4a9f891633a302

- SHA1:fe531e043053fc60a816800b5e85f38dd757eba2 [weak]

- MD5Sum:e5525fc956ea8b3cce5512b2605dc3c7 [weak]

Release file created at: Mon, 26 Aug 2019 12:54:57 +0000

Get:7 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages [261 kB]

Fetched 88.7 kB in 1s (161 kB/s)

Reading package lists...

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/bionic-pgdg/main/binary-amd64/Packages.gz  File has unexpected size (261012 != 261548). Mirror sync in progress? [IP: 34.96.81.152 80]

   Hashes of expected file:

- Filesize:261548 [weak]

- SHA256:37a20f862843d0ea2ca9178efdd7f46cd8fcbe85926304893f4a9f891633a302

- SHA1:fe531e043053fc60a816800b5e85f38dd757eba2 [weak]

- MD5Sum:e5525fc956ea8b3cce5512b2605dc3c7 [weak]
 Release file created at: Mon, 26 Aug 2019 12:54:57 +0000

E: Some index files failed to download. They have been ignored, or old ones used instead.

The command "sudo apt-get update" failed and exited with 100 during .

Your build has been stopped.

Sometimes, rerunning failed build solves the problem.

[scottlowe]

We are seeing similar issues for postgresql packages with the Xenial image, this morning:

Get:58 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4,428 B]
Fetched 52.0 MB in 5s (9,564 kB/s)
Reading package lists...
E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-amd64/Packages  Writing more data than expected (790354 > 789362)
E: Some index files failed to download. They have been ignored, or old ones used instead.
travis_time:end:00f70d60:start=1566830908143179564,finish=1566830920312008614,duration=12168829050
e[0Ke[31;1mThe command "sudo ./travis/install_base.sh" failed and exited with 100 during .e[0m

I’ve tried clearing local Travis caches, and it doesn’t help. We’ve tried 7 or 8 rebuilds with no resolution.

I note that Postgresql.org have updated a bunch of their distro packages today: http://apt.postgresql.org/pub/repos/apt/dists/

[scottlowe]

This issue seems to have resolved itself for us. Don’t know if somebody stepped in and changed anything… or not, but our builds are now completing the ./travis/install_base.sh script without any problems.

[MikailBag]

Failed again for me: https://travis-ci.org/MikailBag/jjs/builds/577420147

[native-api]

This looks like an upstream problem. Traditional apt mirror format is inherently prone to race conditions. This is fixed in Bug #1430011 “support apt by-hash mirrors” : Bugs : Launchpad by providing a mirror format with hash-based file names, support for which is available since Ubuntu 16.04. But http://apt.postgresql.org/pub/repos/apt/ does not use this new format.

I reported this at https://redmine.postgresql.org/issues/4679.

[scottlowe]

Thanks for the clarification.

[nibbles83]

This appears to be an issue again today.

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-amd64/Packages.bz2 Hash Sum mismatch
225E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-i386/Packages.bz2 Writing more data than expected (209052 > 209044)
226E: Some index files failed to download. They have been ignored, or old ones used instead.

[df7cb]

The problem here seems to be that Travis is adding a caching layer on top of apt.postgresql.org that we (PostgreSQL.org) don’t control. The mentioned IP 34.96.81.152 is not part of the PostgreSQL infrastructure. So whoever added that layer on the Travis side should make sure they got the mirror sync correct.

[Lars-m]

Yes it’s definitely a problem today! Hope this will be fixed soon since it’s a serious “show stopper”

But NOW it seems to be fixed

[yourivoet]

This is also a problem right now for us. Still not fixed it seems…

[cosmos-sajal]

It is still not resolved, we are kind of stuck because of this. Can someone please step in and help out?

[andreineculau]

I can confirm that we’re hitting someone else’s server (cache proxy) inside TravisCI, and not the source apt.postgresql.org (apt.postgresql.org resolves to 72.32.157.246 on travis, and to 204.145.124.244 on my computer)

so, if you really really need to, you could add this in the before_install phase, at the expense of potentially slower transfers: “echo ‘apt.postgresql.org 204.145.124.244’ | sudo tee -a /etc/hosts; sudo apt-get clean”

[MikailBag]

I should note that as a workaround I changed my travis config to look like:

- sudo apt-get update || true

While it is not proper solution, it prevents build from fail

[native-api]

The upstream reports that they cannot migrate to the by-hash mirror scheme because the repo generating software that they use doesn’t support it. It appears that a grant is available to implement that feature. See https://redmine.postgresql.org/issues/4679 for details.

---

So, the options currently are:

• Disable Travis’ caching (this would only mitigate the problem)
• Implement the by-hash scheme at the Travis’ mirror privately
  • since the upstream repo is impossible to sync reliably, we’d have to integrity-check the result before making it available (though a mirroring software that supports syncing from “unreliable” repos should be able to do this automatically)
• Implement the above-mentioned feature so the upstream can use it – then the transparent mirror can be left unchanged

[BanzaiMan]

We’ve deployed

Don't cache packages from postgresql.org

to mitigate this problem.

Please report if you continue to encounter.

Thanks.

[j0k3r]

We got the problem today …

travis_fold:start:before_install.1
e[0Ktravis_time:start:12a2148a
e[0K$ sudo apt-get update -q
Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:5 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease [46.3 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main amd64 Packages [262 kB]
Get:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages [261 kB]
Err:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages
  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
  Hashes of expected file:
   - Filesize:260866 [weak]
   - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
   - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
   - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
  Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
Fetched 308 kB in 1s (529 kB/s)
Reading package lists...
E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/bionic-pgdg/main/binary-i386/Packages.gz  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
   Hashes of expected file:
    - Filesize:260866 [weak]
    - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
    - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
    - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
   Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
travis_time:end:12a2148a:start=1567602318172404383,finish=1567602320793469877,duration=2621065494,event=before_install
e[0Ke[31;1mThe command "sudo apt-get update -q" failed and exited with 100 during .e[0m

Your build has been stopped.

[chavafg]

We are still seeing this issue on ppc64le environments:

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-ppc64el/Packages.gz Hash Sum mismatch

196E: Some index files failed to download. They have been ignored, or old ones used instead.

197Failed at 15: sudo -E apt update

198The command ".ci/setup.sh" failed and exited with 100 during .

[dominic]

@j0k3r Sorry to hear this happened again. Do you have a link to that build where this happened today? If it’s from a private build, could you please send it over to support [AT] travis-ci [DOT] com. Thanks!

[j0k3r]

It got fixed after around ten retries.

[dominic]

Could you still provide a link to that build so we can check things?