Sometimes build fails when apt is updating postgresql apt repository

MikailBag · August 26, 2019, 2:47pm

For example, you can see this build log.

Let me highlight important part:

Err:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main amd64 Packages

File has unexpected size (261012 != 261548). Mirror sync in progress? [IP:34.96.81.152 80]

Hashes of expected file:

- Filesize:261548 [weak]

- SHA256:37a20f862843d0ea2ca9178efdd7f46cd8fcbe85926304893f4a9f891633a302

- SHA1:fe531e043053fc60a816800b5e85f38dd757eba2 [weak]

- MD5Sum:e5525fc956ea8b3cce5512b2605dc3c7 [weak]

Release file created at: Mon, 26 Aug 2019 12:54:57 +0000

Get:7 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages [261 kB]

Fetched 88.7 kB in 1s (161 kB/s)

Reading package lists...

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/bionic-pgdg/main/binary-amd64/Packages.gz  File has unexpected size (261012 != 261548). Mirror sync in progress? [IP: 34.96.81.152 80]

   Hashes of expected file:

- Filesize:261548 [weak]

- SHA256:37a20f862843d0ea2ca9178efdd7f46cd8fcbe85926304893f4a9f891633a302

- SHA1:fe531e043053fc60a816800b5e85f38dd757eba2 [weak]

- MD5Sum:e5525fc956ea8b3cce5512b2605dc3c7 [weak]
 Release file created at: Mon, 26 Aug 2019 12:54:57 +0000

E: Some index files failed to download. They have been ignored, or old ones used instead.

The command "sudo apt-get update" failed and exited with 100 during .

Your build has been stopped.

Sometimes, rerunning failed build solves the problem.

scottlowe · August 26, 2019, 2:51pm

We are seeing similar issues for postgresql packages with the Xenial image, this morning:

Get:58 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4,428 B]
Fetched 52.0 MB in 5s (9,564 kB/s)
Reading package lists...
E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-amd64/Packages  Writing more data than expected (790354 > 789362)
E: Some index files failed to download. They have been ignored, or old ones used instead.
travis_time:end:00f70d60:start=1566830908143179564,finish=1566830920312008614,duration=12168829050
e[0Ke[31;1mThe command "sudo ./travis/install_base.sh" failed and exited with 100 during .e[0m

I’ve tried clearing local Travis caches, and it doesn’t help. We’ve tried 7 or 8 rebuilds with no resolution.

I note that Postgresql.org have updated a bunch of their distro packages today: http://apt.postgresql.org/pub/repos/apt/dists/

scottlowe · August 26, 2019, 7:27pm

This issue seems to have resolved itself for us. Don’t know if somebody stepped in and changed anything… or not, but our builds are now completing the ./travis/install_base.sh script without any problems.

MikailBag · August 27, 2019, 4:12pm

Failed again for me: https://travis-ci.org/MikailBag/jjs/builds/577420147

native-api · August 27, 2019, 7:56pm

This looks like an upstream problem. Traditional apt mirror format is inherently prone to race conditions. This is fixed in Bug #1430011 “support apt by-hash mirrors” : Bugs : Launchpad by providing a mirror format with hash-based file names, support for which is available since Ubuntu 16.04. But http://apt.postgresql.org/pub/repos/apt/ does not use this new format.

I reported this at https://redmine.postgresql.org/issues/4679.

scottlowe · August 27, 2019, 8:45pm

Thanks for the clarification.

nibbles83 · August 28, 2019, 11:46am

This appears to be an issue again today.

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-amd64/Packages.bz2 Hash Sum mismatch
225E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-i386/Packages.bz2 Writing more data than expected (209052 > 209044)
226E: Some index files failed to download. They have been ignored, or old ones used instead.

df7cb · August 28, 2019, 11:59am

The problem here seems to be that Travis is adding a caching layer on top of apt.postgresql.org that we (PostgreSQL.org) don’t control. The mentioned IP 34.96.81.152 is not part of the PostgreSQL infrastructure. So whoever added that layer on the Travis side should make sure they got the mirror sync correct.

Lars-m · August 28, 2019, 12:18pm

Yes it’s definitely a problem today! Hope this will be fixed soon since it’s a serious “show stopper”

But NOW it seems to be fixed

yourivoet · August 28, 2019, 1:06pm

This is also a problem right now for us. Still not fixed it seems…

cosmos-sajal · August 28, 2019, 1:08pm

It is still not resolved, we are kind of stuck because of this. Can someone please step in and help out?

andreineculau · August 28, 2019, 1:19pm

I can confirm that we’re hitting someone else’s server (cache proxy) inside TravisCI, and not the source apt.postgresql.org (apt.postgresql.org resolves to 72.32.157.246 on travis, and to 204.145.124.244 on my computer)

so, if you really really need to, you could add this in the before_install phase, at the expense of potentially slower transfers: “echo ‘apt.postgresql.org 204.145.124.244’ | sudo tee -a /etc/hosts; sudo apt-get clean”

MikailBag · August 28, 2019, 1:54pm

I should note that as a workaround I changed my travis config to look like:

- sudo apt-get update || true

While it is not proper solution, it prevents build from fail

native-api · August 28, 2019, 5:26pm

The upstream reports that they cannot migrate to the by-hash mirror scheme because the repo generating software that they use doesn’t support it. It appears that a grant is available to implement that feature. See https://redmine.postgresql.org/issues/4679 for details.

So, the options currently are:

Disable Travis’ caching (this would only mitigate the problem)
Implement the by-hash scheme at the Travis’ mirror privately
- since the upstream repo is impossible to sync reliably, we’d have to integrity-check the result before making it available (though a mirroring software that supports syncing from “unreliable” repos should be able to do this automatically)
Implement the above-mentioned feature so the upstream can use it – then the transparent mirror can be left unchanged

BanzaiMan · August 29, 2019, 5:34pm

We’ve deployed

to mitigate this problem.

Please report if you continue to encounter.

Thanks.

j0k3r · September 4, 2019, 1:09pm

We got the problem today …

travis_fold:start:before_install.1
e[0Ktravis_time:start:12a2148a
e[0K$ sudo apt-get update -q
Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:5 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease [46.3 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main amd64 Packages [262 kB]
Get:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages [261 kB]
Err:6 http://apt.postgresql.org/pub/repos/apt bionic-pgdg/main i386 Packages
  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
  Hashes of expected file:
   - Filesize:260866 [weak]
   - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
   - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
   - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
  Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
Fetched 308 kB in 1s (529 kB/s)
Reading package lists...
E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/bionic-pgdg/main/binary-i386/Packages.gz  File has unexpected size (260865 != 260866). Mirror sync in progress? [IP: 34.96.81.152 80]
   Hashes of expected file:
    - Filesize:260866 [weak]
    - SHA256:433bef097d8a54a9899350c182d0074c1a13f62c8e7e9987cc6c63cd11242abc
    - SHA1:1be55e080a1dd277929f095690ae9b9cf01e971f [weak]
    - MD5Sum:08189bf54aa297f53b9656bc3c529c62 [weak]
   Release file created at: Mon, 02 Sep 2019 10:25:33 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
travis_time:end:12a2148a:start=1567602318172404383,finish=1567602320793469877,duration=2621065494,event=before_install
e[0Ke[31;1mThe command "sudo apt-get update -q" failed and exited with 100 during .e[0m

Your build has been stopped.

chavafg · September 4, 2019, 1:32pm

We are still seeing this issue on ppc64le environments:

E: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-ppc64el/Packages.gz Hash Sum mismatch

196E: Some index files failed to download. They have been ignored, or old ones used instead.

197Failed at 15: sudo -E apt update

198The command ".ci/setup.sh" failed and exited with 100 during .

dominic · September 4, 2019, 5:22pm

@j0k3r Sorry to hear this happened again. Do you have a link to that build where this happened today? If it’s from a private build, could you please send it over to support [AT] travis-ci [DOT] com. Thanks!

j0k3r · September 5, 2019, 7:42am

It got fixed after around ten retries.

dominic · September 5, 2019, 12:41pm

Could you still provide a link to that build so we can check things?

Topic		Replies	Views
PostGIS addon for PostgreSQL 10 started failing Xenial	1	1279	August 28, 2019
Apt addon with configured timeouts fails Travis CI Discussions & Feedback	1	816	April 14, 2021
"sudo apt-get -qq update" failed and exited with 100 during Travis CI Discussions & Feedback travis-build	1	981	September 20, 2022
"sudo apt update" failure in linux Linux	2	2605	July 23, 2019
Bionic builds fail, getting Xenial instead R	3	1691	August 28, 2019

Sometimes build fails when apt is updating postgresql apt repository

Related Topics