Sudden failures for the same commit despite no changes in the list of installed packages

Languages Python
1

Five days ago, the last merge to our master branch succeeded on Travis

Today, I tried to run some builds in a test branch to see if bumping a certain dependency would break anything. However, after getting very confusing build results I got sidetracked and eventually discovered that the exact same commit now fails.

Python 3.3 builds say there’s No module named 'typing' (true, but while there’s a PyPI package to backport it the list of installed packages hasn’t changed between jobs).

Python 3.5 builds point to a syntax error in one of our dependencies (geoip2 2.9.0) that hasn’t changed in years.

These errors don’t make sense, since the exact same build succeeded a few days ago. I can only conclude that something in the Travis CI Python archives has changed due to a recent update. It’s either that or the package artifact on PyPI changed, and we all know the policy there. (In case not: It’s impossible to replace an uploaded artifact, period. Broken releases must be version-bumped.)

2

Please check your list of packages being installed more closely. Since you are using >= specificaltions rather than ==, you will be getting newer versions whenever they are available.

Travis CI - Test and Deploy with Confidence vs Travis CI - Test and Deploy with Confidence

-Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-3.13 Pygments-2.6.1 alabaster-0.7.12 babel-2.8.0 certifi-2020.6.20 chardet-3.0.4 coverage-5.2 coveralls-2.1.1 dnspython-1.15.0 docopt-0.6.2 docutils-0.16 enum34-1.1.10 flake8-3.5.0 flake8-coding-1.3.2 flake8-future-import-0.4.5 flake8-import-order-0.18.1 geoip2-2.9.0 idna-2.10 imagesize-1.2.0 maxminddb-1.5.4 mccabe-0.6.1 packaging-20.4 praw-5.4.0 prawcore-0.14.0 py-1.9.0 pycodestyle-2.3.1 pyflakes-1.6.0 pyparsing-2.4.7 pytest-3.2.5 pytest-vcr-0.3.0 pytz-2020.1 requests-2.24.0 six-1.15.0 snowballstemmer-2.0.0 sphinx-3.1.2 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-autoprogram-0.1.5 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 sqlalchemy-1.2.19 update-checker-0.17 urllib3-1.22 vcrpy-1.11.1 wrapt-1.12.1 xmltodict-0.11.0
+Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-3.13 Pygments-2.6.1 alabaster-0.7.12 babel-2.8.0 certifi-2020.6.20 chardet-3.0.4 coverage-5.2.1 coveralls-2.1.1 dnspython-1.15.0 docopt-0.6.2 docutils-0.16 enum34-1.1.10 flake8-3.5.0 flake8-coding-1.3.2 flake8-future-import-0.4.5 flake8-import-order-0.18.1 geoip2-2.9.0 idna-2.10 imagesize-1.2.0 maxminddb-2.0.1 mccabe-0.6.1 packaging-20.4 praw-5.4.0 prawcore-0.14.0 py-1.9.0 pycodestyle-2.3.1 pyflakes-1.6.0 pyparsing-2.4.7 pytest-3.2.5 pytest-vcr-0.3.0 pytz-2020.1 requests-2.24.0 six-1.15.0 snowballstemmer-2.0.0 sphinx-3.1.2 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-autoprogram-0.1.5 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 sqlalchemy-1.2.19 update-checker-0.17 urllib3-1.22 vcrpy-1.11.1 wrapt-1.12.1 xmltodict-0.11.0

Travis CI - Test and Deploy with Confidence vs Travis CI - Test and Deploy with Confidence

-Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 Pygments-2.6.1 alabaster-0.7.12 babel-2.8.0 chardet-3.0.4 coverage-5.2 coveralls-2.1.1 dnspython-1.16.0 docopt-0.6.2 docutils-0.16 entrypoints-0.3 flake8-3.7.9 flake8-coding-1.3.2 flake8-future-import-0.4.5 flake8-import-order-0.18.1 geoip2-2.9.0 idna-2.10 imagesize-1.2.0 importlib-metadata-1.7.0 maxminddb-1.5.4 mccabe-0.6.1 multidict-4.7.6 packaging-20.4 pluggy-0.13.1 praw-5.4.0 prawcore-0.14.0 pycodestyle-2.5.0 pyflakes-2.1.1 pyparsing-2.4.7 pytest-4.6.11 pytest-vcr-1.0.2 pytz-2020.1 requests-2.24.0 snowballstemmer-2.0.0 sphinx-3.1.2 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-autoprogram-0.1.5 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 sqlalchemy-1.3.18 update-checker-0.17 urllib3-1.25.9 vcrpy-2.1.1 wcwidth-0.2.5 wrapt-1.12.1 xmltodict-0.12.0 yarl-1.3.0 zipp-1.2.0
+Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 Pygments-2.6.1 alabaster-0.7.12 babel-2.8.0 chardet-3.0.4 coverage-5.2.1 coveralls-2.1.1 dnspython-1.16.0 docopt-0.6.2 docutils-0.16 entrypoints-0.3 flake8-3.7.9 flake8-coding-1.3.2 flake8-future-import-0.4.5 flake8-import-order-0.18.1 geoip2-2.9.0 idna-2.10 imagesize-1.2.0 importlib-metadata-1.7.0 maxminddb-2.0.0 mccabe-0.6.1 multidict-4.7.6 packaging-20.4 pluggy-0.13.1 praw-5.4.0 prawcore-0.14.0 pycodestyle-2.5.0 pyflakes-2.1.1 pyparsing-2.4.7 pytest-4.6.11 pytest-vcr-1.0.2 pytz-2020.1 requests-2.24.0 snowballstemmer-2.0.0 sphinx-3.1.2 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-autoprogram-0.1.5 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 sqlalchemy-1.3.18 update-checker-0.17 urllib3-1.25.10 vcrpy-2.1.1 wcwidth-0.2.5 wrapt-1.12.1 xmltodict-0.12.0 yarl-1.3.0 zipp-1.2.0

Diffing the raw logs in WinMerge highlights the changed parts: coverage, maxminddb and urllib3.

3

Well, this gives cwd a whole new definition for me: “committing while drowsy”.

I did compare log output—and even used WinMerge to do so—but I must have copied the wrong log.

maxminddb is a sub-dependency of geoip2, which also uses a >= requirement specifier. Sigh.

Accidentally comparing the wrong logs is embarrassing :sweat_smile: :joy_cat: but I’m glad it turned out to be something fairly trivial. We just have to explicitly cap maxminddb's version ourselves, since the newest major version dropped support for older Python releases that we still support until our next major version.

1 Like
4

The problem is, it doesn’t specify so in its metadata which is a bug. If it did, Pip woudn’t have picked it.

I see that this release is yanked. But Pip only recognizes yanked releases since 19.2. So you need to upgrade Pip to prevent this from happening in the future.