"pyenv global 3.7" now fails on Xenial (it was working 3 days ago)

Environments Linux
1

In my xenial config, I call pyenv global 3.7 to set a Python 3.7 environment, see:

This has always worked for the past few years, here is a successful build from 3 days ago:

https://travis-ci.com/github/vgc/vgc/jobs/363245525

However, it now fails, see:

https://travis-ci.com/github/vgc/vgc/jobs/364662456

Did anything change? I don’t see anything related in the docs. Thanks!

2
1 Like
3

Thanks a lot, this is very helpful. It seems a strange decision that the now-preinstalled Python 3 version (3.5) is lower than before (3.6 and 3.7), but I’m sure there are reasons, and at least that explains the problem. I guess I’ll have to install Python 3.7 as part of the ‘install’ script. I’ll report back when I have something working.

4

Btw you can cache the resulting Python installation as per

1 Like
5

After a lot of trial and error, including trying to install Python 3.7 via the deadsnakes PPA rather than pyenv (see https://stackoverflow.com/a/37660551/1951907), what worked in my case was to keep using pyenv, but I had to update pyenv first. Here are my steps:

  install:
    - PYTHON_XY=python3.7
    - PYTHON_VERSION=3.7.8
    - pushd /opt/pyenv
    - git branch --set-upstream-to=origin/master deploy
    - git pull --ff-only
    - git checkout v1.2.20
    - pyenv install -v ${PYTHON_VERSION}
    - popd
  before_script:
    - pyenv global ${PYTHON_VERSION}
    - CMAKE_EXTRA_ARGS+=" -DPYTHON_EXECUTABLE=$(python-config --prefix)/bin/${PYTHON_XY}"
    - CMAKE_EXTRA_ARGS+=" -DPYTHON_LIBRARY=$(python-config --prefix)/lib/lib${PYTHON_XY}m.so"
    - CMAKE_EXTRA_ARGS+=" -DPYTHON_INCLUDE_DIR=$(python-config --prefix)/include/${PYTHON_XY}m"
  script:
    - cmake .. -DCMAKE_BUILD_TYPE=Release ${CMAKE_EXTRA_ARGS}
    - make

Note that this compiles Python from source and takes a few minutes. I haven’t tried caching yet.

1 Like
6

See https://github.com/pyenv/pyenv#upgrading for the official way to update pyenv.
E.g. you can save an extra checkout with fetch instead of pull, and I don’t see a use for git branch if you checkout something else afterwards anyway.

As an aside, specifying -DPYTHON_EXECUTABLE is enough for CMake to use a Python installation IIRC, and if you pyenv global it before that, it’s going to be at which python (the shim) or pyenv which python (the real executable), no need to hardcode.

7

Thanks for the added info. Using fetch instead of pull seems like a better idea indeed. The branch call was suggested by git after git pull complained that the current branch didn’t have an upstream. I’ll remove it if I use fetch.

Regarding the explicit -DPYTHON_EXECUTABLE, -DPYTHON_LIBRARY, and -DPYTHON_INCLUDE_DIR, I remember having problems in the past if I didn’t specify all of them, reason why I now always do it like this at least in CI builds, since I don’t have to type it each time anyway. I think the main problem is that even if CMake finds the correct -DPYTHON_EXECUTABLE thanks to pyenv global, it still sometimes fails to detect the correct -DPYTHON_LIBRARY, because two python installations might actually have the same --prefix. Zen of Python: Explicit is better than implicit :wink:

8

I am now using fetch instead of pull for updating pyenv as recommended. Here is my current solution:

install:
  - PYENV_VERSION=1.2.20
  - PYTHON_VERSION=3.7.8
  - pushd $(pyenv root)
  - git fetch
  - git checkout v$PYENV_VERSION
  - pyenv install -v $PYTHON_VERSION
  - popd
before_script:
  - pyenv global $PYTHON_VERSION
  - PYTHON_PREFIX=$(python-config --prefix)
  - PYTHON_XY=python${PYTHON_VERSION%.*} # X.Y.Z -> pythonX.Y
  - CMAKE_EXTRA_ARGS+=" -DPYTHON_EXECUTABLE=$PYTHON_PREFIX/bin/$PYTHON_XY"
  - CMAKE_EXTRA_ARGS+=" -DPYTHON_LIBRARY=$PYTHON_PREFIX/lib/lib${PYTHON_XY}m.so"
  - CMAKE_EXTRA_ARGS+=" -DPYTHON_INCLUDE_DIR=$PYTHON_PREFIX/include/${PYTHON_XY}m"
script:
  - cmake .. -DCMAKE_BUILD_TYPE=Release ${CMAKE_EXTRA_ARGS}
  - make
1 Like