Are there reasons against deactivating the virtual environment?

The Travis machine is not a productive environment. So there is no high risk to destroy something.

I think about if it might be OK to deactivate the virtual environment and run Python and my build-install-test stuff on the whole system.

What I don’t know if I could use Python interpreters outside the virtual environment that are not shipped with Ubuntu.

You could do:

before_install:
  - sudo apt-get update
  - sudo apt-get install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev wget

install:
  - wget https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tgz
  - tar xzf Python-3.9.1.tgz
  - cd Python-3.9.1
  - ./configure --enable-optimizations
  - make
  - sudo make altinstall  # Use altinstall to avoid replacing the default python binary

This is a more manual process, but you won’t then need to use pyenv.