You don’t. “System site-packages” is only a thing for Python versions provided by distro – which for Xenial are 2.7.12 and 3.5.1. Other Python versions are installed manually (into /opt as of this writing), and the only global packages there are pip and setuptools (you can see all this by examining the Python tarball that the build log refers to).
So if you’re using alternative Python versions, you don’t have access to any apt-get Python packages anyway.
As such, the most straightforward solution would be to install all Python packages you need with pip (or otherwise) into the virtualenv – regardless of whether apt also provides them for distro-supplied versions – and be done with it.
The fact that using system_site_packages: true limits you to distro-provided Python versions is documented, so it’s not a bug, either:
Building a Python Project - Travis CI :
If you decide to use apt anyway, note that for compatibility reasons, you’ll only be able to use the default Python versions that are available in Ubuntu (e.g. for Trusty, this means 2.7.6 and 3.4.3).