A workaround is suggested here: Permission issue while building wheels for various python packages
You can put this into before_install; you can see how I applied the workaround for a project I work on here: https://github.com/gap-system/gap/blob/5d2fe4aad4721b03a24dd47af239b27765c49f19/.travis.yml#L163
Sadly, the arm64/s390x/ppc64le “support” in Travis seems to be essentially unsupported resp. unmaintained… Plenty of issues are reported, but they are not even acknowledge, not to mention fixed, and that’s been going on for a couple of months now.