This issue has been reported many times before, but Travis seems to have abandoned their multiarch support (other than for PR on their blog). For a workaround and a list of previous threads mentioning this, see e.g. here: Travis Build gets stalled for ARM64 job in pandas package