This is exactly what happens, except it’s not an issue, it’s the intended behavior: your script is not supposed to terminate the active shell (or rather, Travis’ build logic is currently incapable of continuing normally if it does and no-one knows how to fix that).
Your scripts’ commands are executed in the same shell that runs all the build logic, with eval – this is required so that your changes to execution environment persist. The build logic relies on the internal function containing the above-linked code to return normally every time to be able to do its work.
As a workaround, you can trap 'travis_terminate 1' ERR (this is an internal function that the build logic calls at the end and on some fatal errors to flush the remaining logs and correctly terminate the logging facility). But since this is internal functionality, there are no guarantees that it works in every case or will continue working for any amount of time. In any case, no build logic that normally runs after a build error (e.g. saving the cache) will fire.