The compilation and test of the projects usually take seconds locally, but may take several minutes on Travis CI. Most of the time is used on installing Mono every time. It seems wasteful of the build server resources.
I think this is by design. Travis CI uses containerized builds, meaning each build of your project gets a fresh image of some sort of C# (.NET Core) container, and it’s the deployment/startup of the container that takes so long.
The upside is that your builds are isolated from each other (and from everyone else’s builds/environments).