C/C++ vast build performance differences between Linux and OSX?

I’m building/testing a fairly large C/C++ package on travis across multiple compiler versions on both Linux and OSX. I’ve done my best to make sure the build uses ccache properly and enabled caching on Travis which seems to be working.

However, what I’m seeing between OSX and Linux is very different with a “warm” cache:

On Linux, we see still see build times > 1hr, while a warm cache on OSX cuts the build to ~15 minutes. I can’t see anything wrong my settings so I’m at a bit of loss now as to what’s happening. Can anyone help?


-- Check for working CXX compiler: /usr/bin/g++-5

Means that you are not using CCache.

Maybe your many compiler packages and envvar assignments has confused CCache installer and/or CMake.

  • Check the state of the system at the time of CMake invocation: relevant paths, envvars.

  • If the error is in CMake invocation, you can run CMake in trace mode with cmake --trace-expand (lots of output!) to pinpoint the exact place in its logic where it goes wrong.

  • If the error is before it, running Bash code with xtrace as per https://wiki-dev.bash-hackers.org/scripting/debuggingtips#making_xtrace_more_useful also proved invaluable for me to see wrong values and trace them back to their origin straight away, without rerunning anything.

I’ve implemented ccache via the LAUNCHER options on CMake:

I added some instrumentation to the build here to check if ccache is getting filled/used, it appears it is:

At this point, I believe the issue is the build isn’t using GOLD for its linker, and since its C++, thats a huge amount of the time spent. I have fixed up the linker config and tested it locally, and I believe that’s sorted. Waiting on this:

This will make it trickier to see if you are actually using CCache. Overriding compilers on PATH would be enough, really.
You can run ccache -s at the end then to see how much it was actually used.

If that’s the case, there should probably be no problem if using the same toolchain as in OSX.

Confirmed now that GOLD wasn’t getting used in Linux, greatly increasing linking time (there is only one linker available on OSX, and its fast)