Use build stages. Then save each artifact to both cache (to avoid rebuilding each time) and a workspace (for other stages to use).
Also keep in mind that build VMs are 2-core – so you should be able to speed up things with the likes of make -j2.
Alternatively and preferably, depending on your build system, you should be able to find prebuilt artifacts for common libraries. See e.g. How do I use mingw and msys2 shells?. A Windows builder VM also contains plenty of prebuilt libraries, too, as can be seen from a Windows VM snapshot.