Fine grained caching, caching full project for next steps

#1

Hello,

We’re trying to speed up our build pipeline. Currently I’m playing around in my private fork, but there shouldn’t be a difference to the open source derivate.

This is one of the builds, almost finished, just need to get that Cypress binary cache fixed.
https://travis-ci.org/maxmarkus/luigi/jobs/569241939

Currently we’re building our monorepo once and reuse in all of the stages, which is way faster then building and downloading eg 5 times (even when sometimes only documentation binaries are required, it’s still faster using the cached stuff).

cache: 
  directories:
    - $TRAVIS_BUILD_DIR
    - /home/travis/.cache/Cypress

What I recognized is that it now - when using $TRAVIS_BUILD_DIR , it also caches .npm/_logs/ folder, which triggers an archive update after every build in the last 5 steps - this would not be necessary.

I would like to somehow only cache the project folder, with everything included (also node_modules and dot files), additionally also the Cypress binary from the travis cache. This does not seem to get cached, alltough I added the second cache dir.

Maybe someone can give me a hint on how to improve or what are the best practices cache-wise.

Best, Markus

#2

I don’t know what kind of directory structure a Node build produces so giving general considerations.

  • The source code is git clone'd in every job, so there’s no use caching it. Moreover, overwriting it with cached data will cause a mess because you will not be actually building the code in your commit.

  • There’s no way to exclude files and subfolders but you can rm them in the before_cache: script so that only what you want to cache is left in the corresponding subtrees.

    • Files under Git control can be listed with git ls-files.
    • But unless your build artifacts’ locations are extremely numerous and dynamic, it’s going to be less hassle to just cache the specific subtrees they are in.

  • Caching all intermediate build files is prone to infinitely accumulating redundant data as you rename or delete source files and the artifacts they produce. ccache is designed for such a use case by having a preset max cache size and discarding stuff on the LRU principle (and also keeping its cache away from your build tree so that any orphans are not left dangling there). Whatever you use, you are also going to need some kind of orphan control if you are going this way.

#3

In a typical setup you would only cache ~/.npm to prevent re-downloading, but still run npm ci for your project, which would install and sometimes compile stuff in order to be able to build the project. npm ci downloads everything to node_modules folder and compiles if required (eg node-sass, node-gyp always gets compiled upon install), which takes 3-5 minutes on travis.

This is a monorepo with 3 node based projects, which means 3 times npm ci, so compiling once and caching everything is kind of crucial, or at least saves lots of time in our case.

The project structure is similar to:

  • /client
  • /core
  • /example/angular (uses core and client)
  • /docs (not directly part of the project, but verified with some other node based tools)

Once everything is installed with lerna bootstrap -ci, core and client of our framework are bundled to be ready to be used in the example.
The travis jobs do things like running unit tests in core, running e2e tests in example, verifying documentation. In general (except of documentation), always the full project cache including binaries is required.
Else we would need to either install or bundle the application again.

My goal was to install, bundle and compile once and reuse in the following steps.
In one of the last runs now I was able to (npm publish is excluded now too) reduce from over 15 minutes to around 10 minutes (https://travis-ci.org/maxmarkus/luigi/builds/569345890) … which will even get better the more e2e tests we have, as they are now running in parallel with 4 instances.

I followed your hin with before_cache to remove some log files and stuff, to prevent cache update.
Cache rehydration takes around 1 1/2 minutes, which is quite fast I’d say. E2E still needs to run the Angular application tough and start Cypress, but also this one starts after 2-3 Minutes with e2e testing.

Looks like I was still on the right track?

#4

Caching the entire $TRAVIS_BUILD_DIR violates the 1st bullet point.

Imprint