Currently Caching Dependencies and Directories - Travis CI says:
There is one cache per branch and language version / compiler version / JDK version / Gemfile location, etc.
This is too vague as it doesn’t give any way to determine if two jobs will be using the same or a different cache.
https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/shared/directory_cache/base.rb#L236-L253 and https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script.rb#L147-L157 suggest that:
In the cache path, which looks like master/cache-osx-xcode8-88897247c8594340a4240d868adfe866078e00d71e4ad6a77ca222c2d9970d84.tgz,
the long hash is the hash of all environment variables specified in the .yml (they can be seen at the start of a build log ) except the secret ones.
So,
The only way to reuse another job’s cache is to use the same exact system image and environment variables
Is all this right?
Does this mean that I cannot make a “shared” cache for multiple jobs? In particular, I’m looking into making a source distribution, then building it on all supported platforms.
Further down in the document , there is a more comprehensive explanation of how cache names are determined. I will not repeat what’s written there.
Does this mean that I cannot make a “shared” cache for multiple jobs?
That is correct. If two jobs differ in any of the factors mentioned, they do not share the cache. The reason here is that, when multiple jobs share the same cache name, the cache can corrupt due to simultaneous access and it can cause a lot of problems.
opened 09:20AM - 07 Apr 17 UTC
caching
important
Is it possible to use the same cache for all builds of a job?
We have a lot o… f dependencies and so we decided to create an extra job which build the dependencies and store it in the cache.
All other (following) builds of that job should use the dependencies in the cache.
opened 07:44AM - 02 Jun 17 UTC
closed 08:04PM - 13 Nov 18 UTC
feature-request
build stages
locked
important
We want to achieve a 2 stage CI:
* A `build` stage that compiles a program. 2 … builds: mac and linux.
* A `test` stage that use the compiled program. Multiple builds: 2 OSes * number of different language versions (e.g. `python: 2.7` and `python: 3.5`).
It is hard to pass the compiled program to the `test` stage. 2 approaches I thought of:
* use Travis's caching: Not feasible since the `language:` / `python:` keys are different between the `build` stage and the `test` stage. The cache wouldn't be shared according to https://docs.travis-ci.com/user/caching#Caches-and-build-matrices. Don't even consider what would happen when there are parallel builds of different commits, or when one restart a build of an older commit.
* use custom S3: It can work in our own repo, however, without exposing our S3 credentials, our forks wouldn't be able to use CI (unless they override the S3 credentials with their own).
It would be great if there is some ad-hoc feature for passing build artifacts across stages.
For reference, the GitLab CI artifacts is pretty perfect: https://docs.gitlab.com/ee/ci/yaml/#artifacts
Their `artifacts` from all previous stages are passed by default, but also can be fine-tuned with `dependencies`.
opened 07:51AM - 26 Aug 18 UTC
closed 01:24AM - 01 Dec 18 UTC
feature-request
caching
I'd like to be able to exclude some `env` vars from the travis cache-name digest… . I'm not asking for cache shared between jobs with different env -- although this feature would make that possible, it's up to the user to make sure the remaining non-excluded env vars result in a unique cache name.
For example when we're caching a `ccache` directory and nothing else, it only contains compiled object files. So it makes sense for the cache-name to depend on `CC`, `CFLAGS`, etc., but not other env vars that have no effect on what goes into the ccache (e.g. `LDFLAGS`). Sometimes we use extra env vars to enable/disable additional post-compilation steps in selected jobs, changing these also unnecessarily invalidates the cache.
I was thinking of adding a `cache_friendly_env` key, or `cache_friendly` under `env`, or some magic prefix to variable name that'd be stripped away before passing to bash, but I think the least intrusive (and less error-prone) way will be to simply list variables that the digest should ignore in cache config:
```yaml
cache:
directories: [ "$HOME/.ccache" ]
env_ignore: [ MAKEFLAGS, LDFLAGS ]
```
all aim to address this issue to varying degrees. We have ideas about how to strike a balance (useful caching strategy while keeping the likelihood of cache corruption to a minimum), but the implementation would still need to be discussed and the actual work prioritized.
1 Like
Thanks! I trusted the info to be complete because this is reference documentation.Would you mind adding a reference?