Allow a next-stage job read-only access to the cache of a previous-stage one; or make exported/imported artifacts



The are a whole bunch of feature requests for passing artifacts between build stages that @BanzaiMan kindly outlined in What is the cache ID for a job generated from? How can staged jobs use the same cache? .

According to him, the stumbling block is inevitable cache corruption if multiple concurrent jobs use the same cache.

But that is only a problem if those jobs all write to that cache. If they only read from it but write to their own caches, there is no problem.

So, a (rather straightforward) solution seems to be to allow a job to specify:

  • A previous job whose cache it wishes to import
    (How exactly it can be specified to harmonize with the build matrix is another question; see below)
  • Optionally, which files/directories from it should be imported

Alternatively, since this is not so much caching but rather passing build artifacts, the existing artifacts mechanism (hxxps:// can be (re)used.

How to specify the previous job (about this, I’m not yet quite sure):

  • The linked discussions showed that the primary use case is to prebuild some dependencies once, that many jobs will then be using – i.e. a one preceding – many succeeding jobs relationship.
  • So, with the build matrix, the corresponding preceding job is going to be found by removing a few succeeding job’s parameters. Or, equivalently, only taking a few of its job parameters ignoring the rest. The specification should thus be these exclusions or inclusions.
    • If the preceding job’s “cache profile” (the parameters that affect cache ID) is not a strict subset, it may be allowed to specify additional parameters to add to “search terms”. Though I suspect such cases can instead be solved by hiding such discrepancies into scripts.

New Build Stages