The problem we’re seeing is as follows:
- We have multiple build stages.
- Lets say in first stage we are building our code and pushing artifacts out. In second stage - we are deploying and running FVT.
- Both core code and FVT are based on the same git repo and should be in lockstep, i.e. FVT tests should be compatible with the compiled code.
- What we’re seeing is build stage checks out merge preview with the state of master at the beginning of the stage.
- In the middle of that stage, master gets updated. Since travis always looks at the tip of master - the FVT stage now checks out the merge preview with the NEW head of master.
- In the case where new head of master contains new FVT tests that are incompatible with the code compiled in build stage - we get a build failure.
This is a completely illegitimate failure, since its essentially a mismatch between compiled code and FVT code. What should be happening - the code tested across stages should be the same code, same merge preview, otherwise this introduces all sorts of instabilities into the build.
We can work around this behaviour with Git tags and all, but it seems like we’re creating functionality that should be in core product.