Experience report: Sharing complex build configuration

Travis CI Discussions & Feedback
,
1

Encoding complex build steps within .travis.yml is a real pain.

For simple setups one can certainly write a short one-liner in the before_install section (or whichever phase fits best), but the moment these get longer than a “if X; then Y; else Z;” the problems begin. The documentation rightfully suggests to move complex scripts into files, which would be delivered together with the .travis.yml file.
The down-side of this approach is of course that one now needs to remember that the build configuration is not just a single file. This can be solved by using naming conventions (a .travis directory, for example).

Unfortunately this breaks when trying to avoid duplication of build configuration and the mess that comes with keeping these duplicates in-sync. Config importing allows to share the .travis.yml file (snippets) from another repository, but there is no way to say “and also please take these scripts and put them over here”.

I tried yesterday:

  1. “Somehow” do a svn cat/git cat-file/… from the configuration repository. This fails because authentication.
  2. “Somehow” use a here-document in a before_install step that produces a script. This fails because Travis seems to do weird things for each of these lines, so a here-document didn’t work (the end-marker ended up in the file???). The quoting is also odd, it seems one needs to escape ‘$’ in the before_install steps? From errors I fear there is some eval going on there :slight_smile:
  3. Try to encode the script as a multiline yaml environment variable. This breaks in really funny ways: A regular yaml parser showed that the value was correct, but travis nonetheless managed to split the value on line-breaks and then did an export for each of the lines of my script.

I ended up giving up, and added a manual step into my shared repository: Whenever I modify a script, I have to run a conversion that produces a base64 encoded form of the script, and then encodes that into a template .yml file that travis can import and that dumps the script.

#!/bin/sh

mkdir -p generated
for script in scripts/*.sh; do
    short_name=$(basename "${script}" .sh)
    name=$(echo "${short_name}" | tr "[:lower:]/-" "[:upper:]__")
    encoded=$(base64 < "${script}")
    cat <<EOF >"generated/${short_name}.yml"
#
# Generated for ${script}
#
before_install:
- echo "\${${name}}" | base64 -d > "\${HOME}/.local/bin/${short_name}"
- chmod +x "\${HOME}/.local/bin/${short_name}"
env:
  global:
  - ${name}=${encoded}
EOF
done

Note that I have to do this manually: Travis’ import feature pulls from the source repository, which smells like it is repeating the mistake of bower and other systems that don’t understand the different between “sources” and “generated content”. The only way to automate this from what I can see would be either to distribute pre-commit hooks to developers of the configuration repository (which is manual!), or to have a Travis job on the configuration repository which then publishes the generated content into a different branch of the repository. Likely this will produce more mess …

What I would love to have:

  1. Allow me to create files directly from travis.yml:
    files:
    • target: /tmp/foo.sh
      mode: 0777
      source: |-
      #!/bin/sh
      my script here
  2. Document how generated artifacts would work for travis imports so at least not every developer has to reinvent the wheel
  3. Document that imports need to have a .yml extension (without that I got odd errors, didn’t capture them though)

Other than that: The imports is definitely a step in the right direction, and it ultimately allowed me to clean up about 40 repositories with duplicated-but-slightly different build configurations of ~100 lines, and reduce that to a rather simple “import:” + “env:” structure.
Thanks for that!

3 Likes
2

++1

3

Have you tried using git submodules for this purpose, and if so, what problems did you encounter?

4

git submodules

I have considered that, and on the surface it would probably work, but produces a more complex setup, as I now need to make sure to actually work with submodules (that’s training effort), and I need tooling to be able to handle this.

Using git submodules as I understand it would mean that each of my repositories needs to be configured to include the build repository.

The simple thing first: I’m not sure whether Travis handles submodules automatically. If not I would have to build another shared config to handle it, which might be doable (but don’t I then hit the same issues with credentials? Maybe.)

The bigger issue I see right now: What about updating? I end up having to go over each repo, and manually/script the update of the submodule. That’s not better than before shared configurations, where I had to do magic with git am, patch (and a lot of wiggle-ing) to keep my configurations roughly in sync.

A bit of context on the setup: My ~40ish repositories that this experience affected are independent from each other, and contain a TypeScript/NodeJS microservice that builds into a docker image. I generate these through a yeoman generator, which implements our best practices.
The things the developer has to do is

  • git init locally and commit
  • configure .travis.yml environment variables (encrypted credentials I cannot produce through the generator right now)
  • make a github repo and get the changes to git.

Not all these repositories get continuously modified, it is quite likely that any specific repository doesn’t see functional changes for weeks. I want to keep the non-functional changes down to a minimum.

Writing this all down: maybe git submodules is a viable idea, and I’m just scared that this will go wrong with my developers. I have touched submodules a few times before, and each time determined that the effort to work with them is not worth it.

5

Can I ask how many files you need to share, @ankon? And how big are they?

We share a simple db schema across a couple repos and simply curl GitHub’s raw content API https://github.com/travis-ci/build-configs/blob/postgres-9.6/db-setup.yml#L6. It seems for most cases this would be enough. Except if you have to share a large number of large files?

As for creating files in your .travis.yml configuration, this should work: https://travis-ci.org/github/svenfuchs/test/jobs/667412152/config … I admit it’s not very pretty, but that’s why we generally recommend moving such executables into files in your repo (again, if you want to share them, there are several ways to do that, e.g. curl, git, …)

As for credentials needed by an imported build configuration, you can share those in the imported config. They will be available on the importing repo, granted those repos are owned by the same user or organization.

6

How many and how much?

Right now: A single shell script – a somewhat trivial wrapper around docker push that also triggers another part of our build pipeline.

curl-ing the raw API

Interesting idea, might work (assuming again authentication works, as the file is in a private repo!), I’ll try that :slight_smile:

… this should work

Interestingly enough your example shows the odd problem with here documents:

$ cat << EOF | sed 's/ //' > foo
#/bin/bash
echo bar
/home/travis/.travis/functions: line 111: warning: here-document at line 109 delimited by end-of-file (wanted `EOF')

Try adding the EOF marker – does that work? Or does the marker end up in the file?
For me: It ended up in the file then.

As for credentials needed by an imported build configuration, you can share those in the imported config. They will be available on the importing repo, granted those repos are owned by the same user or organization.

I’m interested here: Assuming repo A imports from repo B, would I then encrypt the credentials for repo B, and store them in the file in repo B?

7

assuming again authentication works, as the file is in a private repo

sure, if it’s a private repo you’ll need to authenticate it. check GitHub’s documentation on how to use curl with an authorization header. you can try that command on your local machine until it works, then encrypt your token, store it alongside the curl command in the imported config.

Try adding the EOF marker – does that work?

no, it doesn’t because YAML will keep those extra spaces in the beginning. there’s no feature in YAML for removing leading whitespace. but that’s not our fault :slight_smile:

if you can live with the bash warning though it should work quite fine.

Assuming repo A imports from repo B, would I then encrypt the credentials for repo B, and store them in the file in repo B?

yes, precisely. there are some rules on what can be imported from where to where (check our docs on build config imports about that), but if both of your repos are private, and both owned by the same owner account then you’ll be fine.

also, … not that we recommend this, but we have many customers doing this. if all of these repos are private then you could get away with just hardcoding your token to the imported configs, too. again, it’s not something we recommend.

8

Assuming repo A imports from repo B, would I then encrypt the credentials for repo B, and store them in the file in repo B?

yes, precisely. there are some rules on what can be imported from where to where (check our docs on build config imports about that), but if both of your repos are private, and both owned by the same owner account then you’ll be fine.

I found the rules, and tried this, but the results were “meh”. The TL;DR first: I coouldn’t get it to work, and I couldn’t find out why either.

Structure of my shared repo travis-configs:

service.yml
  imports node-docker-image.yml (deep_merge_prepend)
    imports includes/nodejs.yml (deep_merge_prepend)
    imports includes/ecr.yml (deep_merge_prepend)
    imports generated/travis-deploy.yml (deep_merge_prepend; this is the script-magic mentioned above)

I want to store a NPM_TOKEN encrypted in nodejs.yml, and a set of AWS credentials for ECR access in ecr.yml. Right now these are all over the place in the other repos, stored in the same pattern:

  1. travis encrypt FOO=bar
  2. Add the secure: ... output to env.global.
    I never use travis encrypt --add is as it foobars the formatting of the file.

What I therefore tried is do the same steps, and add the env.global credentials to the imported files. I then removed the environment variables from my actual build files, which makes things definitely look great:

version: ~> 1.0

import:
- source: .../travis-configs:service.yml
  mode: deep_merge_prepend

env:
  global:
    SERVICE_NAME=service

The only issue is: It doesn’t work. Somehow travis doesn’t seem to be able to decrypt the variables – I once saw it working (on a test branch, with explicit ‘@’-commit-refs for the imports), but wasn’t able to reproduce it at all.
Travis doesn’t tell me anything, looking at the build config validation tab shows the variables as ‘[secure]’ (ok) or in the build logs as export gibberish (very much not ok).

The repos are both private, owned by the same organization.

The biggest issue here is actually that I have nothing to go with to figure out whether I’m doing something wrong, or something else is unhappy.