When configuration validation is on, environment variable definition order may change

Environments Linux
1

Originally reported at https://travis-ci.community/t/environment-variable-solr-ver-is-now-empty/6932 but since I’ve learned more about this issue and I can’t edit the old post

I have the following line in my .travis.yml (https://travis-ci.com/wodby/base-solr/jobs/278196558):

language: bash

env:
  matrix:
  - SOLR_VER="8.4.1" TAGS="${SOLR_VER},8.4,8,latest"

Then I print:

script:
- echo "${SOLR_VER}"
- echo "${TAGS}"

And it shows me:

8.4.1
,8.4,8,latest

As you can see, for some reason when I use a variable as a part of another variable (with bash best practices) it cannot be recognized and instead, we have an empty string here.

So I decided to test whether it’s a generic issue or not and added:

env:
  matrix:
  - TESTABC="1.2.3" TESTZXC="${TESTABC},4,5"

And to my surprise it worked fine:

1.2.3
1.2.3,4,5

Then I tried to add an underscore to the variable name:

env:
  matrix:
  - TESTABC_ASD="1.2.3" TESTZXC="${TESTABC_ASD},4,5"

And the issue reproduced:

1.2.3
,4,5

This build worked fine before with pretty much the same lines and stopped working a couple of days ago. So my guess is the bash was updated recently on .com version and the behavior changed (probably a bug). I have a similar variable definition in another build on .org version of travis and it’s working fine https://travis-ci.org/wodby/base-php/jobs/639255533

1 Like
2

The use of underscores in your environment variable names has nothing to do with the issue you are seeing.

The issue is the order in which they are currently evaluated (that is, TAGS before SOLR_VER, which TAGS depends on). This may be due to the recent introduction of the configuration validator (wodby/base-php does not use the validator, it appears), and we are looking into it.

Thanks.

3

We’ve tracked this to the data type we are using to store configuration data. Seems that the best solution would be to change the database schema, but doing so will take some time in planning and execution.

In the meantime, there are a few possible solutions:

  1. Turn off configuration validation in https://travis-ci.com/wodby/base-solr/settings
  2. Change environment variable names so that the the ones that depend on others are LONGER than the ones that don’t.
  3. Defer defining dependent environment variables to before_install, and use export DEPENDENT=${FOO}_XYZ.

Please note that not all of them may be applicable to your particular use case.

Thank you so much for reporting this issue.

1 Like
4

See https://github.com/wodby/base-solr/blob/master/.travis.yml#L6

env:
  matrix:
  - SOLR_VER="8.4.1" TAGS="${SOLR_VER},8.4,8,latest" TESTABC_ASD="1.2.3" TESTZXC="${TESTABC_ASD},4,5"

as you can see I defined SOLR_VER before TAGS. The order of variables definition is what matters not the order how I print them, right?

And it worked fine just a few days ago.

That sounds reaaaally weird. It doesn’t sound like something solid I can rely on.

Well, I guess you did change something recently and changing it back won’t be quick. I think it would really helpful if you disclose this issue, I believe many people like me spend time trying to understand why their builds broke.

Thank you so much for replying to this issue, I thought I was going nuts.

5

In theory, yes. But, as I said, there is an issue with the way we process the configuration to compile the build script, and the order changes. If you click on the TAGS and SOLR_VERS, you see that the evaluation order changed when the configuration validator is on.

6

We are rolling out the configuration validator to more and more repositories.

7

Well, I guess you did change something recently and changing it back won’t be quick. I think it would really helpful if you disclose this issue, I believe many people like me spend time trying to understand why their builds broke.

The issue you have encountered is a combination of a couple different things:

  • The feature Build Config Validation is being rolled out automatically to existing repositories, so your repository was picked up recently.
  • The feature parses env vars that are given as strings into env vars represented as hashes (the reasoning behind this is a long story by itself, but it is related to long standing feature requests, in short: hashes are more processable).
  • The build config is stored in our database in a column with the type jsonb which re-orders hashes by key length.
  • Thus the env vars in your build config end up being reordered by their name’s lengths, which is wrong.

We are working on moving the respective database columns from from jsonb to json.

1 Like
8

Thank you for all the detailed explanations. Disabling build validation resolved my issue.