Did Ruby upgrade on our build servers?

Environments
#1

Yesterday, our Drupal 8 Build processes started failing with these distinct errors:
…
/home/travis/.rvm/rubies/ruby-2.4.5/lib/ruby/site_ruby/2.4.0/rubygems.rb:284:in `find_spec_for_exe’: Could not find ‘bundler’ (2.2.1) required by your /home/travis/build/scotts-ebiz/smg-drupal/docroot/themes/custom/scotts_base/Gemfile.lock. (Gem::GemNotFoundException)

To update to the latest version installed on your system, run bundle update --bundler.
To install the missing version, run gem install bundler:2.2.1
…

To get around this error, we added the explicit ‘gem install’ call to our build scripts.
We never had to add this explicit call before yesterday morning around 10am ET.
Our Builds had been running cleanly without this call, for the past few years.

Any insight would be helpful, but it appears Ruby was upgraded/changed.

#2

Can’t say anything without seeing an old and a new build.

Under the “Build system information” fold near the start of the log, there should be the build system image version and the list of preinstalled software.

#3

Here are the “Build System Information” for 3 builds: Successful build Prior, Failed build, Successful build after workaround.

2020-12-15 Successful Master build - prior Ruby change?

Build system information
Build language: php
Build group: edge
Build dist: xenial
Build id: 208938642
Job id: 460674345
Runtime kernel version: 4.15.0-1088-gcp
travis-build version: 53b53106
Build image provisioning date and time
Wed Dec 9 07:47:53 UTC 2020
Operating System Details
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial
Systemd Version
systemd 229
Cookbooks Version
883ec55 https://github.com/travis-ci/travis-cookbooks/tree/883ec55
git version
git version 2.29.0
bash version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:11:02 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:09:05 2018
OS/Arch: linux/amd64
Experimental: false
clang version
clang version 7.0.0 (tags/RELEASE_700/final)
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.7.0
shfmt version
v2.6.3
ccache version
3.2.4
cmake version
cmake version 3.12.4
heroku version
heroku/7.47.4 linux-x64 node-v12.16.2
imagemagick version
Version: ImageMagick 6.8.9-9 Q16 x86_64 2019-11-12 http://www.imagemagick.org
md5deep version
4.4
mercurial version
version 4.8
mysql version
mysql Ver 14.14 Distrib 5.7.32, for Linux (x86_64) using EditLine wrapper
openssl version
OpenSSL 1.0.2g 1 Mar 2016
packer version
1.3.3
postgresql client version
psql (PostgreSQL) 10.15 (Ubuntu 10.15-1.pgdg16.04+1)
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
sudo version
1.8.16
gzip version
gzip 1.6
zip version
Zip 3.0
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 13 2020 16:04:38)
iptables version
iptables v1.6.0
curl version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
wget version
GNU Wget 1.17.1 built on linux-gnu.
rsync version
rsync version 3.1.1 protocol version 31
gimme version
v1.5.4
nvm version
0.37.2
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew - App::perlbrew/0.89
phpenv version
rbenv 1.1.2-40-g62d7798
rvm version
rvm 1.29.10 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.5.0
Installed Firefox version
firefox 63.0.1
MongoDB version
MongoDB 4.0.21
PhantomJS version
2.1.1
Pre-installed PostgreSQL versions
9.4.26
9.5.24
9.6.20
Redis version
redis-server 6.0.6
Pre-installed Go versions
1.11.1
ant version
Apache Ant™ version 1.9.6 compiled on July 20 2018
mvn version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
gradle version
Gradle 5.1.1!
lein version
Leiningen 2.9.5 on Java 11.0.2 OpenJDK 64-Bit Server VM
Pre-installed Node.js versions
v10.23.0
v11.0.0
v12.20.0
v14.15.1
v4.9.1
v6.17.1
v8.12.0
v8.17.0
v8.9
phpenv versions
system
5.6
5.6.40
7.1
7.1.27
7.2

  • 7.2.15 (set by /home/travis/.phpenv/version)
    hhvm
    hhvm-stable
    composer --version
    Composer version 1.8.4 2019-02-11 10:52:10
    Pre-installed Ruby versions
    ruby-2.3.8
    ruby-2.4.5
    ruby-2.5.3

2020-12-16 UnSuccessful Master build - after Ruby change?

Build system information
Build language: php
Build group: edge
Build dist: xenial
Build id: 209178387
Job id: 461347299
Runtime kernel version: 4.15.0-1090-gcp
travis-build version: 53b53106
Build image provisioning date and time
Wed Dec 16 07:32:43 UTC 2020
Operating System Details
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial
Systemd Version
systemd 229
Cookbooks Version
883ec55 https://github.com/travis-ci/travis-cookbooks/tree/883ec55
git version
git version 2.29.0
bash version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:11:02 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:09:05 2018
OS/Arch: linux/amd64
Experimental: false
clang version
clang version 7.0.0 (tags/RELEASE_700/final)
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.7.0
shfmt version
v2.6.3
ccache version
3.2.4
cmake version
cmake version 3.12.4
heroku version
heroku/7.47.5 linux-x64 node-v12.16.2
imagemagick version
Version: ImageMagick 6.8.9-9 Q16 x86_64 2020-12-04 http://www.imagemagick.org
md5deep version
4.4
mercurial version
version 4.8
mysql version
mysql Ver 14.14 Distrib 5.7.32, for Linux (x86_64) using EditLine wrapper
openssl version
OpenSSL 1.0.2g 1 Mar 2016
packer version
1.3.3
postgresql client version
psql (PostgreSQL) 10.15 (Ubuntu 10.15-1.pgdg16.04+1)
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
sudo version
1.8.16
gzip version
gzip 1.6
zip version
Zip 3.0
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 13 2020 16:04:38)
iptables version
iptables v1.6.0
curl version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
wget version
GNU Wget 1.17.1 built on linux-gnu.
rsync version
rsync version 3.1.1 protocol version 31
gimme version
v1.5.4
nvm version
0.37.2
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew - App::perlbrew/0.89
phpenv version
rbenv 1.1.2-40-g62d7798
rvm version
rvm 1.29.10 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.5.0
Installed Firefox version
firefox 63.0.1
MongoDB version
MongoDB 4.0.21
PhantomJS version
2.1.1
Pre-installed PostgreSQL versions
9.4.26
9.5.24
9.6.20
Redis version
redis-server 6.0.6
Pre-installed Go versions
1.11.1
ant version
Apache Ant™ version 1.9.6 compiled on July 20 2018
mvn version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
gradle version
Gradle 5.1.1!
lein version
Leiningen 2.9.5 on Java 11.0.2 OpenJDK 64-Bit Server VM
Pre-installed Node.js versions
v10.23.0
v11.0.0
v12.20.0
v14.15.2
v4.9.1
v6.17.1
v8.12.0
v8.17.0
v8.9
phpenv versions
system
5.6
5.6.40
7.1
7.1.27
7.2

  • 7.2.15 (set by /home/travis/.phpenv/version)
    hhvm
    hhvm-stable
    composer --version
    Composer version 1.8.4 2019-02-11 10:52:10
    Pre-installed Ruby versions
    ruby-2.3.8
    ruby-2.4.5
    ruby-2.5.3

2020-12-17 Successful Master build - after adding explicit Bundler version

Build system information
Build language: php
Build group: edge
Build dist: xenial
Build id: 209413101
Job id: 461885187
Runtime kernel version: 4.15.0-1090-gcp
travis-build version: 53b53106
Build image provisioning date and time
Wed Dec 16 07:32:43 UTC 2020
Operating System Details
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial
Systemd Version
systemd 229
Cookbooks Version
883ec55 https://github.com/travis-ci/travis-cookbooks/tree/883ec55
git version
git version 2.29.0
bash version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:11:02 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:09:05 2018
OS/Arch: linux/amd64
Experimental: false
clang version
clang version 7.0.0 (tags/RELEASE_700/final)
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.7.0
shfmt version
v2.6.3
ccache version
3.2.4
cmake version
cmake version 3.12.4
heroku version
heroku/7.47.5 linux-x64 node-v12.16.2
imagemagick version
Version: ImageMagick 6.8.9-9 Q16 x86_64 2020-12-04 http://www.imagemagick.org
md5deep version
4.4
mercurial version
version 4.8
mysql version
mysql Ver 14.14 Distrib 5.7.32, for Linux (x86_64) using EditLine wrapper
openssl version
OpenSSL 1.0.2g 1 Mar 2016
packer version
1.3.3
postgresql client version
psql (PostgreSQL) 10.15 (Ubuntu 10.15-1.pgdg16.04+1)
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
sudo version
1.8.16
gzip version
gzip 1.6
zip version
Zip 3.0
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 13 2020 16:04:38)
iptables version
iptables v1.6.0
curl version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
wget version
GNU Wget 1.17.1 built on linux-gnu.
rsync version
rsync version 3.1.1 protocol version 31
gimme version
v1.5.4
nvm version
0.37.2
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew - App::perlbrew/0.89
phpenv version
rbenv 1.1.2-40-g62d7798
rvm version
rvm 1.29.10 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.5.0
Installed Firefox version
firefox 63.0.1
MongoDB version
MongoDB 4.0.21
PhantomJS version
2.1.1
Pre-installed PostgreSQL versions
9.4.26
9.5.24
9.6.20
Redis version
redis-server 6.0.6
Pre-installed Go versions
1.11.1
ant version
Apache Ant™ version 1.9.6 compiled on July 20 2018
mvn version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
gradle version
Gradle 5.1.1!
lein version
Leiningen 2.9.5 on Java 11.0.2 OpenJDK 64-Bit Server VM
Pre-installed Node.js versions
v10.23.0
v11.0.0
v12.20.0
v14.15.2
v4.9.1
v6.17.1
v8.12.0
v8.17.0
v8.9
phpenv versions
system
5.6
5.6.40
7.1
7.1.27
7.2

  • 7.2.15 (set by /home/travis/.phpenv/version)
    hhvm
    hhvm-stable
    composer --version
    Composer version 1.8.4 2019-02-11 10:52:10
    Pre-installed Ruby versions
    ruby-2.3.8
    ruby-2.4.5
    ruby-2.5.3
#4

First,

Build group: edge

group: edge shouldn’t be in your config. These are beta, officially unsupported images. Travis only advises the use of this sometimes to get a preview of some upcoming features.

Now, you are using PHP images, so there aren’t any specific promises about Ruby in them – if it’s present, the default will probably be from the distro.

I see that the kernel version has been updated but the Cookbooks commit (the logic that install Travis-specific software) wasn’t.
So my guess is the image was regenerated on an updated distro, and the preinstalled rubies (built with RVM) probably got the latest bundler for that date.

To say anything else, I’d need to see the builds proper.

#5

Thank you for the insight/clues/hints on our next steps to understanding the Root Cause and more importantly how we should clean up our travis.yml for the future.

I would put the full logs into this post, but they are rather lengthy. I’m a newbie to this forum, so could you give me guidance on how people typically share their build logs? I’d love to share to get more insight, but for now I’ll go dig away… thanks again!

#6

The usual way is to just post a link to the build page – the “build #XX” text gives a permalink. (But if your build is private, only Travis staff will be able to open it.)
The problem with copypasting is that besides the log and the config, the codebase is sometimes needed – to see what exactly you’re doing, be able to reproduce the problem and try various things to fix it. But in this case, the problem is apparently the environment so that might not be necessary.

Imprint