Ruby Openssl: Python deployment fails on osx image

Hi, I have two builds for my Python library failing to deploy from macOS. It seems to have something to do with the openssl ruby gem for the deployment.

These are the failing jobs:

I’ll try to restart them, see the error message also below.

This is my deployment config:

deploy_template: &deploy_template
  on:
    tags: true
    branch: master
  skip_cleanup: true

deploy:
  - provider: releases
    <<: *deploy_template
    api_key: $GitHubOAUTH
    file_glob: true
    file: "./dist/pupil_apriltags*"
  - provider: script
    <<: *deploy_template
    script: bash ./.travis/pypi_deploy.sh

This is the error message:

Error loading RubyGems plugin "/Users/travis/.rvm/gems/ruby-2.4.2@global/gems/executable-hooks-1.3.2/lib/rubygems_plugin.rb": dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle (LoadError)
Error loading RubyGems plugin "/Users/travis/.rvm/gems/ruby-2.4.2/gems/gem-wrappers-1.4.0/lib/rubygems_plugin.rb": dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle (LoadError)
4.33s$ rvm $(travis_internal_ruby) --fuzzy do ruby -S gem install dpl
Warning, new version of rvm available '1.29.9-next', you are using older version '1.27.0'.
You can disable this warning with:    echo rvm_autoupdate_flag=0 >> ~/.rvmrc
You can enable  auto-update  with:    echo rvm_autoupdate_flag=2 >> ~/.rvmrc
Error loading RubyGems plugin "/Users/travis/.rvm/gems/ruby-2.4.2@global/gems/executable-hooks-1.3.2/lib/rubygems_plugin.rb": dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle (LoadError)
Error loading RubyGems plugin "/Users/travis/.rvm/gems/ruby-2.4.2/gems/gem-wrappers-1.4.0/lib/rubygems_plugin.rb": dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle (LoadError)
Error loading RubyGems plugin "/Users/travis/.rvm/gems/ruby-2.4.2@global/gems/executable-hooks-1.3.2/lib/rubygems_plugin.rb": dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle (LoadError)
Error loading RubyGems plugin "/Users/travis/.rvm/gems/ruby-2.4.2/gems/gem-wrappers-1.4.0/lib/rubygems_plugin.rb": dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle (LoadError)
ERROR:  Loading command: install (LoadError)
	dlopen(/Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
  Reason: image not found - /Users/travis/.rvm/gems/ruby-2.4.2/gems/openssl-2.0.6/lib/openssl.bundle
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass
The command "rvm $(travis_internal_ruby) --fuzzy do ruby -S gem install dpl" failed and exited with 1 during .

A likely explanation here is that your upgrading Homebrew updated OpenSSL on which our pre-built Ruby depends on (from 1.0.x to 1.1.y), and made it unusable. A newer macOS image may not (or may) have this problem.

Hi @BanzaiMan

thanks for the fast response. I noticed that previous builds would also upgrade homebrew’s openssl to the exact same version (openssl@1.1-1.1.1d) and there I had no problems. Here’s the log from the previous (successfull) deployment: https://travis-ci.org/pupil-labs/apriltags/jobs/588936482

So I did not change anything really. It appears the dependencies of the deployment changed instead?

I can’t tell you why it was successful before, but I’d expect a lot of things to change, including Homebrew itself and how it manages stuff.

Whatever the reason, I advise you to try a newer image.

Unfortunately I have to use this image, as the Python wheels will be only compatible for macOS versions above the one used for building this wheel. This library is used in a bigger context for a software which still has to support OSX 10.12. So I have to build wheels from 10.12 somehow.

Do you have any hints on what I could try to get this working?

@BanzaiMan
I ran brew list --versions right at the start (before upgrading homebrew) of the image → output
Note that there seem to be 2 versions of openssl installed:

139  openssl 1.0.2l
140  openssl@1.1 1.1.0f

If you go back to the broken deployment job, you can see that your deployment tries to reference /usr/local/opt/openssl/lib/libssl.1.0.0.dylib, which appears to be the first of the two installed versions.
My upgrade command will only update openssl@1.1, which also lives in a different folder: /usr/local/opt/openssl@1.1.

From what I am seeing here it appears that the base-installation of homebrew got updated on the image? From the travis docs:

Homebrew is installed and updated every time the virtual machines are updated.

That would mean that the deployment on this image (xcode8.3) is broken in general.

@pfaion First af all, as https://travis-ci.org/pupil-labs/apriltags/jobs/633671779#L3963 says, 10.12 is unsupported by Apple and Homebrew so your builds are bound to break eventually and you’ll be on your own (Homebrew does accept PRs from users that fix breakages though). Consider upgrading to a supported version.


Now, the reason is that the preinstalled ruby-2.4.2 that Travis uses for its logic uses OpenSSL 1.0 from Homebrew. Which Homebrew dropped recently in favor of 1.1 because 1.0 is EOL.

So this gets broken whenever one brew installs anything that requires openssl (pyenv in your case).


@BanzaiMan The solution is to either:

  • update Homebrew in affected images and rebuild travis_internal_ruby version against the newer OpenSSL; or
  • rebuild travis_internal_ruby version in affected images against the versioned location, /usr/local/opt/openssl@1.0, which is left intact after Homebrew upgrades OpenSSL (see below).

As a workaround, running the following in before_deploy: might work (I can’t test it):

rvm reinstall $(travis_internal_ruby)

It’s supposed to implement the first option. It might not work if rvm reinstall wipes custom installed gems and some need to be present for Travis machinery to work properly.

For the second option, need to also supply --with-openssl-lib=/usr/local/opt/openssl@1.0/lib --with-openssl-include=/usr/local/opt/openssl@1.0/include to rvm install/rvm reinstall as per https://github.com/kelaberetiv/TagUI/issues/635#issuecomment-564369194.

@native-api As I said, I have to support macOS 10.12 for now with this library. I specifically looked at the travis docs for macOS and did not find any information that any of those versions were not officially supported by travis anymore. It seems the image was updated (probably with a security update?) since OpenSSL 1.0 is not even installed anymore on the bare image, even without me upgrading homebrew?


Anyways. The first workaround unfortunately did not work, see this log.

I am not sure how to proceed with the second option as I have no experience with ruby.

I’ll see if I can just get the PyPI deployment working on macOS, as I am using the script deployment provider for this. It might be that just the GitHub releases provider is broken. This would be unfortunate but at least it would push my release to the public on some channel.

Update:

The script deployment also does not work. I can’t even get it to work by e.g. including

deploy:
  on:
    - condition: os != osx

Even though it’s disabled by condition, dpl still is getting loaded up, which causes the crash.

I made a manual release now by running custom code in after_success on a custom branch, but this is definitely only a workaround.


Is this supposed to get fixed at some point, or do I have to fix or appropriately workaround this myself?

Since I don’t know how to mess with the ruby installation, I’d probably just stop using deploy and implement custom scripts for branch/tag testing and manual deploy.

@pfaion Okay, I devised a working workaround (it implements the 1st option – rebuilds Ruby against the newer OpenSSL):

before_deploy:
  - for lib in ssl crypto; do ln -s /usr/local/opt/openssl{@1.0,}/lib/lib${lib}.1.0.0.dylib; done
  - rvm reinstall $(travis_internal_ruby) --disable-binary
  - for lib in ssl crypto; do rm /usr/local/opt/openssl/lib/lib${lib}.1.0.0.dylib; done

Thank you so much!