Sauce Labs SSL certificate not accepted for Selenium tests on https://ondemand.saucelabs.com

Hi, I am a maintainer of the CATMAID open source project and we run our tests on Travis. For oue fron-end tests we use Selenium with Sauce Labs and so far we had the remote webdriver connect to localhost:4445. This doesn’t work anymore since the recent update of Sauce Connect. Instead it seems to be recommended now to let the remote webdriver connect to https://ondemand.saucelabs.com/wd/hub/. Doing so however results in SSL verification errors:

OpenSSL.SSL.Error: [(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)]

An example build for this can be found here: https://travis-ci.org/github/catmaid/CATMAID/jobs/695946597

I could imagine updating the local certificate store with the respective certificate, but wonder if I am missing something here.

Edit: I tried adding the certificate of ondemand.saucelabs.com:443 to our Travis test VM, but this didn’t change anything unfortunately. I did the following:

  1. Store a copy of openssl s_client -showcerts -connect ondemand.saucelabs.com:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >saucelabs.crt in our repo.
  2. Add the following two lines to the before_install step:
    sudo cp scripts/travis/saucelabs.crt /usr/local/share/ca-certificates/saucelabs.crt
    sudo update-ca-certificates

Thanks,
Tom

I am not sure how you are checking the certificate, but curl is able to verify it.

https://travis-ci.org/github/BanzaiMan/travis_production_test/jobs/696174371#L470
https://travis-ci.org/github/BanzaiMan/travis_production_test/jobs/696174372#L184
https://travis-ci.org/github/BanzaiMan/travis_production_test/jobs/696174373#L230

If you would like, we can enable the debug feature for you, so that you can troubleshoot this further.

Looks like you are using some 3rd-party SSL package (urllib3). You need to provide a certificate bundle to it in some way specific to that package.

Usually people use requests which IIRC has a built-in bundle as well as can use one from certifi if it’s installed. But you seem to be using urllib3 directly.

https://dokk.org/documentation/urllib3/1.14/security/

Sorry, for some reason I first “replied” using the edit function of my last post, where I linked to the HTTP version of the code connecting to Sauce Labs. I meant to link to the latest debug version with HTTPS. It is just not pretty because of many debugging attempts:

I can’t see for sure from the provided information who specifically is the responsible party so can’t advise anything with confidence: for some reason, the traceback is incomplete.

http in command_executor=f"http://{hub_url}/wd/hub" strikes me as odd: if you are not using encryption, why an SSL error? I guess that server has an automatic redirection to HTTPS but due to you using http://, whatever executes that command doesn’t prepare itself for certificate verification.

So see if specifying https:// solves the problem, that’s my best idea with the information given.

Sorry for the confusion, the linked source code is from our master branch, containing the last working version using HTTP. I only linked it, to point to the source code that connects to Sauce Labs, regardless of the URL (we can’t update master with the https:// version right now, because the build would fail). I should have linked to the source code of the build I linked in the first post:

It just isn’t pretty due to the many debug attempts I did, but it contains the https:// link that fails. It also shows that just using https doesn’t change the situation and is ultimately what lead me to creating this post.

I also tried installing the most recent version of certifi, but it didn’t help.

Googling “webdriver.Remote command_executor ssl certificate verification”, then looking at https://github.com/SeleniumHQ/selenium/issues/6534 and https://github.com/SeleniumHQ/selenium/commit/bbc0ec057b2aedb584e5a99ba1f0842e9b4fa4f6, the last (as of this writing) alpha version of Selenium, selenium-4.0.0-alpha-6 (check https://pypi.org/project/selenium/#history for what you need to supply to PIP), does now use certifi to verify certificates.

With that in mind, looking at your linked build, you are using a 2-years-old version of certifi, so you probably want to change that, too.

Thanks, that’s good to know and it looks like I am in the same boat as “awrichar” in that thread you linked (issue 6534 in the Selenium repo):

however it means that I can’t use Selenium against any remote server with HTTPS, because I now always get a MaxRetryError even though the server has valid certs."

That’s exactly what I am seeing after the local relay of Sauce Connect isn’t enabled by default anymore.

It’s good to know that Selenium 4 will do better with SSL, but for now I will stick to Selenium 3, because I don’t feel easy about using an alpha build in my stable environment.

however it means that I can’t use Selenium against any remote server with HTTPS, because I now always get a MaxRetryError even though the server has valid certs.

I didn’t link to the debug built with the most recent certifi, but doing so didn’t change a thing, unfortunately. So far we actually haven’t specified certifi as explicit dependency.

Ultimately, it looks like this is a Selenium issue and only a few people seem to run into it. Thanks for your help in figuring out that this is unrelated to Travis and pointing me at the right place to look. I somehow need to convince urllib3 to accept the certificate (like I said, latest certifi didn’t help) and will keep digging. Since this a CI test environment with artificial users and passwords, I might just go unencrypted for now so that my tests can succeed again. HTTPS is recommended by Sauce Labs, but this might have to wait until Selenium 4 is out or I find another workaround.

Read that topic more carefully. It provides a workaround.

I suppose you mean the last comment from the issue:

For now I’ve pinned my urllib3 to v1.24, which retains the (insecure) behavior detailed in this Issue. However, I’d like to find a path toward using the (more secure) urllib3 v1.25 with proper cert validation

This does unfortunately not work for me, because I have other dependencies that require urlib3 >= 1.25.7:

ERROR: cloud-volume 1.9.1 has requirement urllib3[brotli,secure]>=1.25.7, but you’ll have urllib3 1.24 which is incompatible.

If that was the workaround you had in mind, I apologize and should have said that I tried that before. I believe I read the issue carefully, but there is of course still a chance I missed something and would be happy to hear if you are referring to something else.

1 Like

Yes, it was, and thanks for clarifying, too.

If using an older urllib3 or a non-stable selenium version is not an option for you, I’m afraid, there isn’t any option I can suggest that is compatible with your requirements.

You can try applying just the PR that commit belongs to to a release version of Selenium, but that would effectively be a non-stable version just as well.

Imprint