Sauce Labs SSL certificate not accepted for Selenium tests on https://ondemand.saucelabs.com

tomka · June 8, 2020, 10:08am

Hi, I am a maintainer of the CATMAID open source project and we run our tests on Travis. For oue fron-end tests we use Selenium with Sauce Labs and so far we had the remote webdriver connect to localhost:4445. This doesn’t work anymore since the recent update of Sauce Connect. Instead it seems to be recommended now to let the remote webdriver connect to https://ondemand.saucelabs.com/wd/hub/. Doing so however results in SSL verification errors:

OpenSSL.SSL.Error: [(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)]

An example build for this can be found here: Travis CI - Test and Deploy with Confidence

I could imagine updating the local certificate store with the respective certificate, but wonder if I am missing something here.

Edit: I tried adding the certificate of ondemand.saucelabs.com:443 to our Travis test VM, but this didn’t change anything unfortunately. I did the following:

Store a copy of openssl s_client -showcerts -connect ondemand.saucelabs.com:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >saucelabs.crt in our repo.

Add the following two lines to the before_install step:
sudo cp scripts/travis/saucelabs.crt /usr/local/share/ca-certificates/saucelabs.crt
sudo update-ca-certificates

Thanks,
Tom

BanzaiMan · June 8, 2020, 7:22pm

I am not sure how you are checking the certificate, but curl is able to verify it.

https://travis-ci.org/github/BanzaiMan/travis_production_test/jobs/696174371#L470
https://travis-ci.org/github/BanzaiMan/travis_production_test/jobs/696174372#L184
https://travis-ci.org/github/BanzaiMan/travis_production_test/jobs/696174373#L230

If you would like, we can enable the debug feature for you, so that you can troubleshoot this further.

native-api · June 8, 2020, 7:52pm

Looks like you are using some 3rd-party SSL package (urllib3). You need to provide a certificate bundle to it in some way specific to that package.

Usually people use requests which IIRC has a built-in bundle as well as can use one from certifi if it’s installed. But you seem to be using urllib3 directly.

https://dokk.org/documentation/urllib3/1.14/security/

tomka · June 8, 2020, 8:34pm

Sorry, for some reason I first “replied” using the edit function of my last post, where I linked to the HTTP version of the code connecting to Sauce Labs. I meant to link to the latest debug version with HTTPS. It is just not pretty because of many debugging attempts:

github.com

catmaid/CATMAID/blob/950bc2fc0b28e6a54f24245c2a6d4792e2ae83bd/django/applications/catmaid/tests/gui/test_basic_gui.py#L134


        "build": os.environ["TRAVIS_BUILD_NUMBER"],
        "tags": [os.environ["TRAVIS_PYTHON_VERSION"], "CI"],
        #"username": username,
        #"access_key": access_key,
    }
    #hub_url = f"{username}:{access_key}@localhost:4445"
    #hub_url = f"{username}:{access_key}@localhost:8888"
    #hub_url = f"{username}:{access_key}@saucelabs.com:4445"
    #hub_url = f"{username}:{access_key}@ondemand.saucelabs.com:443"
    hub_url = f"{username}:{access_key}@ondemand.saucelabs.com"
    self.selenium = webdriver.Remote(
        desired_capabilities=capabilities,
        #command_executor="https://ondemand.saucelabs.com:4444/wd/hub"
        #command_executor="https://ondemand.saucelabs.com:443/wd/hub"
        command_executor=f"https://{hub_url}/wd/hub",
    )
else:
    self.selenium = webdriver.Firefox()

# Give browser a chance to load elements
self.selenium.implicitly_wait(20)

native-api · June 9, 2020, 2:16am

I can’t see for sure from the provided information who specifically is the responsible party so can’t advise anything with confidence: for some reason, the traceback is incomplete.

http in command_executor=f"http://{hub_url}/wd/hub" strikes me as odd: if you are not using encryption, why an SSL error? I guess that server has an automatic redirection to HTTPS but due to you using http://, whatever executes that command doesn’t prepare itself for certificate verification.

So see if specifying https:// solves the problem, that’s my best idea with the information given.

tomka · June 9, 2020, 4:30am

Sorry for the confusion, the linked source code is from our master branch, containing the last working version using HTTP. I only linked it, to point to the source code that connects to Sauce Labs, regardless of the URL (we can’t update master with the https:// version right now, because the build would fail). I should have linked to the source code of the build I linked in the first post:

github.com

catmaid/CATMAID/blob/950bc2fc0b28e6a54f24245c2a6d4792e2ae83bd/django/applications/catmaid/tests/gui/test_basic_gui.py#L134


        "build": os.environ["TRAVIS_BUILD_NUMBER"],
        "tags": [os.environ["TRAVIS_PYTHON_VERSION"], "CI"],
        #"username": username,
        #"access_key": access_key,
    }
    #hub_url = f"{username}:{access_key}@localhost:4445"
    #hub_url = f"{username}:{access_key}@localhost:8888"
    #hub_url = f"{username}:{access_key}@saucelabs.com:4445"
    #hub_url = f"{username}:{access_key}@ondemand.saucelabs.com:443"
    hub_url = f"{username}:{access_key}@ondemand.saucelabs.com"
    self.selenium = webdriver.Remote(
        desired_capabilities=capabilities,
        #command_executor="https://ondemand.saucelabs.com:4444/wd/hub"
        #command_executor="https://ondemand.saucelabs.com:443/wd/hub"
        command_executor=f"https://{hub_url}/wd/hub",
    )
else:
    self.selenium = webdriver.Firefox()

# Give browser a chance to load elements
self.selenium.implicitly_wait(20)

It just isn’t pretty due to the many debug attempts I did, but it contains the https:// link that fails. It also shows that just using https doesn’t change the situation and is ultimately what lead me to creating this post.

I also tried installing the most recent version of certifi, but it didn’t help.

native-api · June 9, 2020, 10:02am

Googling “webdriver.Remote command_executor ssl certificate verification”, then looking at https://github.com/SeleniumHQ/selenium/issues/6534 and https://github.com/SeleniumHQ/selenium/commit/bbc0ec057b2aedb584e5a99ba1f0842e9b4fa4f6, the last (as of this writing) alpha version of Selenium, selenium-4.0.0-alpha-6 (check https://pypi.org/project/selenium/#history for what you need to supply to PIP), does now use certifi to verify certificates.

With that in mind, looking at your linked build, you are using a 2-years-old version of certifi, so you probably want to change that, too.

tomka · June 10, 2020, 6:18am

Thanks, that’s good to know and it looks like I am in the same boat as “awrichar” in that thread you linked (issue 6534 in the Selenium repo):

however it means that I can’t use Selenium against any remote server with HTTPS, because I now always get a MaxRetryError even though the server has valid certs."

That’s exactly what I am seeing after the local relay of Sauce Connect isn’t enabled by default anymore.

It’s good to know that Selenium 4 will do better with SSL, but for now I will stick to Selenium 3, because I don’t feel easy about using an alpha build in my stable environment.

however it means that I can’t use Selenium against any remote server with HTTPS, because I now always get a MaxRetryError even though the server has valid certs.

I didn’t link to the debug built with the most recent certifi, but doing so didn’t change a thing, unfortunately. So far we actually haven’t specified certifi as explicit dependency.

Ultimately, it looks like this is a Selenium issue and only a few people seem to run into it. Thanks for your help in figuring out that this is unrelated to Travis and pointing me at the right place to look. I somehow need to convince urllib3 to accept the certificate (like I said, latest certifi didn’t help) and will keep digging. Since this a CI test environment with artificial users and passwords, I might just go unencrypted for now so that my tests can succeed again. HTTPS is recommended by Sauce Labs, but this might have to wait until Selenium 4 is out or I find another workaround.

native-api · June 10, 2020, 7:45am

Read that topic more carefully. It provides a workaround.

tomka · June 10, 2020, 10:00am

I suppose you mean the last comment from the issue:

For now I’ve pinned my urllib3 to v1.24, which retains the (insecure) behavior detailed in this Issue. However, I’d like to find a path toward using the (more secure) urllib3 v1.25 with proper cert validation

This does unfortunately not work for me, because I have other dependencies that require urlib3 >= 1.25.7:

ERROR: cloud-volume 1.9.1 has requirement urllib3[brotli,secure]>=1.25.7, but you’ll have urllib3 1.24 which is incompatible.

If that was the workaround you had in mind, I apologize and should have said that I tried that before. I believe I read the issue carefully, but there is of course still a chance I missed something and would be happy to hear if you are referring to something else.

native-api · June 10, 2020, 11:04am

Yes, it was, and thanks for clarifying, too.

If using an older urllib3 or a non-stable selenium version is not an option for you, I’m afraid, there isn’t any option I can suggest that is compatible with your requirements.

You can try applying just the PR that commit belongs to to a release version of Selenium, but that would effectively be a non-stable version just as well.

Topic		Replies	Views
Polymer async/promises tests timeout in sauce labs, but not locally Travis CI Discussions & Feedback travis-build , sauce-integration	1	424	May 13, 2021
SSL error setting up Xcode 7.3 macOS	0	962	December 22, 2018
Opening test url times out over Sauce Connect Integrations	10	2509	September 24, 2020
Find: 'sc-*': No such file or directory , sauce connect is failing Windows	4	628	September 8, 2020
Problem when integrating with SauceLabs, none of the test reports are stored Swift	0	1057	March 29, 2019

Sauce Labs SSL certificate not accepted for Selenium tests on https://ondemand.saucelabs.com

Related topics