Do not do HTTP header rewriting

One of my tests ensures that parent directories are not accessed, specifically testing the URL:

http://localhost:5000/tools/../../README.md

Sometime after June18, the server appears to be receiving /README.md instead of the expected /tools/../../README.md.

In my attempts to figure out where this is happening, I included a modified version of werkzeug that reports the first line of the request. That line’s output can be seen in the logs:

GET /README.md HTTP/1.1

Locally, on my development machine, the first line of the request is as expected.

GET /tools/../../README.md HTTP/1.1

I can not dig to lower level code, so I am assuming something changed in Travis that may be “cleaning” these requests.

I have made a much simpler repo to demonstrate the problem:

https://travis-ci.org/klahnakoski/test-travis-get/builds/569009204

image

…notice the requested path is not the same as the received path. My dev environment behaves as expected

image

Check your dependencies carefully. This is not something Travis CI would be involved in.

Check your dependencies carefully. This is not something Travis CI would be involved in.

The only dependency is Python and requests-2.22.0.

Two months ago it was good

Collecting requests (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)

6 days ago it was bad

Collecting requests (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)

and my simpler code that shows the problem uses the same requests-2.22.0

Collecting requests
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)

I have demonstrated the problem with a small piece of code. I do not believe this is a library dependency problem.

the server echo.py

import socket

connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
connection.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
connection.bind(('0.0.0.0', 8000))
connection.listen(10)

current_connection, address = connection.accept()
data = current_connection.recv(2048)
print ("Recieving: " + data)

the client request.py

import requests

try:
    url = "http://localhost:8000/tools/../../README.md"
    print("Requesting: "+url)
    requests.get(url)
except Exception as e:
    pass

the response travis

notice the path changed while outside the control of my code.

This is not true. According to https://travis-ci.org/klahnakoski/test-travis-get/builds/569009204#L192 your build relies on: chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3

Try restarting the previously successful build. I predict that it will fail, due to the changes in the underlying dependencies.

The other dependencies do not seem to have changed either.

Example uses (https://api.travis-ci.org/v3/job/569009205/log.txt)

/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

Good (https://api.travis-ci.org/v3/job/546060497/log.txt)

[2019-06-15T08:47:52,120]
/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

Bad (https://api.travis-ci.org/v3/job/547877602/log.txt)

[2019-08-06T14:05:37,927]
/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

You are right that restarting the last good version will fail, but the dependencies have not changed

Good (restarted) (https://api.travis-ci.org/v3/job/546060497/log.txt)

[2019-08-12T16:37:04,336]
/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

@BanzaiMan I have a ubuntu machine that is also misbehaving like this, let me get back to you…

If it helps, here are the packages that changed version in your restarted build:

Succeeding vs failing
Flask-1.0.3 vs Flask-1.1.1
Werkzeug-0.15.4 vs Werkzeug-0.15.5
certifi-2019.3.9 vs certifi-2019.6.16
mo-future-2.46.19127 vs mo-future-2.48.19205
moz-sql-parser-2.44.19084 vs moz-sql-parser-2.49.19205
pyparsing-2.4.0 vs pyparsing-2.3.1

I would suggest pinning the versions of these dependencies to those who were succeeding.

@dominic @BanzaiMan thank you both for your patience with this. You are correct it was in a Python library:

1 Like
Imprint