Do not do HTTP header rewriting

Languages Python
1

One of my tests ensures that parent directories are not accessed, specifically testing the URL:

http://localhost:5000/tools/../../README.md

Sometime after June18, the server appears to be receiving /README.md instead of the expected /tools/../../README.md.

In my attempts to figure out where this is happening, I included a modified version of werkzeug that reports the first line of the request. That line’s output can be seen in the logs:

GET /README.md HTTP/1.1

Locally, on my development machine, the first line of the request is as expected.

GET /tools/../../README.md HTTP/1.1

I can not dig to lower level code, so I am assuming something changed in Travis that may be “cleaning” these requests.

2

I have made a much simpler repo to demonstrate the problem:

https://travis-ci.org/klahnakoski/test-travis-get/builds/569009204

image

3

…notice the requested path is not the same as the received path. My dev environment behaves as expected

image

4

Check your dependencies carefully. This is not something Travis CI would be involved in.

5

Check your dependencies carefully. This is not something Travis CI would be involved in.

The only dependency is Python and requests-2.22.0.

Two months ago it was good

Collecting requests (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)

6 days ago it was bad

Collecting requests (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)

and my simpler code that shows the problem uses the same requests-2.22.0

Collecting requests
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)

I have demonstrated the problem with a small piece of code. I do not believe this is a library dependency problem.

the server echo.py

import socket

connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
connection.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
connection.bind(('0.0.0.0', 8000))
connection.listen(10)

current_connection, address = connection.accept()
data = current_connection.recv(2048)
print ("Recieving: " + data)

the client request.py

import requests

try:
    url = "http://localhost:8000/tools/../../README.md"
    print("Requesting: "+url)
    requests.get(url)
except Exception as e:
    pass

the response travis

image
image1016×267 9.14 KB

notice the path changed while outside the control of my code.

6

This is not true. According to Travis CI - Test and Deploy with Confidence your build relies on: chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3

Try restarting the previously successful build. I predict that it will fail, due to the changes in the underlying dependencies.

7

The other dependencies do not seem to have changed either.

Example uses (https://api.travis-ci.org/v3/job/569009205/log.txt)

/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

Good (https://api.travis-ci.org/v3/job/546060497/log.txt)

[2019-06-15T08:47:52,120]
/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

Bad (https://api.travis-ci.org/v3/job/547877602/log.txt)

[2019-08-06T14:05:37,927]
/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)

You are right that restarting the last good version will fail, but the dependencies have not changed

Good (restarted) (https://api.travis-ci.org/v3/job/546060497/log.txt)

[2019-08-12T16:37:04,336]
/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
8

@BanzaiMan I have a ubuntu machine that is also misbehaving like this, let me get back to you…

9

If it helps, here are the packages that changed version in your restarted build:

Succeeding vs failing
Flask-1.0.3 vs Flask-1.1.1
Werkzeug-0.15.4 vs Werkzeug-0.15.5
certifi-2019.3.9 vs certifi-2019.6.16
mo-future-2.46.19127 vs mo-future-2.48.19205
moz-sql-parser-2.44.19084 vs moz-sql-parser-2.49.19205
pyparsing-2.4.0 vs pyparsing-2.3.1

I would suggest pinning the versions of these dependencies to those who were succeeding.

10

@dominic @BanzaiMan thank you both for your patience with this. You are correct it was in a Python library:

1 Like