The issue: Cannot accurately test Pull Requests from outside repositories when the Travis-CI tests require secure environment variables.
Proposal: If the origin of the PR has environment variables in its Repository Settings, include those variables for the Travis-CI tests.
Example:
- I created commit duanehutchins/heroku-buildpack-python@f7680ab
- Travis-CI includes the environment variables from my repo’s Repository Settings: https://travis-ci.com/duanehutchins/heroku-buildpack-python/settings
- That commit passes CI: https://travis-ci.com/duanehutchins/heroku-buildpack-python/builds/87256182
- Same commit is used in PR: heroku/heroku-buildpack-python#766
- Travis-CI does not include the environment variables from my repo’s Repository Settings
- PR (same commit) fails CI because of missing environment variables: https://travis-ci.org/heroku/heroku-buildpack-python/jobs/438849455#L561
If Travis-CI included my repo’s Repository Settings when testing PR originating from my repo, then that PR would have passed CI.
To distinguish something, PR-Target envs still won’t be included because the PR can expose the secure envs. I’m only proposing that we include PR-Origin envs because the Origin of the PR will control the code which is being tested through CI.
This maintains security because the origin of the PR is aware (or should be aware) of the Travis-CI tests prior to sending the PR.
Alternatively, if the commit has already passed CI (as in the example above), then skip CI tests on PR.