How can I share coverage information between build stages in PRs from forks?


#1

I have a lot of projects that use matrices and build stages to build across different python versions and a stage that checks whether coverage remains at 100%. I need this to work on branches, such as master, and on PRs, including those from forks.

I had this almost working using coveralls.io, but because encrypted environment variables do not work with PRs from forks, I can’t tell the coveralls all the parallel builds are done.

The advice seems to be to use S3, but this looks like this has the same problem in that encrypted environment variables do not work with PRs from forks.

I’ve tried to (ab)use build caches to get this information across, but this build stage that checks coverage only sees cache files for the python version used to run it.

I can’t see any way through this, what should I do?


#2

I see a fundamental conceptual problem here:
If a PR build would in some way be able to push to a third-party, all credentials that are needed for this push are essentially public (that’s why the travis docs call builds like PR builds untrusted).

I would suggest the following (though I’m not sure how well this integrates with coveralls.io in particular): Create a second account (or repository or whatever) and make all untrusted builds push there using unencrypted credentials. This way you can still see the result of PRs, you will have them in your main account (after you merged them) and your main account is shielded from untrusted builds.


#3

I’m not following I’m afraid.
Fundamentally, I’m looking to aggregate some files from the build matrix into a later build stage. I only really want to share those within the Travis job like the build cache does.

What should I do in that case?


#4

Mainly just so I can link things up:

Here’s my original bug report to Travis CI, correctly closed by pointing to the docs saying this is by design:

Here’s the issue I have open with the coveralls guys to see what they can do, if anything:

The above is following up on a closed issue that suggests plugging in notifications:

notifications:
  webhooks: https://coveralls.io/webhook

It’s unclear whether that works at all, but even so, how would I build this in to a job such as this which uses both a matrix and build stages? I want the notification to fire once the matrix is done but before I check for 100% coverage.


#5

Here’s a convoluted workaround that uses the webhook in combination with GitHub’s protected branches for pull requests, and the coverall.io API for the master branch:


#6

Suggestion from @dominic at Travis CI:

I think that given the credentials restrictions on PR builds, your best bet to achieve what you want is by using our caching feature. Have you tried defining the cache’s name to ensure that every jobs will use the same cache?

e.g. CACHE_NAME=JOB1

You would need to set this in every job of your matrix and also ensure that every job doesn’t override the files you need.


#7

@dominic - doing the cache thing like that sounds like a bit of a hack, what’s the “right” way that Travis is going to provide to solve this problem?


#8

@cjw296 Thank for updating the thread with everything you’ve tried so far. It’s really giving me a better understanding of what you are trying to achieve.

First, I don’t think that the CACHE_NAME suggestion will work in this case. Neither will the webhook: recipe because it’s only called at the end of a build and you want it at the end of a job.

I’ve reviewed Coveralls’ documentation and I think it could work by calling directly their API via cURL e.g.

curl -k $COVERALLS_ENDPOINT/webhook -d "payload[build_num]=$BUILD_NUMBER&payload[status]=done" 

You would need to add this command in your .travis.yml file e.g. in after_script: of your jobs.

I’m not quite sure but it’s possible that your contributors would need to register on https://www.coveralls.io/. Maybe this could be an acceptable requirement?

Please let me know what you think and the result you’ll get if you try this out.

Thanks!


#9

I haven’t tried the CACHE_NAME thing, so thanks for saving me some time. The webhook: thing, which was the coveralls.io suggestion, can be made to work for PRs when combined with a GitHub branch protection preventing a PR from being merged without that Coveralls check posting back.

@dominic: posting to that webhook is exactly what I do on master branches, as I explained above, that’s what I wrapped up in the coveralls-check helper I wrote. BUT, it needs credentials plugging in, which Travis prevents on PRs.

Contributors needing to register on coveralls.io is both not an acceptable requirement and won’t help. How would they plug their credentials into the Travis CI job that processes their PR?

It would be really useful if all the jobs within a build, matrix and build stages included, could share some (limited) disk space between them in the same way that caches are stored but explicitly available in all builds, and only for the life time of that build.

Until either that happens (where should I file the feature request?) or you do something different with secrets (which I’m not sure would be a good idea), then the kludge I have above, with PRs and master having to be handled in different ways, will have to suffice.


#10

Hey @cjw296,

Sorry for the delay of my reply.

If you look into coveralls.io documentation, you can see that you don’t need the repo token/credentials for public repos:

With public repos, you can omit the repo token

notifications:
webhooks: https://coveralls.io/webhook

Do you think you could try this out?

Thanks!


#11

@dominic - as I’ve said a few times now, the hacky solution I currently have does use the notifications/webhook stuff, but it can only be used for PRs, rather than when I push direct to master, as the notifications section of a .travis.yml only runs at the end of a build, not in a stage, so I have no way to tell coveralls that my parallel builds are done before checking for a result.


#12

Coveralls.io is completely non-functional at the moment. So, back to looking at ways to share a few small coverage files between build stages. How can I do this? Why won’t the cache hack work?