How can I share coverage information between build stages in PRs from forks?

@dominic - doing the cache thing like that sounds like a bit of a hack, what’s the “right” way that Travis is going to provide to solve this problem?

@cjw296 Thank for updating the thread with everything you’ve tried so far. It’s really giving me a better understanding of what you are trying to achieve.

First, I don’t think that the CACHE_NAME suggestion will work in this case. Neither will the webhook: recipe because it’s only called at the end of a build and you want it at the end of a job.

I’ve reviewed Coveralls’ documentation and I think it could work by calling directly their API via cURL e.g.

curl -k $COVERALLS_ENDPOINT/webhook -d "payload[build_num]=$BUILD_NUMBER&payload[status]=done" 

You would need to add this command in your .travis.yml file e.g. in after_script: of your jobs.

I’m not quite sure but it’s possible that your contributors would need to register on https://www.coveralls.io/. Maybe this could be an acceptable requirement?

Please let me know what you think and the result you’ll get if you try this out.

Thanks!

I haven’t tried the CACHE_NAME thing, so thanks for saving me some time. The webhook: thing, which was the coveralls.io suggestion, can be made to work for PRs when combined with a GitHub branch protection preventing a PR from being merged without that Coveralls check posting back.

@dominic: posting to that webhook is exactly what I do on master branches, as I explained above, that’s what I wrapped up in the coveralls-check helper I wrote. BUT, it needs credentials plugging in, which Travis prevents on PRs.

Contributors needing to register on coveralls.io is both not an acceptable requirement and won’t help. How would they plug their credentials into the Travis CI job that processes their PR?

It would be really useful if all the jobs within a build, matrix and build stages included, could share some (limited) disk space between them in the same way that caches are stored but explicitly available in all builds, and only for the life time of that build.

Until either that happens (where should I file the feature request?) or you do something different with secrets (which I’m not sure would be a good idea), then the kludge I have above, with PRs and master having to be handled in different ways, will have to suffice.

Hey @cjw296,

Sorry for the delay of my reply.

If you look into coveralls.io documentation, you can see that you don’t need the repo token/credentials for public repos:

With public repos, you can omit the repo token

notifications:
webhooks: https://coveralls.io/webhook

Do you think you could try this out?

Thanks!

@dominic - as I’ve said a few times now, the hacky solution I currently have does use the notifications/webhook stuff, but it can only be used for PRs, rather than when I push direct to master, as the notifications section of a .travis.yml only runs at the end of a build, not in a stage, so I have no way to tell coveralls that my parallel builds are done before checking for a result.

Coveralls.io is completely non-functional at the moment. So, back to looking at ways to share a few small coverage files between build stages. How can I do this? Why won’t the cache hack work?

coveralls continues to be flakey, and codecov will only do anything once the whole build is finished.
I don’t really need either of them, to be fair, since all my stuff considers <100% coverage a failure.
What I do need is a way to share some small coverage files between all jobs in a build for both master and PR builds. How can I achieve this?

@dominic - any thoughts?

You can use build caches, but you’ll need a separate coverage job for each configuration. This is logical since each configuration of your project is, strictly speaking, a separate piece of software.

Not sure that’ll help: I need to combine coverage information from all the jobs in the “test” stage to form a complete picture of code coverage; no individual job will hit 100%, but combined they will do.

That’s why I mentioned that “each configuration of your project is, strictly speaking, a separate piece of software.”

Since each your configuration works independently from others, it needs to be considered a separate product, and its test coverage needs to be considered separately. “Aggregate score” that you have in mind will show if each piece of code is tested in some configuration but will say nothing about how well each individual configuration is covered.

Imagine e.g. that you run one half of tests in one configuration and the other half in another. The sum is 100% but each product is only half covered.

Of course, that means that you need to somehow exclude code from coverage measurement that will only be executed in some configurations. One way is to somehow physically cut it from the sources (e.g. with a preprocessor) since in those configurations, it’s effectively dead code. Another is to somehow instruct coverage.py to exclude it from measurement.

@native-api - I strongly disagree, sorry, but your approach is not one I wish to consider. I want to combine coverage from the jobs in the test stage and check that the combined coverage is 100%. I know how to do this on the project side using python’s coverage tools, I do it happily for Jenkins-based CI with my commercial stuff. I’d love to do it using Travis for all my open source stuff. So, respectfully, if you cannot contribute to a solution to the problem I’m asking for help with, I’d ask that you refrain from commenting further.

Well, then

  • Travis cache is out of question:
  • Using any 3rd-party service is impossible due to conflicting requirements:
    • You require build agents to be authenticated to be able to contribute data, but you refuse to provide them with any means to authenticate themselves because they are untrusted.

The only idea I have is to use the travis console utility to extract information about the current build, locate the appropriate jobs within it and extract the necessary data from their build logs (a build log is the only build result that Travis retains).

I’m currently experimenting with this hack: https://twitter.com/webKnjaZ/status/1097078272867921920

It’s just frustrating that there’s not build-wide shared space, then I wouldn’t need any third party service or any hackery. The cache is potentially huge, and I’m talking about a few kb of coverage information here…

Feature request in: Shared disk space between all jobs in a build

@cjw296 Sorry for the long delay here.

What happens is that our cache feature is language version dependent e.g. a cache for a Python 2.6 job can’t use a cache from a Python 2.7 job.

This behaviour can be overridden with the CACHE_NAME feature but then packages from all language versions would overwrite each other possibly leading to incompatible caches.

A possible solution to use CACHE_NAME could be to ensure your dependencies are installed in a location specific to the Python version + Django version of each job. You could then specify a directory shared by each job as well.

e.g.

cache:
  directories:
    - $HOME/.cache/pip-2.6-latest
    - $HOME/.cache/pip-2.7-latest
    ...
    - coverall-shared-dir

Do you think this could work?

@dominic - thanks for the reply, I don’t need to use caching (I often don’t), so what do I set CACHE_NAME to (and where in my .travis.yaml?) just that I get some shared space across all jobs in a build?

@dominic - I don’t believe that CACHE_NAME will work for the reasons given here: https://github.com/travis-ci/travis-ci/issues/7392

I think you are right. CACHE_NAME would only work without environment variables (or they would need all the same names and values) or language runtime version.

I think at this point you would need to use some kind of 3rd party hosting provider, possibly one that doesn’t require authentication.

I’m sorry for not having an out of the box solution for you right now.

Third party hosting provider that doesn’t require authentication? You’re a funny guy :wink:

Please can you do what you can to support my feature request:

…in the meantime, I’m going to have to go with a horrible hack of re-triggering a travis build via the api using a custom config so that I can check all-but-the-current-travis-status on the current commit using the github status api :cry:

Sorry for the delay of my reply but I thought you might be interested in this new feature we have released:

It would allow you to share files within one build without requiring an external hosting provider nor sharing its credentials.

Let us know what you think in the linking post.

Thanks!

1 Like

@cjw296 I know you were using coveralls, but in case you’re still looking for a solution, CodeCov automatically merges all reports from a build together.
I stumbled upon this thread while looking for a solution to this for an open-source project of mine, and then found out that CodeCov already handles this scenario automatically! Hopefully this is a useful note for any future readers :slight_smile: