Unit test in R with testthat fails but Travis CI build is successful

Languages R
, ,
1

I am currently working on a toy R package to see how it works with CI. I am trying to use unit tests with the testthat package. I use GitHub and Travis for CI integration.

For reproducing the results, here is my repository:
git clone https://github.com/Redcart/helda.git

  1. I have coded a simple unit test in a script called test-proc_freq.R that should works

  • When I run locally devtools::test(), the test is passing as expected.

  • When I run locally devtools::check(), all is fine.

  1. then I modify the test-proc_freq.R file adding some line that should fails the test

  • When I run locally devtools::test(), the tests fail as expected.

  • When I run locally devtools::check(), I get no messages of any failures. I have the same messages when the test was passing.

I am wondering what the check function of the package devtools is doing for the unit tests ? I have the impression that it just checks that the unit tests are properly coded and can be launched but not whether it has passed or failed.

When I push my code on GitHub, Travis CI is triggered thanks to the following .travis-ci.yml file:

# some stuff before

script:
 - R CMD build .
 - R CMD check --as-cran helda_0.1.tar.gz
 - Rscript -e "library(testthat); testthat::test_dir('tests/testthat')"

# some stuff after

However in the two situations described above, the build with Travis CI is successful.

Even adding Rscript -e "library(testthat); testthat::test_dir('tests/testthat')" has not helped:

build
build1444×562 54.5 KB

How can I make Travis CI fail the build when the unit tests fail ?

2

According to https://www.rdocumentation.org/packages/testthat/versions/2.3.1/topics/test_dir, test_dir returns “A list of test results” – i.e. nothing that would cause the code passed to Rscript to terminate abnormally.

3

You can make test_dir throw an error by changing the reporter function.

I have tried running travis with the line below and the build halted when the tests failed:

  - Rscript -e "library(testthat); testthat::test_dir('tests/testthat', reporter = c('progress', 'fail'))"

The ‘fail’ reporter is the one that throws the error; and the ‘progress’ reporter coming before ensures you still get some information on exactly what failed.

Here is the ‘fail’ reporter in action:

image
image1203×83 7.27 KB

Hope it helps :+1:

1 Like
4

Thank you for your answer. However, I use now the test_check function of the package devtools which is more convenient. I understood later that problem I faced came from the fact that the test_dir function has the argument stop_on_failure set by default to FALSE contrary to the other function of the devtools package. Therefore, in order to make the build on Travis CI fail when the unit test fail, using either the test_check function with default settings or the test_dir function with stop_on_failure set to TRUE seems the right way to do.

1 Like