Unnecessary work in R package build under macos

Languages R
, , , ,
1

Recently, I’ve noticed that OSX package checks that used to run in <10min have been taking 3-4 times as long. The biggest use of time is in the installation of homebrew dependencies. Among the packages installed are many that are not directly relevant or, it would seem, needed, for the package checking: gnupg, pinentry utilities, gdal/geos/postgis, postgresql utilties, and many others. An example is located here.

  1. How can I configure the build so as to include only necessary packages?
  2. Failing that, is it possible to set up caching so that they are not laboriously reinstalled unnecessarily?
  3. Where (if anywhere) can I find documentation on this topic?

Thanks very much!

2

Try caching, brew update metadata, you only need to cache the .git folders within /usr/local/Homebrew. This should speed the build process up somewhat, see if this helps at all.

Travis configuration for everyone else reading:

cache:
  directories:
    - $HOME/Library/Caches/Homebrew
    - /usr/local/Homebrew

before_cache:
  - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then brew cleanup; fi
  - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then find /usr/local/Homebrew \! -regex ".+\.git.+" -delete; fi
1 Like
3

Thanks @Montana for the speedy reply. I still find the YAML config file a bit mystifying. Suppose I have the lines

language: r
cache: packages

Would I replace these with

cache:
  - packages
  - directories:
    - $HOME/Library/Caches/Homebrew
    - /usr/local/Homebrew

or something else? Thanks in advance for your help!

4

No problem for the quick response! It would be simple, just add this line into your .travis.yml.

cache:
  directories:
    - /usr/local/Homebrew/
    - $HOME/Library/Caches/Homebrew/
1 Like
5

I get the following Build config error:

root: duplicate key: cache

This is because I already have

cache: packages

in the .travis.yml file. My understanding is that the latter is needed to cache R packages.

6

So something like:

--- 
addons: 
  apt: 
    packages: 
      - libgit2-dev
  homebrew: 
    packages: 
      - libgit2
cache: 
  - packages
  - 
    directories: 
      - externalobjects
      - $HOME/Library/Caches/Homebrew
      - /usr/local/Homebrew
jobs: 
  include: 
    - 
      os: linux
      r: release
    - 
      os: linux
      r: devel
    - 
      os: linux
      r: "4.0.0"
    - 
      os: osx
      osx_image: xcode12.2
      r: release
    - 
      os: linux
      r: release
      r_packages: 
        - covr
      script: 
        - "travis_wait 30 Rscript -e 'covr::codecov(type=c(\"tests\",\"examples\"))'"
      stage: coverage
    - 
      deploy: 
        file: 
          - pomp_*.tgz
          - pomp_*.tar.gz
        file_glob: true
        provider: releases
        skip_cleanup: true
        token: 
          secure: YL3RZDogJaNmBi+jA9cxLLka5i4NQrNT3gM44ssgWAYh01NrmoIU0m0Lm/Q+7PLOebOSq2KHKU8mZDQ3CGXcLWTJfBgUr4deGeVGLsYT7Y0kQMFDzbYJkRe3UbTMGJSmwwJAqRigDP8HZmuw5/TJnKy0mLUYt97nmSzi7+vdnUUrkMRdmh+12mEM9FUEPP3DrRngzOOHBT3ZfD4MWUsLvaLltUSjweFOT1mwdeAUCDZGcfXb7Ian16ihb/FZ823cLq/elIynF7EenjzkP58k1r9vO3O8iBJ530GT/XDI9Fcev6NVD1rzznSR9+gPsVu1tFFabUXO2uHVFRzOCg8k7Pbn7/K7QbW9ctPKPO5TXKAQ96r6di0WaJBhrcINt9ZxXILxM8zOwQyxnaeJxobrnmMGrge0ASNh4KTRLWTEqcU12Ulh4wmZbaGPYSp41Q15mKCxPrFUEwWSqYdYS6kySl8CEv8vMAhjpW0Es68E8CYlxVnCILryn4Kl/b7YxOybFLNqg8hoFr3aaXz4uYR7Bb5P2FRdOAoZy09uNImDQbVpdUgt7vVwN4QjmERVsVjRiJzHYe9UrzM1rMtXDyB+joEEW2FYMleRSC8au8cp0KJj2emzcb+XoukNNsfW5qmlT+L/N6Ye1dgnqmPohmQSiYXo4s+Z5TkDcEMGuh6ZXAg=
        true: 
          tags: true
      os: osx
      osx_image: xcode12.2
      r: release
      script: 
        - "make binary"
      stage: deploy
language: r
latex: false
pandoc: false
r_packages: 
  - roxygen2
  - nloptr
stages: 
  - 
    if: "tag IS blank"
    name: test
  - 
    if: "tag IS blank"
    name: coverage
  - 
    if: "tag IS present"
    name: deploy
warnings_are_errors: true
1 Like
7

Brilliant! And instructive. Thanks!

1 Like
8

Hey @kingaa,

Glad this could be of help to you.

Montana Mendy
Travis CI Staff

1 Like
9

@montana! thank u a lot, had similar issue with my r build running macOS build and adding cache lines, worked perfectly!

10

Great to hear, I’m glad they helped you as well.

11

@Montana: With these changes, I am able to reduce the build time from >50min to ~25min. However, I wonder whether it is in principle to get the macos build to run approximately as fast as the linux build (~7min). Here is the current .travis.yml:

language: r
latex: false
pandoc: false
warnings_are_errors: true

r_packages:
  - roxygen2
  - nloptr

addons:
  apt:
    packages:
    - libgit2-dev
  homebrew:
    packages:
    - libgit2

stages:
- name: test
  if: tag IS blank
- name: coverage
  if: tag IS blank
- name: deploy
  if: tag IS present

jobs:
  include:
  - r: release
    os: linux
    cache: packages
  - r: devel
    os: linux
    cache: packages
  - r: 4.0.0
    os: linux
    cache: packages
  - r: release
    os: osx
    osx_image: xcode12.2
    cache:
      - directories: 
        - externalobjects
        - $HOME/Library/Caches/Homebrew
        - /usr/local/Cellar
        - /usr/local/gfortran
        - $HOME/R
  - stage: coverage
    r: release
    os: linux
    r_packages:
    - covr
    script:
    - travis_wait 30 Rscript -e 'covr::codecov(type=c("tests","examples"))'
    cache: packages
  - stage: deploy
    r: release
    os: osx
    osx_image: xcode12.2
    cache:
      - directories: 
        - externalobjects
        - $HOME/Library/Caches/Homebrew
        - /usr/local/Cellar
        - /usr/local/gfortran
        - $HOME/R
    script:
    - make binary
    deploy:
      provider: releases
      token:
        secure: YL3RZDogJaNmBi+jA9cxLLka5i4NQrNT3gM44ssgWAYh01NrmoIU0m0Lm/Q+7PLOebOSq2KHKU8mZDQ3CGXcLWTJfBgUr4deGeVGLsYT7Y0kQMFDzbYJkRe3UbTMGJSmwwJAqRigDP8HZmuw5/TJnKy0mLUYt97nmSzi7+vdnUUrkMRdmh+12mEM9FUEPP3DrRngzOOHBT3ZfD4MWUsLvaLltUSjweFOT1mwdeAUCDZGcfXb7Ian16ihb/FZ823cLq/elIynF7EenjzkP58k1r9vO3O8iBJ530GT/XDI9Fcev6NVD1rzznSR9+gPsVu1tFFabUXO2uHVFRzOCg8k7Pbn7/K7QbW9ctPKPO5TXKAQ96r6di0WaJBhrcINt9ZxXILxM8zOwQyxnaeJxobrnmMGrge0ASNh4KTRLWTEqcU12Ulh4wmZbaGPYSp41Q15mKCxPrFUEwWSqYdYS6kySl8CEv8vMAhjpW0Es68E8CYlxVnCILryn4Kl/b7YxOybFLNqg8hoFr3aaXz4uYR7Bb5P2FRdOAoZy09uNImDQbVpdUgt7vVwN4QjmERVsVjRiJzHYe9UrzM1rMtXDyB+joEEW2FYMleRSC8au8cp0KJj2emzcb+XoukNNsfW5qmlT+L/N6Ye1dgnqmPohmQSiYXo4s+Z5TkDcEMGuh6ZXAg=
      file_glob: true
      file:
      - pomp_*.tgz
      - pomp_*.tar.gz
      skip_cleanup: true
      on:
        tags: true

You can see the results here. It seems that quite a bit of time (~400sec) is spent installing R, despite the fact that I have asked to cache $HOME/R. At least, that was my thought in including $HOME/R in the cache. Perhaps that needs to be done differently?