Mount a private r/o volume with a large testsuite

I have an open source project that I’d like to start CI for. I have a testsuite that consists of a large number of documents I can’t publish, as I don’t have rights for distributing them.

I’d like to make this testsuite available for test workers. Is it possible to do so? E.g. a r/o EBS (or similar) volume that could be mounted, with credentials stored in secret storage.

Sure, use secret (aka encrypted) variables for credentials (either in .travis.yml or Traivis settings for the repo).

Note though that secret variables are not available to PR builds (because code in them is not trusted so anything available to one of those is effectively public).

The problem is not in credentials. The problem is in attaching a large (tens of gigabytes) volume to a build. How do I do that?

Whatever you do, it boils down to getting this data onto the test machine somehow. There’s no way around unless you run part of the logic on some other infrastructure.

So – use any measures that would allow you to reduce the amount of data transferred over the wire and/or maximize the speed:

  • Aggressively compress/otherwise optimize the data
  • Move the storage closer to Travis’ data centers if that makes a difference (they are in the East US AFAIK, trace their IPs)
  • Transfer the data in chunks in parallel with testing
  • Split testing into multiple jobs and transfer only part of the data to each

For Travis’ cache, tens of gigabytes are a bit too much. It will necessitate using a prolonged timeout: and may not even be any faster than transferring from your storage. In any case, your logic has to be able to work without the cache, too, anyway.

Due to a long transfer, you may need to log progress so as not to trigger automatic termination of a build due to no output for 10 minutes.

I see.

I wondered if Travis is hosted on a public cloud, then I could create a volume that can be directly mounted.

Linux VMs are currently hosted on the Google Compute Engine. But whatever you use to “mount” it, it still boils down to transferring the data.

I am trying to avoid transferring this data. I could create a disk on GCE and give permissions to use it. But alas it needs a feature in Travis to mount the disk to the VM and mount it into the container.

AFAICS you can use gscfuse to mount a Google Cloud drive.
What I wanted to strike is that this would still involve transferring the data – it would just happen under the hood and/or on demand rather than beforehand and explicitly. And, what is more important, it would likewise cost time and bandwidth – since GCE is not one humongous computer, however hard they try to make it look otherwise.