GCS uploads do not preserve directory structure?

I’m building a pipeline for a static site that is served out of GCS. The job uploads the files directly to the locations in the bucket serving the site. Due to the nature of static sites, it’s pretty critical that the files are put in the same directory as they are in the build. However, during deployment, it seems that all the files in my build directory were dropped directly into the root directory of my GCS bucket. Is this behavior expected? If it is, would there be a workaround for this?

I am using the v2 of deployment.

This error seems to exclusively be in v2. I switched off edge and my file structure was preserved.

Please link to a build showcasing the problem and, if possible, a listing of your bucket or something to showcase the actual result.

build: https://travis-ci.org/yiwensong/yiws_io/builds/591369009

bucket contents (all in root dir): AssetManifest.json CupertinoIcons.ttf FontManifest.json LICENSE MaterialIcons-Regular.ttf index.html main.dart.js main.dart.js.map main_web_entrypoint.dart.js.deps

In the original build, AssetManifest.json, FontManifest.json and LICENSE were all in the assets/ directory, and the ttf files were in even deeper directories.

This should also be easily reproducible by creating a git directory:
root
> build
> > file1
> > directory
> > > file2

and uploading the build directory with the legacy API and the edge API.

1 Like

Confirmed.

According to https://travis-ci.org/yiwensong/yiws_io/jobs/591369010#L627-L666 and https://github.com/travis-ci/dpl/blob/master/lib/dpl/providers/gcs.rb#L63, Travis searches the subtree for files and uploads each one individually. While there’s absolutely no need for this because gsutil cp -r is used.

I’ve made a potential fix but cannot test it because I don’t use Google Cloud.

@yiwensong could you test it for me? You need to add the following to the deploy: clause instead of edge: true:

edge:
  source: 'native-api/dpl'
  branch: 'fix_gcs' 

Thanks for looking into this, @native-api. I think your fix probably works if people do not care about content type, encoding, and cache control headers. These are part of the api though, which is why we need to process/upload files individually.

I believe this should work though: https://github.com/travis-ci/dpl/pull/1111

@yiwensong if you want to try this out you can opt in using:

deploy:
  provider: gcs
  edge:
    branch: master
1 Like

Hey!

Thanks for all the proposed fixes. I probably won’t get around to it soon. I will report back when I do get to it!

@yiwensong sounds good. i have merged the PR now after more testing.

hi @svenfuchs, I’m still getting the old behavior of all files in the root directory. Should the change be in production, as it’s on master?

The build is here: https://travis-ci.org/yiwensong/yiws_io/builds/592385520

ah, I see, I have to specify the master branch. It worked! Thanks!

Glad to hear this is working for you @yiwensong, thanks for reporting back!

You’re right, it was still on master only. I have now cut a new release, so you could move back to edge: true.