Files in checkout have EOL changed from LF to CRLF

Environments Windows
1

The checkout is changing line endings of files. This can lead to automated checks such as go fmt failing. In my case I have testdata where the byte offset of tokens is important, leading to my tests failing.

I use the following step to adjust files back to LF

- find . -name '*.go' -type f -print0 | xargs -0 sed -i ':a;N;$!ba;s/\r//g'

I tried to unset the “text” gitattribute, which according to gits documentation should prevent automatic line ending changes. However, the issue still persisted. For reference I have the following for my .gitattributes:

# We have tests which parse files and fail if file offsets change. So disable
# EOL conversions.
**/testdata/* -text

The PR which enabled windows support is https://github.com/sourcegraph/go-langserver/pull/329
I originally mentioned this issue in another discussion Go can't find GCC with go1.11.1 on Windows

2

You might be able to do it with gitattributes, but a bit different syntax. For example:
* text=auto
*.js text eol=lf
*.sass text eol=lf
*.json text eol=lf
*.css text eol=lf
*.html text eol=lf
*.htm text eol=lf

2 Likes
3

The issue here appears to be that git is configured as follows (taken from a Windows-based build):

$ git config -l
core.symlinks=false
core.autocrlf=true
...

This feels like the wrong default (https://git-scm.com/docs/gitattributes#_checking_out_and_checking_in) because otherwise there’s a considerable amount of work required to undo this behaviour. If anything, I would suggest making the default false and then allowing people to configure it to true in the .travis.yml git: section (https://docs.travis-ci.com/user/customizing-the-build/#git-clone-depth)

My current work around is the following addition to my .travis.yml:

before_install:
  - cd ../..
  - mv $TRAVIS_REPO_SLUG _old
  - git config --global core.autocrlf false
  - git clone --depth=50 _old $TRAVIS_REPO_SLUG
  - cd $TRAVIS_REPO_SLUG

Edit to fix env var name

Edit to fix error in before_install

1 Like
4

An update on my side. I have not tried travis windows support since I originally posted this issue. We ran into other issues so I am unsure what the current status of this is.

@AviVahl hi Avi sorry I missed your response. Unsetting the text attribute should disable any EOL conversions in git, which is what I did in the gitattributes. Forcing the line endings sounds like a reasonable workaround (for text data), but either way the proper gitattribute is not being respected. According to the git documentation it should work, but maybe there is an outdated windows version in the image?

5

In rust-clippy we managed to work around this using the dos2unix utility on affected files: https://github.com/rust-lang/rust-clippy/pull/3418/files#diff-354f30a63fb0907d4ad57269548329e3R46

6

Another workaround would be to use the .gitattributes file and enforce LF for lineendings:

# ./.gitattributes

* text=auto eol=lf
1 Like
7
8

This is a dogs breakfast!

None of the solutions presented here actually work. Even the approach suggested by Paul, which should work, doesn’t because of a permissions issue; the mv that is necessary fails because of a permissions failure.

+ pushd ../../..
~/gopath/src ~/gopath/src/gonum.org/v1/gonum
+ mv gonum.org/v1/gonum .
mv: cannot move 'gonum.org/v1/gonum' to './gonum': Permission denied
+ git config --global core.autocrlf false
+ git clone --depth=1 gonum gonum.org/v1/gonum
fatal: repository 'gonum' does not exist

Essentially the same outcome is obtained with a direct transliteration of the approach in above. I suspect that this is due to the fact that the code being run is a script in gonum.org/v1/gonum/.travis/. This is necessary for the approach we are using to control multi-os testing.

What should happen here is that the default mutation of text under the user’s feet should not be happening unless it’s being asked for. The initial clone should not rewrite code; if it’s being developed on windows it will already have CRLF EOLs, if it’s cross-platform, the devs will already have a mechanism to handle differences in EOLs. This should just not be happening.

I have to say though, it does remind me why I don’t want to work on windows and harks back to the era of Clippy.

Edit: I have relented and now just unconditionally perform the reclone as described above. This works, but should not be needed.

1 Like
9

Ranting without giving any error details (a link to the build at least) is not very useful as it doesn’t give any facts that other readers can use or relate to.

I can assure you that changing core.autocrlf and doing one of the options listed on the SO link in my previous post worked for me.

That said, if your software cannot handle OS-specific EOLs that’s a bug in the software in my book (e.g. maybe you are opening text files in binary mode) 'cuz this is far from being something unexpected. autocrlf=true is the default setting in Git for Windows because that’s the EOLs that Windows software expects and produces.

10

Thank you for your comment.

The issue is that our software does handle OS-specific EOLs and we test for that; mutating text automatically and without reference to user configuration breaks our tests that assure that.

The errors in the build log were not meaningful in this context and relate to two components: that windows cannot move running scripts (breaking Paul’s approach in the context of our build system) and that (despite your claims) core.autocrlf=false does not work for us, and does not error except during our tests, which will mean little to nothing to anyone here.

I’m glad it works for you. Though note that if the software being tested were able “handle OS-specific EOLs” as you say, auto-mutating the code would not be necessary. So it seems that this approach is in place to work around buggy software. If the code is Windows-specific, it will have been written on a Windows machine and will no doubt have CRLF EOL; no mutation is necessary.

11

This has been fixed. There is now a .travis.yml attribute under git, autocrlf that allows you to specify true, false and input and uses that value to execute git config --global core.autocrlf <value>.

There is a PR to add documentation pending.

5 Likes