You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Beckerle, Mike" <mb...@owlcyberdefense.com> on 2021/05/18 12:58:30 UTC
Windows debug clues needed
My PR https://github.com/apache/daffodil/pull/560
Keeps failing its tests on MS-Windows.
I am unable to reproduce the failures on Linux obviously.
But... I am also unable to reproduce these failures on MS-Windows.
I have installed sbt, git, intellij idea, emacs, etc. all on MS-Windows. When I run the tests via sbt test... they all pass.
So one possibility is that I have git configured differently (w.r.t the autocrlf stuff) than is done by the windows CI systems.
Suggestions?
Mike Beckerle | Principal Engineer
[cid:a2c52957-7bf9-4dbc-b1c3-17e950b58bf8]
mbeckerle@owlcyberdefense.com<ma...@owlcyberdefense.com>
P +1-781-330-0412
Re: Windows debug clues needed
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
So some of these errors seem to be font & charset related.
I installed the japanese language pack, so I have unicode character capability, but perhaps I have to set something up to tell it to use unicode and a sensible font which has the Kanji characters that are in our tests, for example.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Tuesday, May 18, 2021 12:35 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: Windows debug clues needed
I'm not sure where it exists on windows, but if you run
git config --global core.autocrlf true
then it should enable it globally. I think you could also try
reinstallting git for Windows. Last I installed git on Windows (a long
time ago) I think the installer let me pick how to handle line endings.
I'm not sure if it explicitly mentions the autocrlf option though, or if
it even gives that option anymore.
On 5/18/21 12:30 PM, Beckerle, Mike wrote:
> The XML comments was definitely part of it. Found that one and fixed it with the override def comment(..) as you suggest.
>
> There are still other tests failing though.
>
> I have been spinning up a windows dev environment and I noticed that the comments contain CRLFs on windows, and likely due to autoCRLF stuff, do not on Linux.
>
> Where do I find this autoCRLF setting? The .gitconfig in my linux home doesn't have an autocrlf setting. And I am not finding one on Windows, though not sure entirely where that would live on Windows.
>
>
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Tuesday, May 18, 2021 11:57 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Windows debug clues needed
>
> Finished the review and I didn't find anything, but I just noticed that
> some of the failed TMDL tests have infosets that include comments, that
> include newlines.
>
> I'm guessing these comments aren't stripped out by the loader and also
> aren't normalized. So when git autocrlf kicks in, it changes these
> comments to have \r\n, we don't normalize that, and the the infoset does
> contain \r's and we error.
>
> It looks like the ConstructingParser has a 'def comment' function, so
> maybe we just need to override that to normalize the comment contents?
>
> - Steve
>
>
> On 5/18/21 8:58 AM, Beckerle, Mike wrote:
>> My PR https://github.com/apache/daffodil/pull/560
>> <https://github.com/apache/daffodil/pull/560>
>>
>> Keeps failing its tests on MS-Windows.
>>
>> I am unable to reproduce the failures on Linux obviously.
>>
>> But... I am also unable to reproduce these failures on MS-Windows.
>>
>> I have installed sbt, git, intellij idea, emacs, etc. all on MS-Windows. When I
>> run the tests via sbt test... they all pass.
>>
>> So one possibility is that I have git configured differently (w.r.t the autocrlf
>> stuff) than is done by the windows CI systems.
>>
>> Suggestions?
>>
>>
>>
>>
>> Mike Beckerle | Principal Engineer
>>
>> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
>>
>> P +1-781-330-0412
>>
>
>
Re: Windows debug clues needed
Posted by Steve Lawrence <sl...@apache.org>.
I'm not sure where it exists on windows, but if you run
git config --global core.autocrlf true
then it should enable it globally. I think you could also try
reinstallting git for Windows. Last I installed git on Windows (a long
time ago) I think the installer let me pick how to handle line endings.
I'm not sure if it explicitly mentions the autocrlf option though, or if
it even gives that option anymore.
On 5/18/21 12:30 PM, Beckerle, Mike wrote:
> The XML comments was definitely part of it. Found that one and fixed it with the override def comment(..) as you suggest.
>
> There are still other tests failing though.
>
> I have been spinning up a windows dev environment and I noticed that the comments contain CRLFs on windows, and likely due to autoCRLF stuff, do not on Linux.
>
> Where do I find this autoCRLF setting? The .gitconfig in my linux home doesn't have an autocrlf setting. And I am not finding one on Windows, though not sure entirely where that would live on Windows.
>
>
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Tuesday, May 18, 2021 11:57 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Windows debug clues needed
>
> Finished the review and I didn't find anything, but I just noticed that
> some of the failed TMDL tests have infosets that include comments, that
> include newlines.
>
> I'm guessing these comments aren't stripped out by the loader and also
> aren't normalized. So when git autocrlf kicks in, it changes these
> comments to have \r\n, we don't normalize that, and the the infoset does
> contain \r's and we error.
>
> It looks like the ConstructingParser has a 'def comment' function, so
> maybe we just need to override that to normalize the comment contents?
>
> - Steve
>
>
> On 5/18/21 8:58 AM, Beckerle, Mike wrote:
>> My PR https://github.com/apache/daffodil/pull/560
>> <https://github.com/apache/daffodil/pull/560>
>>
>> Keeps failing its tests on MS-Windows.
>>
>> I am unable to reproduce the failures on Linux obviously.
>>
>> But... I am also unable to reproduce these failures on MS-Windows.
>>
>> I have installed sbt, git, intellij idea, emacs, etc. all on MS-Windows. When I
>> run the tests via sbt test... they all pass.
>>
>> So one possibility is that I have git configured differently (w.r.t the autocrlf
>> stuff) than is done by the windows CI systems.
>>
>> Suggestions?
>>
>>
>>
>>
>> Mike Beckerle | Principal Engineer
>>
>> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
>>
>> P +1-781-330-0412
>>
>
>
Re: Windows debug clues needed
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
The XML comments was definitely part of it. Found that one and fixed it with the override def comment(..) as you suggest.
There are still other tests failing though.
I have been spinning up a windows dev environment and I noticed that the comments contain CRLFs on windows, and likely due to autoCRLF stuff, do not on Linux.
Where do I find this autoCRLF setting? The .gitconfig in my linux home doesn't have an autocrlf setting. And I am not finding one on Windows, though not sure entirely where that would live on Windows.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Tuesday, May 18, 2021 11:57 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: Windows debug clues needed
Finished the review and I didn't find anything, but I just noticed that
some of the failed TMDL tests have infosets that include comments, that
include newlines.
I'm guessing these comments aren't stripped out by the loader and also
aren't normalized. So when git autocrlf kicks in, it changes these
comments to have \r\n, we don't normalize that, and the the infoset does
contain \r's and we error.
It looks like the ConstructingParser has a 'def comment' function, so
maybe we just need to override that to normalize the comment contents?
- Steve
On 5/18/21 8:58 AM, Beckerle, Mike wrote:
> My PR https://github.com/apache/daffodil/pull/560
> <https://github.com/apache/daffodil/pull/560>
>
> Keeps failing its tests on MS-Windows.
>
> I am unable to reproduce the failures on Linux obviously.
>
> But... I am also unable to reproduce these failures on MS-Windows.
>
> I have installed sbt, git, intellij idea, emacs, etc. all on MS-Windows. When I
> run the tests via sbt test... they all pass.
>
> So one possibility is that I have git configured differently (w.r.t the autocrlf
> stuff) than is done by the windows CI systems.
>
> Suggestions?
>
>
>
>
> Mike Beckerle | Principal Engineer
>
> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
>
> P +1-781-330-0412
>
Re: Windows debug clues needed
Posted by Steve Lawrence <sl...@apache.org>.
Finished the review and I didn't find anything, but I just noticed that
some of the failed TMDL tests have infosets that include comments, that
include newlines.
I'm guessing these comments aren't stripped out by the loader and also
aren't normalized. So when git autocrlf kicks in, it changes these
comments to have \r\n, we don't normalize that, and the the infoset does
contain \r's and we error.
It looks like the ConstructingParser has a 'def comment' function, so
maybe we just need to override that to normalize the comment contents?
- Steve
On 5/18/21 8:58 AM, Beckerle, Mike wrote:
> My PR https://github.com/apache/daffodil/pull/560
> <https://github.com/apache/daffodil/pull/560>
>
> Keeps failing its tests on MS-Windows.
>
> I am unable to reproduce the failures on Linux obviously.
>
> But... I am also unable to reproduce these failures on MS-Windows.
>
> I have installed sbt, git, intellij idea, emacs, etc. all on MS-Windows. When I
> run the tests via sbt test... they all pass.
>
> So one possibility is that I have git configured differently (w.r.t the autocrlf
> stuff) than is done by the windows CI systems.
>
> Suggestions?
>
>
>
>
> Mike Beckerle | Principal Engineer
>
> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
>
> P +1-781-330-0412
>
Re: Windows debug clues needed
Posted by Steve Lawrence <sl...@apache.org>.
I'm reviewing the PR now, I'll let you know if I see anything that jumps
out at me.
My guess is this probably an autocrlf issue. It's possible your windows
git installation didn't set this to true so git isn't replacing LF with
CRLF. If you change the core.autocrlf git config option to true and
reclone the repo, you might be able to reproduce the issue.
On 5/18/21 8:58 AM, Beckerle, Mike wrote:
> My PR https://github.com/apache/daffodil/pull/560
> <https://github.com/apache/daffodil/pull/560>
>
> Keeps failing its tests on MS-Windows.
>
> I am unable to reproduce the failures on Linux obviously.
>
> But... I am also unable to reproduce these failures on MS-Windows.
>
> I have installed sbt, git, intellij idea, emacs, etc. all on MS-Windows. When I
> run the tests via sbt test... they all pass.
>
> So one possibility is that I have git configured differently (w.r.t the autocrlf
> stuff) than is done by the windows CI systems.
>
> Suggestions?
>
>
>
>
> Mike Beckerle | Principal Engineer
>
> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
>
> P +1-781-330-0412
>