You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Zhijie Shen <zs...@hortonworks.com> on 2013/08/01 22:31:23 UTC
Re: git line endings trouble since recent trunk
The problem seems to happen again on the latest trunk
On Mon, Jul 1, 2013 at 11:44 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> On Mon, Jul 1, 2013 at 11:09 AM, Raja Aluri <ra...@cmbasics.com> wrote:
> > They are not that many that I can think of unless you are using notepad
> for
> > editing, but some of the windows related files might require CRLF. This
> can
> > be handled in .gitattributes
> > I think it's a very good idea to add this check to test-patch script and
> > reject the patch based on the CRLF check.
> > -Raja
>
> I don't see why you would need a hook. Just set svn:eol-style=crlf on
> the file that need CRLF.
>
> >
> >
> > On Mon, Jul 1, 2013 at 11:00 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> why not just add a precommit hook in svn to reject commits with CRLF?
> >>
> >>
> >> On Mon, Jul 1, 2013 at 10:51 AM, Raja Aluri <ra...@cmbasics.com> wrote:
> >>
> >> > I added a couple of links that discusses 'line endings' when I added
> >> > .gitattributes in this JIRA.
> >> > HADOOP-8912<https://issues.apache.org/jira/browse/HADOOP-8912>
> >> >
> >> > I am just reproducing them here.
> >> >
> >> > 1.
> >> http://git-scm.com/docs/gitattributes#_checking_out_and_checking_in
> >> > 2.
> >> >
> >> >
> >>
> http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git
>
> Regardless of what we do or don't do in git, we should have the line
> endings correct in subversion.
>
> cheers.
> Colin
>
>
> >> >
> >> > --
> >> > Raja
> >> >
> >> >
> >> > On Mon, Jul 1, 2013 at 10:42 AM, Colin McCabe <cmccabe@alumni.cmu.edu
> >> > >wrote:
> >> >
> >> > > On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu <ll...@vicaya.com> wrote:
> >> > > > The problem is due to relnotes.py generating the html containing
> some
> >> > > CRLF
> >> > > > (from JIRA) and the release manager not using git-svn, which
> caused
> >> the
> >> > > > html with mixed eol getting checked in. The problem would then
> >> manifest
> >> > > for
> >> > > > git users due to text=auto in .gitattributes (see HADOOP-8912)
> that
> >> > auto
> >> > > > converts CRLF to LF, hence the persistent modified status of the
> >> > > > releasenotes.html for a fresh checkout.
> >> > > >
> >> > > > Adding svn:eol-style=native would only fix half the problem. We
> need
> >> to
> >> > > fix
> >> > > > relnotes.py to avoid generating html with mixed eols (by
> normalizing
> >> > > > everything to LF or native).
> >> > >
> >> > > While I agree that it would be nice to fix relnotes.py, it seems to
> me
> >> > > that setting svn:eol-style=native should fix the problem completely.
> >> > > Files with this attribute set are stored internally by subversion
> with
> >> > > all newlines as LF, and converted to CRLF as needed. After all,
> >> > > eol-style=native would not be very useful if it only applied on
> >> > > checkout. Windows users would be constantly checking in CRLF in
> that
> >> > > case.
> >> > >
> >> > > I'm not an svn expert, though, and I haven't tested the above.
> >> > >
> >> > > Colin
> >> > >
> >> > >
> >> > > >
> >> > > >
> >> > > > On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe <
> >> cmccabe@alumni.cmu.edu
> >> > > >wrote:
> >> > > >
> >> > > >> Clarification: svn:eol-style = native causes the files to contain
> >> > > >> whatever the native platform used to check out the code uses. I
> >> think
> >> > > >> just setting this property on all the HTML files should resolve
> this
> >> > > >> and future problems.
> >> > > >>
> >> > > >> patch posted.
> >> > > >> C.
> >> > > >>
> >> > > >> On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe <
> >> > cmccabe@alumni.cmu.edu>
> >> > > >> wrote:
> >> > > >> > I think the fix for this is to set svn:eol-style to "native" on
> >> this
> >> > > >> > file. It's set on many other files, just not on this one:
> >> > > >> >
> >> > > >> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> >> > > >> > ./hadoop-project-dist/README.txt
> >> > > >> > native
> >> > > >> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> >> > > >> >
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
> >> > > >> > cmccabe@keter:~/hadoopST/trunk>
> >> > > >> >
> >> > > >> > It might also be a good idea to run dos2unix on it.
> >> > > >> >
> >> > > >> > I thought that in general we wanted to have 'LF' everywhere, so
> >> > > >> > perhaps we should add this to the patch.sh script to prevent
> this
> >> > from
> >> > > >> > re-occurring.
> >> > > >> >
> >> > > >> > Colin
> >> > > >> >
> >> > > >> >
> >> > > >> > On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza <
> >> > sandy.ryza@cloudera.com
> >> > > >
> >> > > >> wrote:
> >> > > >> >> I haven't been able to find a solution. Just filed
> >> > > >> >> https://issues.apache.org/jira/browse/HADOOP-9675.
> >> > > >> >>
> >> > > >> >>
> >> > > >> >> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi <
> >> > > ojoshi@hortonworks.com
> >> > > >> >wrote:
> >> > > >> >>
> >> > > >> >>> Sandy... did you fix this? any jira to track? me too facing
> same
> >> > > >> problem..
> >> > > >> >>>
> >> > > >> >>> Thanks,
> >> > > >> >>> Omkar Joshi
> >> > > >> >>> *Hortonworks Inc.* <http://www.hortonworks.com>
> >> > > >> >>>
> >> > > >> >>>
> >> > > >> >>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen <
> >> > > zshen@hortonworks.com>
> >> > > >> >>> wrote:
> >> > > >> >>>
> >> > > >> >>> > Have the same problem here, have to edit the patch
> manually to
> >> > > >> exclude
> >> > > >> >>> the
> >> > > >> >>> > changes in releasenotes.html
> >> > > >> >>> >
> >> > > >> >>> >
> >> > > >> >>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza <
> >> > > >> sandy.ryza@cloudera.com
> >> > > >> >>> > >wrote:
> >> > > >> >>> >
> >> > > >> >>> > > Has anybody else been having trouble with line endings
> since
> >> > > >> pulling
> >> > > >> >>> > trunk
> >> > > >> >>> > > recently?
> >> > > >> >>> > >
> >> > > >> >>> > >
> >> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
> >> > > >> >>> > > shows up as modified even though I haven't touched it,
> and I
> >> > > can't
> >> > > >> >>> check
> >> > > >> >>> > it
> >> > > >> >>> > > out or reset to a previous version to make that go away.
> >> The
> >> > > only
> >> > > >> >>> thing
> >> > > >> >>> > I
> >> > > >> >>> > > can do to neutralize it is to put it in a dummy commit,
> but
> >> I
> >> > > have
> >> > > >> to
> >> > > >> >>> do
> >> > > >> >>> > > this every time I switch branches or rebase.
> >> > > >> >>> > >
> >> > > >> >>> > > This appears to have began after the release notes commit
> >> > > >> >>> > > (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be
> due
> >> > to
> >> > > a
> >> > > >> line
> >> > > >> >>> > > endings change.
> >> > > >> >>> > >
> >> > > >> >>> > > -Sandy
> >> > > >> >>> > >
> >> > > >> >>> >
> >> > > >> >>> >
> >> > > >> >>> >
> >> > > >> >>> > --
> >> > > >> >>> > Zhijie Shen
> >> > > >> >>> > Hortonworks Inc.
> >> > > >> >>> > http://hortonworks.com/
> >> > > >> >>> >
> >> > > >> >>>
> >> > > >>
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
Re: git line endings trouble since recent trunk
Posted by Robert Parker <rp...@yahoo-inc.com>.
Yes on
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html.
This does not appear to affect devs in my group with older git 1.7.1, but
it does affect git 1.8.1.2 and fiddling with the options autocrlf and
safecrlf could not produce a combination that prevents it from changing
the line endings to LF on checkout.
On 8/1/13 3:31 PM, "Zhijie Shen" <zs...@hortonworks.com> wrote:
>The problem seems to happen again on the latest trunk
>
>
>On Mon, Jul 1, 2013 at 11:44 AM, Colin McCabe
><cm...@alumni.cmu.edu>wrote:
>
>> On Mon, Jul 1, 2013 at 11:09 AM, Raja Aluri <ra...@cmbasics.com> wrote:
>> > They are not that many that I can think of unless you are using
>>notepad
>> for
>> > editing, but some of the windows related files might require CRLF.
>>This
>> can
>> > be handled in .gitattributes
>> > I think it's a very good idea to add this check to test-patch script
>>and
>> > reject the patch based on the CRLF check.
>> > -Raja
>>
>> I don't see why you would need a hook. Just set svn:eol-style=crlf on
>> the file that need CRLF.
>>
>> >
>> >
>> > On Mon, Jul 1, 2013 at 11:00 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>> >
>> >> why not just add a precommit hook in svn to reject commits with CRLF?
>> >>
>> >>
>> >> On Mon, Jul 1, 2013 at 10:51 AM, Raja Aluri <ra...@cmbasics.com>
>>wrote:
>> >>
>> >> > I added a couple of links that discusses 'line endings' when I
>>added
>> >> > .gitattributes in this JIRA.
>> >> > HADOOP-8912<https://issues.apache.org/jira/browse/HADOOP-8912>
>> >> >
>> >> > I am just reproducing them here.
>> >> >
>> >> > 1.
>> >> http://git-scm.com/docs/gitattributes#_checking_out_and_checking_in
>> >> > 2.
>> >> >
>> >> >
>> >>
>>
>>http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-st
>>rategy-with-git
>>
>> Regardless of what we do or don't do in git, we should have the line
>> endings correct in subversion.
>>
>> cheers.
>> Colin
>>
>>
>> >> >
>> >> > --
>> >> > Raja
>> >> >
>> >> >
>> >> > On Mon, Jul 1, 2013 at 10:42 AM, Colin McCabe
>><cmccabe@alumni.cmu.edu
>> >> > >wrote:
>> >> >
>> >> > > On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu <ll...@vicaya.com> wrote:
>> >> > > > The problem is due to relnotes.py generating the html
>>containing
>> some
>> >> > > CRLF
>> >> > > > (from JIRA) and the release manager not using git-svn, which
>> caused
>> >> the
>> >> > > > html with mixed eol getting checked in. The problem would then
>> >> manifest
>> >> > > for
>> >> > > > git users due to text=auto in .gitattributes (see HADOOP-8912)
>> that
>> >> > auto
>> >> > > > converts CRLF to LF, hence the persistent modified status of
>>the
>> >> > > > releasenotes.html for a fresh checkout.
>> >> > > >
>> >> > > > Adding svn:eol-style=native would only fix half the problem. We
>> need
>> >> to
>> >> > > fix
>> >> > > > relnotes.py to avoid generating html with mixed eols (by
>> normalizing
>> >> > > > everything to LF or native).
>> >> > >
>> >> > > While I agree that it would be nice to fix relnotes.py, it seems
>>to
>> me
>> >> > > that setting svn:eol-style=native should fix the problem
>>completely.
>> >> > > Files with this attribute set are stored internally by subversion
>> with
>> >> > > all newlines as LF, and converted to CRLF as needed. After all,
>> >> > > eol-style=native would not be very useful if it only applied on
>> >> > > checkout. Windows users would be constantly checking in CRLF in
>> that
>> >> > > case.
>> >> > >
>> >> > > I'm not an svn expert, though, and I haven't tested the above.
>> >> > >
>> >> > > Colin
>> >> > >
>> >> > >
>> >> > > >
>> >> > > >
>> >> > > > On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe <
>> >> cmccabe@alumni.cmu.edu
>> >> > > >wrote:
>> >> > > >
>> >> > > >> Clarification: svn:eol-style = native causes the files to
>>contain
>> >> > > >> whatever the native platform used to check out the code uses.
>> I
>> >> think
>> >> > > >> just setting this property on all the HTML files should
>>resolve
>> this
>> >> > > >> and future problems.
>> >> > > >>
>> >> > > >> patch posted.
>> >> > > >> C.
>> >> > > >>
>> >> > > >> On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe <
>> >> > cmccabe@alumni.cmu.edu>
>> >> > > >> wrote:
>> >> > > >> > I think the fix for this is to set svn:eol-style to
>>"native" on
>> >> this
>> >> > > >> > file. It's set on many other files, just not on this one:
>> >> > > >> >
>> >> > > >> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
>> >> > > >> > ./hadoop-project-dist/README.txt
>> >> > > >> > native
>> >> > > >> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
>> >> > > >> >
>> ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
>> >> > > >> > cmccabe@keter:~/hadoopST/trunk>
>> >> > > >> >
>> >> > > >> > It might also be a good idea to run dos2unix on it.
>> >> > > >> >
>> >> > > >> > I thought that in general we wanted to have 'LF'
>>everywhere, so
>> >> > > >> > perhaps we should add this to the patch.sh script to prevent
>> this
>> >> > from
>> >> > > >> > re-occurring.
>> >> > > >> >
>> >> > > >> > Colin
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza <
>> >> > sandy.ryza@cloudera.com
>> >> > > >
>> >> > > >> wrote:
>> >> > > >> >> I haven't been able to find a solution. Just filed
>> >> > > >> >> https://issues.apache.org/jira/browse/HADOOP-9675.
>> >> > > >> >>
>> >> > > >> >>
>> >> > > >> >> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi <
>> >> > > ojoshi@hortonworks.com
>> >> > > >> >wrote:
>> >> > > >> >>
>> >> > > >> >>> Sandy... did you fix this? any jira to track? me too
>>facing
>> same
>> >> > > >> problem..
>> >> > > >> >>>
>> >> > > >> >>> Thanks,
>> >> > > >> >>> Omkar Joshi
>> >> > > >> >>> *Hortonworks Inc.* <http://www.hortonworks.com>
>> >> > > >> >>>
>> >> > > >> >>>
>> >> > > >> >>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen <
>> >> > > zshen@hortonworks.com>
>> >> > > >> >>> wrote:
>> >> > > >> >>>
>> >> > > >> >>> > Have the same problem here, have to edit the patch
>> manually to
>> >> > > >> exclude
>> >> > > >> >>> the
>> >> > > >> >>> > changes in releasenotes.html
>> >> > > >> >>> >
>> >> > > >> >>> >
>> >> > > >> >>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza <
>> >> > > >> sandy.ryza@cloudera.com
>> >> > > >> >>> > >wrote:
>> >> > > >> >>> >
>> >> > > >> >>> > > Has anybody else been having trouble with line endings
>> since
>> >> > > >> pulling
>> >> > > >> >>> > trunk
>> >> > > >> >>> > > recently?
>> >> > > >> >>> > >
>> >> > > >> >>> > >
>> >> > >
>>hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
>> >> > > >> >>> > > shows up as modified even though I haven't touched it,
>> and I
>> >> > > can't
>> >> > > >> >>> check
>> >> > > >> >>> > it
>> >> > > >> >>> > > out or reset to a previous version to make that go
>>away.
>> >> The
>> >> > > only
>> >> > > >> >>> thing
>> >> > > >> >>> > I
>> >> > > >> >>> > > can do to neutralize it is to put it in a dummy
>>commit,
>> but
>> >> I
>> >> > > have
>> >> > > >> to
>> >> > > >> >>> do
>> >> > > >> >>> > > this every time I switch branches or rebase.
>> >> > > >> >>> > >
>> >> > > >> >>> > > This appears to have began after the release notes
>>commit
>> >> > > >> >>> > > (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must
>>be
>> due
>> >> > to
>> >> > > a
>> >> > > >> line
>> >> > > >> >>> > > endings change.
>> >> > > >> >>> > >
>> >> > > >> >>> > > -Sandy
>> >> > > >> >>> > >
>> >> > > >> >>> >
>> >> > > >> >>> >
>> >> > > >> >>> >
>> >> > > >> >>> > --
>> >> > > >> >>> > Zhijie Shen
>> >> > > >> >>> > Hortonworks Inc.
>> >> > > >> >>> > http://hortonworks.com/
>> >> > > >> >>> >
>> >> > > >> >>>
>> >> > > >>
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Alejandro
>> >>
>>
>
>
>
>--
>Zhijie Shen
>Hortonworks Inc.
>http://hortonworks.com/