You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Laird Nelson <lj...@gmail.com> on 2013/01/17 22:30:11 UTC

Local modification on checkout?

Hello; we're seeing a local modification being reported on a particular
file on a clean checkout.  We're using svn 1.7.7.

The file in question has the svn:eol-style property set to native.

What I mean by this:

A fresh checkout happens (svn co ...).

Down comes the whole project.  So far so good.

Then another part of our build infrastructure does an svn status.

svn status reports that this file is locally modified (M).  There are no
intervening steps.  That is, it's checkout, then svn status.

Does automatic eol conversion show up as a local modification in svn status
output?  I can't reproduce this on either a Windows or a Mac.  It appears
to be only on this machine, which, I know, sounds mad.

Thanks for any help.

Best,
Laird

-- 
http://about.me/lairdnelson

Re: Local modification on checkout?

Posted by Laird Nelson <lj...@gmail.com>.
One specific machine.  :-(


On Thu, Jan 17, 2013 at 1:56 PM, Andy Levy <an...@gmail.com> wrote:

>
>
> On Thu, Jan 17, 2013 at 4:45 PM, Laird Nelson <lj...@gmail.com> wrote:
>
>> On Thu, Jan 17, 2013 at 1:40 PM, Andy Levy <an...@gmail.com> wrote:
>>
>>> Automatic EOL conversions, assuming they're adhering to svn:eol-style,
>>> should not trigger this.
>>>
>>
>> Glad to hear it.
>>
>> The ecosystem in question is Jenkins running on Windows, which invokes
>> the command line svn executable (packaged/shipped by SlikSvn, version
>> 1.7.7).  Nothing else is on that box (no background processes etc.).  We
>> can't reproduce it elsewhere.  I think we'll have to chalk this up to
>> gremlins as much as it thoroughly pains me.
>>
>> Is it Jenkins on Windows on one specific machine, or Jenkins on Windows
> on any machine?
>
>


-- 
http://about.me/lairdnelson

Re: Local modification on checkout?

Posted by Andy Levy <an...@gmail.com>.
On Thu, Jan 17, 2013 at 4:45 PM, Laird Nelson <lj...@gmail.com> wrote:

> On Thu, Jan 17, 2013 at 1:40 PM, Andy Levy <an...@gmail.com> wrote:
>
>> Automatic EOL conversions, assuming they're adhering to svn:eol-style,
>> should not trigger this.
>>
>
> Glad to hear it.
>
> The ecosystem in question is Jenkins running on Windows, which invokes the
> command line svn executable (packaged/shipped by SlikSvn, version 1.7.7).
>  Nothing else is on that box (no background processes etc.).  We can't
> reproduce it elsewhere.  I think we'll have to chalk this up to gremlins as
> much as it thoroughly pains me.
>
> Is it Jenkins on Windows on one specific machine, or Jenkins on Windows on
any machine?

Re: Local modification on checkout?

Posted by Laird Nelson <lj...@gmail.com>.
On Thu, Jan 17, 2013 at 2:01 PM, David Chapman <dc...@acm.org> wrote:

>  Hmm, "only the EOL changed."  Do you mean that literally, meaning there
> is only one line in the file?  If so, does the original file have a newline
> at the end of the last line?  Maybe newline conversion is adding one.
>

Thanks for your reply.  No, it's a file that has two lines in it, and the
line that is reported to have changed locally was the first one.

Best,
Laird

-- 
http://about.me/lairdnelson

Re: Local modification on checkout?

Posted by David Chapman <dc...@acm.org>.
On 1/17/2013 1:45 PM, Laird Nelson wrote:
> On Thu, Jan 17, 2013 at 1:40 PM, Andy Levy <andy.levy@gmail.com 
> <ma...@gmail.com>> wrote:
>
>     Automatic EOL conversions, assuming they're adhering to
>     svn:eol-style, should not trigger this.
>
>
> Glad to hear it.
>
> The ecosystem in question is Jenkins running on Windows, which invokes 
> the command line svn executable (packaged/shipped by SlikSvn, version 
> 1.7.7).  Nothing else is on that box (no background processes etc.). 
>  We can't reproduce it elsewhere.  I think we'll have to chalk this up 
> to gremlins as much as it thoroughly pains me.
>
> svn diff reports that only the EOL changed.
>
> Thanks for your response; if I can figure out the sequence of events 
> that tripped this I'll file a bug.
>
>

Hmm, "only the EOL changed."  Do you mean that literally, meaning there 
is only one line in the file?  If so, does the original file have a 
newline at the end of the last line?  Maybe newline conversion is adding 
one.

Just a shot in the dark...

-- 
     David Chapman      dcchapman@acm.org
     Chapman Consulting -- San Jose, CA
     Software Development Done Right.
     www.chapman-consulting-sj.com

Re: Local modification on checkout?

Posted by Laird Nelson <lj...@gmail.com>.
On Thu, Jan 17, 2013 at 1:40 PM, Andy Levy <an...@gmail.com> wrote:

> Automatic EOL conversions, assuming they're adhering to svn:eol-style,
> should not trigger this.
>

Glad to hear it.

The ecosystem in question is Jenkins running on Windows, which invokes the
command line svn executable (packaged/shipped by SlikSvn, version 1.7.7).
 Nothing else is on that box (no background processes etc.).  We can't
reproduce it elsewhere.  I think we'll have to chalk this up to gremlins as
much as it thoroughly pains me.

svn diff reports that only the EOL changed.

Thanks for your response; if I can figure out the sequence of events that
tripped this I'll file a bug.

Best,
Laird

-- 
http://about.me/lairdnelson

Re: Local modification on checkout?

Posted by Andy Levy <an...@gmail.com>.
On Thu, Jan 17, 2013 at 4:30 PM, Laird Nelson <lj...@gmail.com> wrote:

> Hello; we're seeing a local modification being reported on a particular
> file on a clean checkout.  We're using svn 1.7.7.
>
> The file in question has the svn:eol-style property set to native.
>
> What I mean by this:
>
> A fresh checkout happens (svn co ...).
>
> Down comes the whole project.  So far so good.
>
> Then another part of our build infrastructure does an svn status.
>
> svn status reports that this file is locally modified (M).  There are no
> intervening steps.  That is, it's checkout, then svn status.
>
> Does automatic eol conversion show up as a local modification in svn
> status output?  I can't reproduce this on either a Windows or a Mac.  It
> appears to be only on this machine, which, I know, sounds mad.
>
>
Automatic EOL conversions, assuming they're adhering to svn:eol-style,
should not trigger this.

If it happens on only one computer, I'd inspect that machine very
thoroughly for background processes which may be altering files. As well as
the content of the changes - can you find an explanation for the changes
reported by svn diff on that modified file?

Re: Local modification on checkout?

Posted by Laird Nelson <lj...@gmail.com>.
On Thu, Jan 17, 2013 at 2:58 PM, Bert Huijben <be...@qqmail.nl> wrote:

> A lot of users have SvnKit in their Jenkins installation. Are you sure that
> you aren't mixing a normal svn with some svnkit version?
>

Oh, that is interesting.  I will research that.

Best,
Laird

-- 
http://about.me/lairdnelson

RE: Local modification on checkout?

Posted by Bert Huijben <be...@qqmail.nl>.

> -----Original Message-----
> From: Ryan Schmidt [mailto:subversion-2012c@ryandesign.com]
> Sent: donderdag 17 januari 2013 23:46
> To: Laird Nelson
> Cc: users@subversion.apache.org
> Subject: Re: Local modification on checkout?
> 
> 
> On Jan 17, 2013, at 15:30, Laird Nelson wrote:
> 
> > Hello; we're seeing a local modification being reported on a particular
file
> on a clean checkout.  We're using svn 1.7.7.
> >
> > The file in question has the svn:eol-style property set to native.
> >
> > What I mean by this:
> >
> > A fresh checkout happens (svn co ...).
> >
> > Down comes the whole project.  So far so good.
> >
> > Then another part of our build infrastructure does an svn status.
> >
> > svn status reports that this file is locally modified (M).  There are no
> intervening steps.  That is, it's checkout, then svn status.
> >
> > Does automatic eol conversion show up as a local modification in svn
status
> output?
> 
> No, it shouldn't.
> 
> > I can't reproduce this on either a Windows or a Mac.  It appears to be
only
> on this machine, which, I know, sounds mad.
> 
> When a file has svn:eol-style set to native, the Subversion client is
supposed
> to normalize the file to LF line endings before sending it to the
repository to
> be committed. The Subversion server does not verify that this has been
> done, so it is possible for badly-written Subversion clients to commit
files
> with the wrong line endings. If a third-party svn client (git-svn?) was
used to
> commit this file, that might be a possible cause to investigate. Although
it's
> strange you only see the problem on one system.
> 
> When checking out (or updating) a working copy, if any files have svn:eol-
> style set to native, then the Subversion client transforms the line
endings of
> those files from LF to whatever svn:eol-style says to do before placing
them
> in the working copy. This can lead to unexpected situations if you check
out a
> working copy on an OS with one line ending style and then look at it or
> update it from an OS with a different line ending style. If you think this
might
> have happened, check out a new working copy, and use it only on a single
> system.
> 
> Or perhaps again if you checked out or updated using a third-party svn
client
> that did not transform line endings in response to svn:eol-style native,
then
> you might later have a problem.

A lot of users have SvnKit in their Jenkins installation. Are you sure that
you aren't mixing a normal svn with some svnkit version?

	Bert

Re: Local modification on checkout?

Posted by Johan Corveleyn <jc...@gmail.com>.
On Fri, Jan 18, 2013 at 5:27 PM, Bert Huijben <be...@qqmail.nl> wrote:
>> -----Original Message-----
>> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
>> Sent: vrijdag 18 januari 2013 11:50
>> To: Ryan Schmidt
>> Cc: Laird Nelson; users@subversion.apache.org
>> Subject: Re: Local modification on checkout?
>>
>
>
>> Further, when mixing SVNKit with native svn usage, it's possible that
>> last-mod-times are already mismatching immediately after checkout.
>> That's because of an issue in SVNKit [2], where it only notes the
>> last-mod-time in the svn metadata up to millisecond precision. So if
>> you 'svnkit co', followed by 'nativesvn status', all files will have
>> mismatching timestamps, so they'll all be checksummed.
>
> Thanks for bringing this up..
>
> I expected a problem like this, but was never able to confirm this.
> (if you have a pointer to more details, please let me know)

Yeah, it's a nasty bug, and I had to dig deep to find the ultimate
cause :-). Most details are in the SVNKit issue [1].

I started suspecting this because of exactly this LF-normalization
issue, and the fact that some of my users saw this as "M" and others
didn't. It turned out to be correlated to a feature in IntelliJ IDEA
(which normally uses SVNKit under the hood) where you can optionally
select a native client for 'update' and 'status' to make these
operations faster. Users that had the mix of SVNKit + native saw the
unexpected M.

Apart from that, I also noticed that some user's working copies were
unusually slow when 'svn stat'ed on the command line, with lots of
hard disk activity, which was a bit unexpected for svn 1.7.

It's around that time I started asking on the users-list [2] for
scripts / tools that can report on timing mismatches, and there was
some discussion on letting 'svn status' optionally report or even fix
such mismatches (fixing on the fly turned out to be a bit hard,
because 'status' would have to take a write lock to do that, but there
was an issue filed to do that optionally [3] -- but the reporting of
those mismatches would also still be very valuable I think).

In the end I used this crude patch to diagnose timestamp-problems
quickly when running status:

[[[
Index: subversion/libsvn_wc/questions.c
===================================================================
--- subversion/libsvn_wc/questions.c    (revision 1396561)
+++ subversion/libsvn_wc/questions.c    (working copy)
@@ -363,6 +363,9 @@ svn_wc__internal_file_modified_p(svn_boolean_t *mo
                                                   dirent->filesize,
                                                   dirent->mtime,
                                                   scratch_pool));
+      else
+        fprintf(stdout, "Broken TS: %s (disk: %" APR_TIME_T_FMT " -
db: %" APR_TIME_T_FMT ")\n",
+                local_abspath, dirent->mtime, recorded_mod_time);
     }

   return SVN_NO_ERROR;
]]]

> This would exactly explain this problem.
> (Just like it explained a similar problem in our testsuite yesterday. Python
> appears to have a similar problem in its file timestamps)

Hm, it surprises me that Python has the same problem, but then again,
I don't know Python enough to know.

As mentioned in the issue, with SVNKit the problem is closely related
to Java, because the standard (platform-independent)
java.io.File#lastModified() reports milliseconds. Only as of Java 7
there is (finally) another API to get low-level file attributes, but
the SVNKit people still have to implement the stuff to make use of it
to fix their timestamp handling.

Perhaps Python has a similar (platform-independent)
millisecond-precision API that's used here?

[1] http://issues.tmatesoft.com/issue/SVNKIT-315

[2] http://thread.gmane.org/gmane.comp.version-control.subversion.user/109891

[3] http://subversion.tigris.org/issues/show_bug.cgi?id=4162

-- 
Johan

RE: Local modification on checkout?

Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
> Sent: vrijdag 18 januari 2013 11:50
> To: Ryan Schmidt
> Cc: Laird Nelson; users@subversion.apache.org
> Subject: Re: Local modification on checkout?
> 


> Further, when mixing SVNKit with native svn usage, it's possible that
> last-mod-times are already mismatching immediately after checkout.
> That's because of an issue in SVNKit [2], where it only notes the
> last-mod-time in the svn metadata up to millisecond precision. So if
> you 'svnkit co', followed by 'nativesvn status', all files will have
> mismatching timestamps, so they'll all be checksummed.

Thanks for bringing this up..

I expected a problem like this, but was never able to confirm this.
(if you have a pointer to more details, please let me know)

This would exactly explain this problem.
(Just like it explained a similar problem in our testsuite yesterday. Python
appears to have a similar problem in its file timestamps)

	Bert

Re: Local modification on checkout?

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Jan 17, 2013 at 11:45 PM, Ryan Schmidt
<su...@ryandesign.com> wrote:
>
> On Jan 17, 2013, at 15:30, Laird Nelson wrote:
>
>> Hello; we're seeing a local modification being reported on a particular file on a clean checkout.  We're using svn 1.7.7.
>>
>> The file in question has the svn:eol-style property set to native.
>>
>> What I mean by this:
>>
>> A fresh checkout happens (svn co ...).
>>
>> Down comes the whole project.  So far so good.
>>
>> Then another part of our build infrastructure does an svn status.
>>
>> svn status reports that this file is locally modified (M).  There are no intervening steps.  That is, it's checkout, then svn status.
>>
>> Does automatic eol conversion show up as a local modification in svn status output?
>
> No, it shouldn't.
>
>> I can't reproduce this on either a Windows or a Mac.  It appears to be only on this machine, which, I know, sounds mad.
>
> When a file has svn:eol-style set to native, the Subversion client is supposed to normalize the file to LF line endings before sending it to the repository to be committed. The Subversion server does not verify that this has been done, so it is possible for badly-written Subversion clients to commit files with the wrong line endings. If a third-party svn client (git-svn?) was used to commit this file, that might be a possible cause to investigate. Although it's strange you only see the problem on one system.
>

I'm 99% sure that the above (someone committed a file with
svn:eol-style=native, but it wasn't correctly LF-normalized by the
client) is the root cause of your problem. I've seen this happening
with git-svn (on Windows), but I also suspect some older versions of
SVNKit of a bug in this area (I have not seen it with later version of
SVNKit (>=1.7)). See also issue #4065 [1] about the server not
enforcing this.

A way to verify this is to go looking for the corresponding pristine
file (you can find the checksum of the file by running 'svn info
$thefile', then go looking for the pristine file with the sha1 as
filename in .svn/pristine). That file should normally have LF endings
(for an svn:eol-style=native file). If it has CRLF, then you know its
line-ending normalization was done incorrectly, so you can regard it
as "corrupt". You can fix this by committing the "Modification" that
is now flagged in your working copy (it will then be correctly
normalized by your client).

Now, why does this show up in one working copy, but not in others?
That has to do with the optimization in 'svn status': if filesize and
last-mod-time of the file on disk matches with the svn metadata, then
it will be regarded as not-modified (without looking at the contents).
If the last-mod-time differs, the file contents will be compared with
the pristine, and in that case it shows up as modified. In other
working copies, if you 'touch $thefile', then run 'svn status', it
will probably also show up as modified.

Further, when mixing SVNKit with native svn usage, it's possible that
last-mod-times are already mismatching immediately after checkout.
That's because of an issue in SVNKit [2], where it only notes the
last-mod-time in the svn metadata up to millisecond precision. So if
you 'svnkit co', followed by 'nativesvn status', all files will have
mismatching timestamps, so they'll all be checksummed.


[1] http://subversion.tigris.org/issues/show_bug.cgi?id=4065 - server
should enforce LF normalization for svn:eol-style=native files

[2] http://issues.tmatesoft.com/issue/SVNKIT-315 - SVNKit records
lastModified times truncated to ms precision, even though filesystem
has higher precision, leading to slower working copies because of
mismatching timestamps

-- 
Johan

Re: Local modification on checkout?

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jan 17, 2013, at 15:30, Laird Nelson wrote:

> Hello; we're seeing a local modification being reported on a particular file on a clean checkout.  We're using svn 1.7.7.
> 
> The file in question has the svn:eol-style property set to native.
> 
> What I mean by this:
> 
> A fresh checkout happens (svn co ...).
> 
> Down comes the whole project.  So far so good.
> 
> Then another part of our build infrastructure does an svn status.
> 
> svn status reports that this file is locally modified (M).  There are no intervening steps.  That is, it's checkout, then svn status.
> 
> Does automatic eol conversion show up as a local modification in svn status output?

No, it shouldn't.

> I can't reproduce this on either a Windows or a Mac.  It appears to be only on this machine, which, I know, sounds mad.

When a file has svn:eol-style set to native, the Subversion client is supposed to normalize the file to LF line endings before sending it to the repository to be committed. The Subversion server does not verify that this has been done, so it is possible for badly-written Subversion clients to commit files with the wrong line endings. If a third-party svn client (git-svn?) was used to commit this file, that might be a possible cause to investigate. Although it's strange you only see the problem on one system.

When checking out (or updating) a working copy, if any files have svn:eol-style set to native, then the Subversion client transforms the line endings of those files from LF to whatever svn:eol-style says to do before placing them in the working copy. This can lead to unexpected situations if you check out a working copy on an OS with one line ending style and then look at it or update it from an OS with a different line ending style. If you think this might have happened, check out a new working copy, and use it only on a single system.

Or perhaps again if you checked out or updated using a third-party svn client that did not transform line endings in response to svn:eol-style native, then you might later have a problem.