You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Michael Haggerty <mh...@alum.mit.edu> on 2007/08/20 16:02:35 UTC
Faulty line-end handling in "svnadmin load"? [was: Problem with eol-style=CRLF.]
On the cvs2svn mailing list, somebody wrote the message quoted below.
He claims, basically, that if a file that has svn:eol-style=CRLF is
written to a dump file with different line-end convention, then "svn
diff" gets confused.
I would appreciate guidance about where this should be fixed:
1. Is this a bug in "svnadmin load" (i.e., should it do the line-end
conversion itself before storing the file text)?
2. Is this a bug in "svn diff"?
3. Or is it indeed an intended requirement of "svnadmin load" that files
be committed using the same line-end convention as specified in their
svn:eol-style property (i.e., a cvs2svn bug)?
Thanks,
Michael
-------- Original Message --------
Subject: Problem with eol-style=CRLF.
Date: Fri, 25 May 2007 23:32:29 -0400 (EDT)
From: Bart Oldeman <ba...@users.sourceforge.net>
To: dev@cvs2svn.tigris.org
Hi,
I encountered the following issue converting a CVS repository where all
text files have to be CRLF (FreeDOS):
Used cvs2svn (1.5.0) on Linux, with DefaultEOLStyleSetter('CRLF') in the
options file, conversion happened without problems.
When checking out from the new SVN repository I noticed that all checked
out files had CRLF line endings (correct), but as soon as I changed a file
and 'svn diff'ed it reported that the whole file changed its line endings
from LF to CRLF (not correct).
The cause is that cvs2svn converts the text files to have LF line endings
whereas subversion expects them to have CRLF line endings in the
repository (only when eol-style is set to CRLF obviously).
I worked around it by changing dumpfile_delegate.py to do
if s_item.needs_eol_filter:
data_reader = CRLF_EOL_Filter(LF_EOL_Filter(pipe.stdout))
else:
data_reader = pipe.stdout
and
class CRLF_EOL_Filter:
"""Filter a stream and convert all LF end-of-line markers
into CRLFs."""
def __init__(self, stream):
self.stream = stream
self.eof = False
def read(self, size):
while True:
buf = self.stream.read(size)
self.eof = len(buf) == 0
buf = buf.replace('\n', '\r\n')
if buf or self.eof:
return buf
this is just a quick workaround I could use for FreeDOS; a proper solution
would also care about CR line endings and do the right thing.
Bart
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Faulty line-end handling in "svnadmin load"?
Posted by Karl Fogel <kf...@red-bean.com>.
"David Glasser" <gl...@davidglasser.net> writes:
> On 8/20/07, Michael Haggerty <mh...@alum.mit.edu> wrote:
>> But if it causes Subversion problems to have a file with
>> svn:eol-style=CRLF but contents using RNF, then I claim that "svnadmin
>> load" should reject such data with an error message rather than silently
>> accepting inconsistent data. If you agree then I will check in a
>> Subversion bug to that effect.
>
> I would support a warning in svnadmin load (but probably not an error)
> for any invalid svn:* property; see svn_wc_canonicalize_svn_prop.
I think either error out or warning is okay. Maybe error out, and
user can reinvoke with -f to tell 'svnadmin load' to do its best to
load the data, converting eols if necessary?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Faulty line-end handling in "svnadmin load"?
Posted by David Glasser <gl...@davidglasser.net>.
On 8/20/07, Michael Haggerty <mh...@alum.mit.edu> wrote:
> But if it causes Subversion problems to have a file with
> svn:eol-style=CRLF but contents using RNF, then I claim that "svnadmin
> load" should reject such data with an error message rather than silently
> accepting inconsistent data. If you agree then I will check in a
> Subversion bug to that effect.
I would support a warning in svnadmin load (but probably not an error)
for any invalid svn:* property; see svn_wc_canonicalize_svn_prop.
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Faulty line-end handling in "svnadmin load"?
Posted by Michael Haggerty <mh...@alum.mit.edu>.
C. Michael Pilato wrote:
> Michael Haggerty wrote:
>> 3. Or is it indeed an intended requirement of "svnadmin load" that
>> files be committed using the same line-end convention as specified in
>> their svn:eol-style property (i.e., a cvs2svn bug)?
>
> Yep.
Karl Fogel wrote:
> So I don't know how this person's CRLF file got written to the
> dumpfile with a different line-end convention, but that's where the
> badness stared. If Subversion did that, then Subversion has a bug.
The dumpfile was written by cvs2svn, which currently normalizes all
non-binary files to RNF (i.e., always LF). I will obviously have to
change this behavior.
But if it causes Subversion problems to have a file with
svn:eol-style=CRLF but contents using RNF, then I claim that "svnadmin
load" should reject such data with an error message rather than silently
accepting inconsistent data. If you agree then I will check in a
Subversion bug to that effect.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Faulty line-end handling in "svnadmin load"?
Posted by Karl Fogel <kf...@red-bean.com>.
Michael Haggerty <mh...@alum.mit.edu> writes:
> On the cvs2svn mailing list, somebody wrote the message quoted below.
> He claims, basically, that if a file that has svn:eol-style=CRLF is
> written to a dump file with different line-end convention, then "svn
> diff" gets confused.
What's scary about this question is that it (maybe?) implies that an
SVN dumpfile can have different line-end conventions :-).
http://svn.collab.net/repos/svn/trunk/notes/dump-load-format.txt
strongly implies (without quite stating explicitly) that LF is the
line-ending format for a dumpfile, in the sense that the metadata
sections should have lines ending with LF. Of course, the format is
really binary, since a given file's contents within a dumpfile might
contain anything at all.
Regarding file contents:
As Mike Pilato pointed out, text files with svn:eol-style set to
'native' should be repository normal form (LF); files with
svn:eol-style set to a specific line-ending format should be stored in
that format; and files without the property are just binary blobs, for
the purposes of line-endings.
So I don't know how this person's CRLF file got written to the
dumpfile with a different line-end convention, but that's where the
badness stared. If Subversion did that, then Subversion has a bug.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Faulty line-end handling in "svnadmin load"? [was: Problem with
eol-style=CRLF.]
Posted by "C. Michael Pilato" <cm...@collab.net>.
Michael Haggerty wrote:
> On the cvs2svn mailing list, somebody wrote the message quoted below.
> He claims, basically, that if a file that has svn:eol-style=CRLF is
> written to a dump file with different line-end convention, then "svn
> diff" gets confused.
>
> I would appreciate guidance about where this should be fixed:
>
> 1. Is this a bug in "svnadmin load" (i.e., should it do the line-end
> conversion itself before storing the file text)?
No. 'svnadmin load' should never change anything, period. That would
merely mask bugs in the dumpfile generator.
> 2. Is this a bug in "svn diff"?
I don't think so.
> 3. Or is it indeed an intended requirement of "svnadmin load" that files
> be committed using the same line-end convention as specified in their
> svn:eol-style property (i.e., a cvs2svn bug)?
Yep.
Subversion has a concept of "repository normal format", which is defined to
be Unix-style line endings and contracted keywords.
Unlike CVS, where files are text unless explicitly declared binary (-kb),
Subversion does not assume that everyone wants "native" EOL style on all
files by default. So, IIRC:
* files that don't have svn:eol-style set are stored as-is in the
repository
* files that have svn:eol-style set to a particular fixed format ("CRLF",
"LF", etc) are stored in the repository in that format
* files that have svn:eol-style set to "native" get stored in this
repository normal format, and the client takes care of translating
between RNF and the correct line ending style.
> -------- Original Message --------
> Subject: Problem with eol-style=CRLF.
> Date: Fri, 25 May 2007 23:32:29 -0400 (EDT)
> From: Bart Oldeman <ba...@users.sourceforge.net>
> To: dev@cvs2svn.tigris.org
>
> Hi,
>
> I encountered the following issue converting a CVS repository where all
> text files have to be CRLF (FreeDOS):
>
> Used cvs2svn (1.5.0) on Linux, with DefaultEOLStyleSetter('CRLF') in the
> options file, conversion happened without problems.
>
> When checking out from the new SVN repository I noticed that all checked
> out files had CRLF line endings (correct), but as soon as I changed a file
> and 'svn diff'ed it reported that the whole file changed its line endings
> from LF to CRLF (not correct).
>
> The cause is that cvs2svn converts the text files to have LF line endings
> whereas subversion expects them to have CRLF line endings in the
> repository (only when eol-style is set to CRLF obviously).
>
> I worked around it by changing dumpfile_delegate.py to do
>
> if s_item.needs_eol_filter:
> data_reader = CRLF_EOL_Filter(LF_EOL_Filter(pipe.stdout))
> else:
> data_reader = pipe.stdout
>
> and
>
> class CRLF_EOL_Filter:
> """Filter a stream and convert all LF end-of-line markers
> into CRLFs."""
>
> def __init__(self, stream):
> self.stream = stream
> self.eof = False
>
> def read(self, size):
> while True:
> buf = self.stream.read(size)
> self.eof = len(buf) == 0
> buf = buf.replace('\n', '\r\n')
> if buf or self.eof:
> return buf
>
> this is just a quick workaround I could use for FreeDOS; a proper solution
> would also care about CR line endings and do the right thing.
>
> Bart
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand