You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Joe <de...@freedomcircle.net> on 2006/06/28 21:00:19 UTC
Migrating repository from Windows to UNIX and eol-style
Hi,
I'm trying to migrate some source code repositories that were created on
Windows XP to FreeBSD. The svnadmin dump and load work well, but in the
dump file, the source files end up with visible CRs at the end of each
file and when I checkout the individual files from BSD, they also
contain CRs. If possible, I'd like to remove the CRs in the migration
and not have to do a massive revision at either end. I've read the
discussion in the Subversion book about svn:eol-style and I was hoping
that by setting svn:eol-style to LF on the Windows side I could do have
the svnadmin dump eliminate the CRs. However, it seems that propset
only works on client files, in other words I can't do
svn propset svn:eol-style LF c:/path/to/repos
Any suggestions? If it makes any difference, I have svn 1.2.3 on
Windows and 1.3.2 on FreeBSD [no, it's not a typo].
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Duncan Murdoch <mu...@stats.uwo.ca>.
On 6/28/2006 6:39 PM, Joe wrote:
> Ryan Schmidt wrote:
>> The svn:eol-style property and svnadmin dump have nothing to do with one
>> another. It sounds like you've just been checking in files with DOS
>> linefeeds, and without any svn:eol-style property. Wouldn't matter what
>> OS the server is running on; it'll just accept and store the bytes you
>> send it. If you send it DOS linefeeds, it'll store those.
>
> The files were typically created on Windows using gvim so each original
> file had CR-LF end-of-lines. I was under the (perhaps mistaken)
> impression that unless you specified something to the contrary,
> Subversion used native eol-style, which would be CRLF on Windows, but
> would "actually store the file in the repository using normalized LF EOL
> markers regardless of the operating system" (Chapter 7).
The part you quote is about svn:eol-style=native. The general statement
comes earlier in that section: "bytes are bytes".
>You seem to
> imply that one *has* to specify the svn:eol-style property initially,
> rather than getting the OS default. Is that correct?
I think that is correct. Subversion stores what you give it, *unless*
you ask it not to (via the svn:eol-style property).
Duncan Murdoch
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jul 3, 2006, at 18:05, Joe wrote:
> Erik Huelsmann wrote:
>> Right. We learned from CVS in this respect: many a time people forgot
>> to use the 'this is a binary file, please don't modify'-option when
>> adding an image to the repository. Most of them were damaged,
>> sometimes not even recoverable (because the original was deleted).
>
> OK, that's the strongest argument for doing it the Subversion way
> (and I won't press the issue further).
>
>> Absolutely. This is one of Subversion's weak spots. You as a sysadmin
>> can prevent the file ending up in the repository by checking for the
>> right properties and rejecting the 'newbie commit' in a post-commit
>> hook.
>> [snip] If you have resources to spare (time to discuss design or
>> implementation), you're most welcome to come to dev@ and stirr up the
>> discussion again. You can read about it all through the mailing list
>> archives at http://svn.haxx.se/dev/
>
> I don't have much time to spare, but I'm thinking that one thing
> that would help sysadmins is some way of storing default properties
> at the server, e.g., allowing a project to specify something like
> "from this tree node down, all .c/.h files are defaulted to text
> and svn:eol-style native, and all .jpg/.png files default to the
> respective image mime-types". Of course, one can implement this by
> storing a config file (or config template) at that point in the SVN
> tree (or some parallel tree) and require developers to fetch it and
> adapt it as their first order of business, but a Subversion-
> provided facility would be preferable. What is the rationale for
> not storing default props at the server?
I'm not sure what the rationale is, other than maybe it hasn't been
high enough on anyone's priority list to work out exactly how such a
feature should be implemented. But it's certainly possible to write a
pre-commit hook that implements that behavior. If I ever get around
to writing my web-based Subversion administration toolkit, that would
be one of the scripts I'd provide.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Joe <de...@freedomcircle.net>.
Erik Huelsmann wrote:
> Right. We learned from CVS in this respect: many a time people forgot
> to use the 'this is a binary file, please don't modify'-option when
> adding an image to the repository. Most of them were damaged,
> sometimes not even recoverable (because the original was deleted).
OK, that's the strongest argument for doing it the Subversion way (and I
won't press the issue further).
> Absolutely. This is one of Subversion's weak spots. You as a sysadmin
> can prevent the file ending up in the repository by checking for the
> right properties and rejecting the 'newbie commit' in a post-commit
> hook.
> [snip]
> If you have resources to spare (time to discuss design or
> implementation), you're most welcome to come to dev@ and stirr up the
> discussion again. You can read about it all through the mailing list
> archives at http://svn.haxx.se/dev/
I don't have much time to spare, but I'm thinking that one thing that
would help sysadmins is some way of storing default properties at the
server, e.g., allowing a project to specify something like "from this
tree node down, all .c/.h files are defaulted to text and svn:eol-style
native, and all .jpg/.png files default to the respective image
mime-types". Of course, one can implement this by storing a config file
(or config template) at that point in the SVN tree (or some parallel
tree) and require developers to fetch it and adapt it as their first
order of business, but a Subversion-provided facility would be
preferable. What is the rationale for not storing default props at the
server?
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Erik Huelsmann <eh...@gmail.com>.
On 7/3/06, Joe <de...@freedomcircle.net> wrote:
> Erik Huelsmann wrote:
> > * Subversion was intentionally designed not to modify files under
> > version control until explicitly told to do so. You may not think your
> > files changed when you want different line-endings, but others may:
> > say you checked in a subversion dump file. If the system started to
> > subsitute line endings for you, the file would have been damaged
> > beyond repair, even though it looked texty. The default is (and
> > *should stay*) to protect your data.
> >
> > Please don't project any problems you may have right this instant on
> > the version control system you're using: your problems are (largely)
> > unrelated to version control, but to inter-OS moving of files.
>
> Thanks Erik. I can see your point. However, it is a design decision of
> Subversion to use a heuristic for determining whether something is text
> or not. Compare that with FTP: if you transfer in "ASCII" mode it will
> modify a UNIX file and create it with CRLF on Windows, and vice versa,
> i.e., it uses an explicit mode rather than a heuristic.
If I understand you correctly, the above isn't true: Subversion - by
default - doesn't assign meaning to the contents of files being
committed: it doesn't distinguish between text or binary and it
doesn't do eol conversion. That way, the files don't get damaged. When
Subversion would do eol conversion by default, files would get
modified at checkin and there wouldn't be any way to reconstruct the
original. That's why we make it an explicit choice to have eol
conversion.
The only time Subversion uses a heuristic to estimate textiness is
with 'diff' related operations (blame, diff, merge), never with other
situations. BTW: there is no other way to determine whether Subversion
is looking at a text file but to use a heuristic: all files consist of
bytes, the meaning we assign to them is different on a case-by-case
basis (you can just name your JPG images photo.txt, they won't become
text files).
> Since most VCSs
> started being used for storing and controlling program source files, I
> was expecting a similar behavior.
Right. We learned from CVS in this respect: many a time people forgot
to use the 'this is a binary file, please don't modify'-option when
adding an image to the repository. Most of them were damaged,
sometimes not even recoverable (because the original was deleted).
> It still seems to me that forcing clients to deal with auto-props is not
> the best decision since it's too easy to miss a file suffix or have some
> newbie developer in a large mixed-platform group forget to set props and
> start submitting changes and screwing things up (yes, the peers will
> hopefully catch the problem before it gets too far, but IMHO the VCS
> should be preventing this).
Absolutely. This is one of Subversion's weak spots. You as a sysadmin
can prevent the file ending up in the repository by checking for the
right properties and rejecting the 'newbie commit' in a post-commit
hook.
> As I hinted before, I'd prefer a server-based capability that says: all
> our files are text files and we want to store them at the clients in
> native format and at the server as "collection of logical lines", except
> for files which we specify as binary at the client. This still would
> allow companies that have mostly binary files to specify otherwise
> (either at the server or the client).
Absolutely. We have had extensive discussion on the development list
about use cases and how the implementation behind this should work.
Especially also how changes in the configuration would propagate. In
the past discussions, we didn't reach a final conclusion of how it
should work though.
If you have resources to spare (time to discuss design or
implementation), you're most welcome to come to dev@ and stirr up the
discussion again. You can read about it all through the mailing list
archives at http://svn.haxx.se/dev/
HTH,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Joe <de...@freedomcircle.net>.
Erik Huelsmann wrote:
> * Subversion was intentionally designed not to modify files under
> version control until explicitly told to do so. You may not think your
> files changed when you want different line-endings, but others may:
> say you checked in a subversion dump file. If the system started to
> subsitute line endings for you, the file would have been damaged
> beyond repair, even though it looked texty. The default is (and
> *should stay*) to protect your data.
>
> Please don't project any problems you may have right this instant on
> the version control system you're using: your problems are (largely)
> unrelated to version control, but to inter-OS moving of files.
Thanks Erik. I can see your point. However, it is a design decision of
Subversion to use a heuristic for determining whether something is text
or not. Compare that with FTP: if you transfer in "ASCII" mode it will
modify a UNIX file and create it with CRLF on Windows, and vice versa,
i.e., it uses an explicit mode rather than a heuristic. Since most VCSs
started being used for storing and controlling program source files, I
was expecting a similar behavior.
It still seems to me that forcing clients to deal with auto-props is not
the best decision since it's too easy to miss a file suffix or have some
newbie developer in a large mixed-platform group forget to set props and
start submitting changes and screwing things up (yes, the peers will
hopefully catch the problem before it gets too far, but IMHO the VCS
should be preventing this).
As I hinted before, I'd prefer a server-based capability that says: all
our files are text files and we want to store them at the clients in
native format and at the server as "collection of logical lines", except
for files which we specify as binary at the client. This still would
allow companies that have mostly binary files to specify otherwise
(either at the server or the client).
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Erik Huelsmann <eh...@gmail.com>.
Joe, Ryan,
Sorry to clip all the content from the message, but I just have to say:
* the problems you indicate with svn:eol-style are certainly not
Subversion problems:
- cygwin considers LF its line ending, that's just its way to make
it look like another POSIX system...
- moving working copies between systems which have different
line-endings is a problem with or without svn:eol-style: it's a
problem with the environment you work in. If you move your work
through a commit/update cycle, everything works fine. Everything else:
you're on your own (just as you were without Subversion).
* Subversion was intentionally designed not to modify files under
version control until explicitly told to do so. You may not think your
files changed when you want different line-endings, but others may:
say you checked in a subversion dump file. If the system started to
subsitute line endings for you, the file would have been damaged
beyond repair, even though it looked texty. The default is (and
*should stay*) to protect your data.
Please don't project any problems you may have right this instant on
the version control system you're using: your problems are (largely)
unrelated to version control, but to inter-OS moving of files.
bye,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Joe <de...@freedomcircle.net>.
Ryan Schmidt wrote:
> Here's again where I'm getting that impression. Because you again
> mention a repository migration from Windows to Unix, and it sounds like
> you're implying that this migration would be the cause of the problems
> you're describing. But that can't be the case. A Subversion repository
> is a Subversion repository is a Subversion repository and behaves
> identically regardless of whether the server runs on Mac OS X, Unix,
> Linux, FreeBSD, Sun OS, Windows, pocket lint or good intentions. In the
> context of the svn:eol-style property, the component that behaves
> differently on different platforms is the client, not the server.
I'm sorry if I sound adversarial (:-) but you sound like you're saying
"Oh, it's not my problem if your Windows developers don't know about
auto-props and the files are stored with CRLFs and then they can't be
migrated easily to UNIX". That's like a database administrator telling
me "Oh, it's not my problem if your users (or your application
developers) don't know about encodings and now data that was entered in
Asia appears to be corrupted when retrieved from Europe".
My simple point is this: conceptually, the source files did not change
due to the repository migration; whether it's a client or server problem
is irrelevant to the users/developers. Subversion is not just a server
or just a client, but both.
If (about a year and a half ago) we had had foresight and/or had
understood correctly how auto-props would affect a possible migration
(or sharing the repository from another platform--neither of which we
envisioned at the time), we would've set up our config files
accordingly. What's done is done and all we can do now is correct the
situation.
However, I still believe that by implementing a different default policy
(whether at the client or the server), Subversion could prevent what
happened to us from happening to others. Granted, perhaps I'm presuming
too much and there are not too many who've tripped up or will encounter
this problem in the future.
BTW, the proposed default policy is that if a file appears to be a text
file, then the auto-prop svn:eol-style native *should* (in essence) be
the default (most likely a determination to be done at the svn client).
The main reason I think it's a better default is that in this way,
most text files will be stored in the repository as "logical"
representations (an alternative would be to store each line with a count
of characters and no indication of line terminators). I'll be glad to
argue this in the hackers list if desired.
> Well, you've got it exactly. In the source code of the script, I might
> be looking for the literal "\r\n" within a file, let's say foo.dat. What
> if foo.dat is also a checked-in file? If Subversion does some automatic
> line ending transformation as you suggest, and I now check out foo.dat
> on Unix, the file does not contain DOS linefeeds anymore, and the
> script, which is looking for DOS linefeeds in it, fails.
I'm sorry again but I think that's an unrealistic scenario.
> So there's no particular disadvantage to Windows users here. If you
> check in a file with one kind of line ending, without setting
> svn:eol-style, and then check it out to an environment where another
> line ending style is needed, then you've got a problem.
All of your examples tend to add to my suggestion that a better default
is to store text files as "logical" collections of lines, it doesn't
matter whether the server uses CRLFs, LFs, NULs or character counts.
For sharing across platforms as well as migrating between platforms, an
EOL-neutral representation is best, IMHO of course :-) With the
proposed default policy, if I check out a file and it doesn't look like
it has the correct EOLs, it's then easy to check auto-props or the file
properties to determine what's up.
> For this reason, we never use svn:eol-style=native. We standardize on
> all text files having Unix linefeeds, even when used it on Windows. Our
> Windows text editor can handle this, so this is not a problem. (Editing
> DOS text files with vi on Unix is a bit of a problem, with ^M appearing
> at the end of each line as you said, which is why we didn't standardize
> on DOS line endings.)
From my perspective, you've devised your own workaround (for what I see
as the deficiencies of svn:eol-style) by forcing a standard on your
developers.
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Ryan Schmidt <su...@ryandesign.com>.
Joe,
Just up front, I want to make sure I'm not coming across as
adversarial in this thread. I certainly don't mean to argue with you
on any points. I just want to try to explain how Subversion is, and
give you recommendations for things I'd do and things I'd avoid. As
long as you're aware of what you're doing, I don't want to prevent
you from following any strategy you'd like. Just want to make sure
you make informed decisions. :-)
Now, on to business:
On Jun 29, 2006, at 16:43, Joe wrote:
> Ryan Schmidt wrote:
>
>> You seem to be under the impression that the presence of CRs in
>> the dumpfile would prevent it from loading on a Subversion server
>> running on BSD. But that should not be the case; there should be
>> no problem loading the dumpfile, on BSD or any other OS.
>
> I'm not sure why you think that because I've mentioned that I *did*
> reload the dump file on BSD and everything was fine, except for the
> EOLs on checkout.
Oh. Ok.
>> I don't think that's a good idea. Your repository is supposed to
>> represent historical data, and you're proposing to rewrite history.
>
> "Rewriting history" is where we disagree. I think of it this way.
> A developer creates a source file on Windows. It's mostly
> transparent to him that each line in the file has a CR-LF. He
> checks the file into a version control system and when he checks it
> out, naturally, he don't see any differences. However, with an
> auto-prop of eol-style native set in his config file, underneath
> the covers, the VCS has mapped each CR-LF to an LF on the check-in
> and vice versa on the check out. Another developer (or the same
> one) creates another source file (perhaps a .h instead of .c and he
> doesn't have auto-prop for .h). Check-in, check-out, everything
> appears to work the same (but the VCS is now storing the .h with CR-
> LFs).
>
> Now the repository is moved to UNIX. The administrator does a dump
> on Windows and a load on UNIX and everything looks good. Then the
> developer does a check out of both files from UNIX. The first file
> is fine, but when he opens the second file in an editor there's all
> these ugly ^M's at the end of each line. To the developer, that
> doesn't look right: he didn't change his source file! Yes, he
> forgot to set auto-prop for the second file (or he didn't know
> about auto-props or whatever), but as an administrator I don't
> think I'd be "rewriting history" if I tried to mask the effects of
> the migration (and the lack of foresight and/or knowledge about
> auto-props) from the developer.
Here's again where I'm getting that impression. Because you again
mention a repository migration from Windows to Unix, and it sounds
like you're implying that this migration would be the cause of the
problems you're describing. But that can't be the case. A Subversion
repository is a Subversion repository is a Subversion repository and
behaves identically regardless of whether the server runs on Mac OS
X, Unix, Linux, FreeBSD, Sun OS, Windows, pocket lint or good
intentions. In the context of the svn:eol-style property, the
component that behaves differently on different platforms is the
client, not the server.
>> What if something breaks because of this change? What if you've
>> inadvertently programmed in a dependency on DOS linefeeds at some
>> point in your application, and your app now breaks due to this
>> conversion? You wouldn't be able to go back in the repository to
>> the last version where you know it worked to compare the changes.
>> All your old versions would be broken too.
>
> I'm sorry but I have trouble imagining how could I program such a
> dependency in the source files. If I do code that, say in C, I'd
> use literal strings like "\r\n" in the statements. Any
> "conversion" that I would be looking for, would leave those
> literals untouched and would only strip the CRs from the end of lines.
Well, you've got it exactly. In the source code of the script, I
might be looking for the literal "\r\n" within a file, let's say
foo.dat. What if foo.dat is also a checked-in file? If Subversion
does some automatic line ending transformation as you suggest, and I
now check out foo.dat on Unix, the file does not contain DOS
linefeeds anymore, and the script, which is looking for DOS linefeeds
in it, fails.
>> I don't see why this problem is any more acute for Windows users
>> vs. users of any other OS.
>
> If you're on UNIX, a text file is stored with LFs on your local
> copy *and* in the repository. If you're on Windows it's stored
> with CR-LF's on your local copy and depending on how you set your
> auto-props it may end up as is or with LFs in the repository.
I'll try to construct a scenario that doesn't involve a repository
migration (since it should not be relevant).
You create a text file on Windows with DOS linefeeds and check it in
without setting svn:eol-style to anything. The repository stores the
file as received, with DOS linefeeds. You check it out on Windows and
it looks right, because it has DOS linefeeds. If you check it out on
Unix or Mac OS X, it looks wrong, because it has DOS linefeeds.
Other way around: you create a text file on Unix with Unix linefeeds
and check it in without setting svn:eol-style to anything. The
repository stores the file as received, with Unix linefeeds. You
check it out on Unix or Mac OS X and it looks right, because it has
Unix linefeeds. If you check it out on Windows, it looks wrong,
because it has Unix linefeeds.
So there's no particular disadvantage to Windows users here. If you
check in a file with one kind of line ending, without setting svn:eol-
style, and then check it out to an environment where another line
ending style is needed, then you've got a problem.
This assumes you do not have an editor that can deal with any kind of
line ending. I would expect anyone working in a cross-platform
environment to have a text editor that can edit files of any line
ending. I use TextEdit on the Mac; my colleagues on Windows use
UltraEdit. There are many alternatives, free and otherwise.
There are two problems that can come up with svn:eol-style=native.
Problem 1. In our setup, we check out working copies from Mac and
Windows workstations to a Samba share served from a Linux server.
Suppose we have a file in the repository with svn:eol-style=native.
We check it out using TortoiseSVN on Windows and write it to the
Samba share. When viewed on Windows, the file has DOS linefeeds,
which are correct. When viewed through the shell on the Linux server,
the same file looks incorrect, because it still has DOS linefeeds.
Other way around: say you check the same directory out while logged
into the Linux server, and you put it in the directory that's shared
via Samba. On the Linux server, the file looks correct with its Unix
linefeeds. If you now look at the file from the Windows machine
through the Samba share, it still has Unix linefeeds, only now that's
incorrect.
Point is: "native" is only native according to the Subversion client
that checked it out. It says nothing about the editor later wanting
to use that checked-out file. What if your working copy is on a Zip
disk which might be inserted into either a Mac or a Windows PC?
"native" is quite confusing in this scenario.
For this reason, we never use svn:eol-style=native. We standardize on
all text files having Unix linefeeds, even when used it on Windows.
Our Windows text editor can handle this, so this is not a problem.
(Editing DOS text files with vi on Unix is a bit of a problem, with
^M appearing at the end of each line as you said, which is why we
didn't standardize on DOS line endings.)
Problem 2. I believe the cygwin environment under Windows considers
svn:eol-style=native to mean Unix linefeeds. So you run into the same
problem if you have a working copy checked out under cygwin and want
to use it within the normal Windows environment, or vice-versa. So in
this scenario, I grant that Windows users are at a disadvantage --
but only if svn:eol-style=native is used.
>> As you know, auto-props are described in the book. Was there some
>> part of that section that was unclear? Or do you feel it should
>> have been mentioned earlier in the book? Version control is a
>> complicated topic, which is why there's a big book describing it,
>> and not everything can be taught at once. You really have to start
>> with the basics and then introduce additional concepts as you go
>> along. Auto-props is not central to the function of what
>> Subversion does, which is probably why it's not mentioned in
>> chapter 1. But if you have suggestions for how the book can be
>> improved, I believe they have their own mailing list.
>
> No, it's not unclear and I agree that it's not something that needs
> to be mentioned up front. However, just like there's an appendix
> for CVS users, perhaps there needs to be a section of gotchas for
> users that share repositories across platforms or that have the
> potential of migrating from one platform to another.
And again it doesn't matter if the repository moves across platforms,
it matters if the working copy moves across platfroms.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Joe <de...@freedomcircle.net>.
Hi Ryan,
Ryan Schmidt wrote:
> You seem to be under the impression that the presence of CRs in the
> dumpfile would prevent it from loading on a Subversion server running on
> BSD. But that should not be the case; there should be no problem loading
> the dumpfile, on BSD or any other OS.
I'm not sure why you think that because I've mentioned that I *did*
reload the dump file on BSD and everything was fine, except for the EOLs
on checkout.
> I don't think that's a good idea. Your repository is supposed to
> represent historical data, and you're proposing to rewrite history.
"Rewriting history" is where we disagree. I think of it this way. A
developer creates a source file on Windows. It's mostly transparent to
him that each line in the file has a CR-LF. He checks the file into a
version control system and when he checks it out, naturally, he don't
see any differences. However, with an auto-prop of eol-style native set
in his config file, underneath the covers, the VCS has mapped each CR-LF
to an LF on the check-in and vice versa on the check out. Another
developer (or the same one) creates another source file (perhaps a .h
instead of .c and he doesn't have auto-prop for .h). Check-in,
check-out, everything appears to work the same (but the VCS is now
storing the .h with CR-LFs).
Now the repository is moved to UNIX. The administrator does a dump on
Windows and a load on UNIX and everything looks good. Then the
developer does a check out of both files from UNIX. The first file is
fine, but when he opens the second file in an editor there's all these
ugly ^M's at the end of each line. To the developer, that doesn't look
right: he didn't change his source file! Yes, he forgot to set
auto-prop for the second file (or he didn't know about auto-props or
whatever), but as an administrator I don't think I'd be "rewriting
history" if I tried to mask the effects of the migration (and the lack
of foresight and/or knowledge about auto-props) from the developer.
> What
> if something breaks because of this change? What if you've inadvertently
> programmed in a dependency on DOS linefeeds at some point in your
> application, and your app now breaks due to this conversion? You
> wouldn't be able to go back in the repository to the last version where
> you know it worked to compare the changes. All your old versions would
> be broken too.
I'm sorry but I have trouble imagining how could I program such a
dependency in the source files. If I do code that, say in C, I'd use
literal strings like "\r\n" in the statements. Any "conversion" that I
would be looking for, would leave those literals untouched and would
only strip the CRs from the end of lines.
> I don't see why this problem is any more acute for Windows users vs.
> users of any other OS.
If you're on UNIX, a text file is stored with LFs on your local copy
*and* in the repository. If you're on Windows it's stored with CR-LF's
on your local copy and depending on how you set your auto-props it may
end up as is or with LFs in the repository.
> As you know, auto-props are described in the book. Was there some part
> of that section that was unclear? Or do you feel it should have been
> mentioned earlier in the book? Version control is a complicated topic,
> which is why there's a big book describing it, and not everything can be
> taught at once. You really have to start with the basics and then
> introduce additional concepts as you go along. Auto-props is not central
> to the function of what Subversion does, which is probably why it's not
> mentioned in chapter 1. But if you have suggestions for how the book can
> be improved, I believe they have their own mailing list.
No, it's not unclear and I agree that it's not something that needs to
be mentioned up front. However, just like there's an appendix for CVS
users, perhaps there needs to be a section of gotchas for users that
share repositories across platforms or that have the potential of
migrating from one platform to another.
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jun 29, 2006, at 03:03, Joe wrote:
> Andy Levy wrote:
>
>> If you want to do a mass conversion, you can, and it'll be only a
>> property change, there won't be any delta on the file contents
>> themselves.
>
> OK, I ran a small test on a subtree of one of the repositories. I
> checked out the files, did a propset svn:eol-style native on all of
> them and then checked them in. Now the svnadmin dump output (for
> the files in last revision) is clear of CRs, so it should load
> nicely into BSD.
Again, there should be no connection between the svn:eol-style
property of your files and the fact that you want to dump and load
the repository. In particular, I'm having trouble with your last
sentence. You seem to be under the impression that the presence of
CRs in the dumpfile would prevent it from loading on a Subversion
server running on BSD. But that should not be the case; there should
be no problem loading the dumpfile, on BSD or any other OS.
> However, after the change is committed, any diff between files
> before and after the propset shows as the entire old file followed
> by the new file, and svn blame reflects every line as the latest,
> post-propset revision. That would also affect any attempt to merge
> changes from any branches before and after the propsets.
>
> Pragmatically, I guess there's not much that I can do except bite
> the bullet and apply the propsets
Yes, those full-file diffs are inconvenient, and complicate merging,
so it would be best to inform all developers that they should commit
all changes before you do the mass propset, and encourage everyone to
do an update immediately after before starting on new work.
> (although I must admit I'm tempted to fiddle with the svnadmin dump
> output--like what happens in the load if Text-content-md5: is
> missing or incorrect?).
I don't think that's a good idea. Your repository is supposed to
represent historical data, and you're proposing to rewrite history.
What if something breaks because of this change? What if you've
inadvertently programmed in a dependency on DOS linefeeds at some
point in your application, and your app now breaks due to this
conversion? You wouldn't be able to go back in the repository to the
last version where you know it worked to compare the changes. All
your old versions would be broken too.
> Philosophically, I think the enable-auto-props parameter, at a
> minimum, deserves highlighting in the docs, particularly for
> Windows administrators.
I don't see why this problem is any more acute for Windows users vs.
users of any other OS.
As you know, auto-props are described in the book. Was there some
part of that section that was unclear? Or do you feel it should have
been mentioned earlier in the book? Version control is a complicated
topic, which is why there's a big book describing it, and not
everything can be taught at once. You really have to start with the
basics and then introduce additional concepts as you go along. Auto-
props is not central to the function of what Subversion does, which
is probably why it's not mentioned in chapter 1. But if you have
suggestions for how the book can be improved, I believe they have
their own mailing list.
> Ideally, I think the eol-style property should be reconsidered as a
> server-side parameter. Just as svn uses a "very basic heuristic to
> determine if [a] file consists of human-readable or non-human-
> readable content", it makes sense that if it's human-readable, eol-
> style should be assumed to be native (unless proven otherwise).
That quote is from the auto-props page [2] in the book. It would be
interesting to know where this decision is made -- in the client
before sending the data to the server, or on the server once the data
arrives. I suspect it's in the client, since the behavior would also
have to occur for file:///-based repositories where no server process
exist. Thing is, I don't believe Subversion has any kind of server-
side parameters such as you desire.
Furthermore, I disagree that svn:eol-style should be automatically
assumed on anything. I think it's better for Subversion to assume
that it should do nothing unless you tell it to, as is the case now.
What if, again as above, I have a data file that has DOS linefeeds,
and I program a script to read this file and my programming assumes
DOS linefeeds? (I'm sure there are several scripts like this in our
company's repository right now.) I would test it locally on my Mac,
it works great, and I commit it and go home for the day. Tomorrow
someone updates their working copy on their Linux machine, only that
DOS data file I committed has been converted to Unix linefeeds and my
script doesn't work anymore. I'd get flak for committing something
that didn't work, only it did work before I committed it. That's no
good.
[1] http://svn.collab.net/repos/svn/trunk/notes/skip-deltas
[2] http://svnbook.red-bean.com/en/1.2/
svn.advanced.props.html#svn.advanced.props.auto
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Joe <de...@freedomcircle.net>.
Andy Levy wrote:
> If you want to do a mass conversion, you can, and it'll be only a
> property change, there won't be any delta on the file contents
> themselves.
OK, I ran a small test on a subtree of one of the repositories. I
checked out the files, did a propset svn:eol-style native on all of them
and then checked them in. Now the svnadmin dump output (for the files
in last revision) is clear of CRs, so it should load nicely into BSD.
However, after the change is committed, any diff between files before
and after the propset shows as the entire old file followed by the new
file, and svn blame reflects every line as the latest, post-propset
revision. That would also affect any attempt to merge changes from any
branches before and after the propsets.
Pragmatically, I guess there's not much that I can do except bite the
bullet and apply the propsets (although I must admit I'm tempted to
fiddle with the svnadmin dump output--like what happens in the load if
Text-content-md5: is missing or incorrect?).
Philosophically, I think the enable-auto-props parameter, at a minimum,
deserves highlighting in the docs, particularly for Windows
administrators. Ideally, I think the eol-style property should be
reconsidered as a server-side parameter. Just as svn uses a "very basic
heuristic to determine if [a] file consists of human-readable or
non-human-readable content", it makes sense that if it's human-readable,
eol-style should be assumed to be native (unless proven otherwise).
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jun 29, 2006, at 00:50, Andy Levy wrote:
> svn:eol-style only affects the eol-style that appears when you do the
> checkout. Internally, SVN always uses LF. eol-style is more for
> user/developer convenience and ease of merging than anything else.
Internally, SVN always uses LF -- IF you set the svn:eol-style to one
of the valid values. If you do not set the svn:eol-style property,
then the files' bytes are stored exactly as received.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Andy Levy <an...@gmail.com>.
On 6/28/06, Joe <de...@freedomcircle.net> wrote:
> Ryan Schmidt wrote:
> > The svn:eol-style property and svnadmin dump have nothing to do with one
> > another. It sounds like you've just been checking in files with DOS
> > linefeeds, and without any svn:eol-style property. Wouldn't matter what
> > OS the server is running on; it'll just accept and store the bytes you
> > send it. If you send it DOS linefeeds, it'll store those.
>
> The files were typically created on Windows using gvim so each original
> file had CR-LF end-of-lines. I was under the (perhaps mistaken)
> impression that unless you specified something to the contrary,
> Subversion used native eol-style, which would be CRLF on Windows, but
> would "actually store the file in the repository using normalized LF EOL
> markers regardless of the operating system" (Chapter 7). You seem to
> imply that one *has* to specify the svn:eol-style property initially,
> rather than getting the OS default. Is that correct?
svn:eol-style only affects the eol-style that appears when you do the
checkout. Internally, SVN always uses LF. eol-style is more for
user/developer convenience and ease of merging than anything else.
IOW, if you're moving the *repository* from Windows to UNIX, this
property doesn't mean a thing. If you've been doing all your
checkouts on Windows and are moving your clients to BSD or will have a
mixed environment, then it's a really good idea to apply this
property.
If you want to do a mass conversion, you can, and it'll be only a
property change, there won't be any delta on the file contents
themselves. You can take this opportunity to set up a robust
auto-props section in your Subversion client configuration, and apply
all those properties across your whole WC using
svn_apply_autoprops.pl. To do all your code, however, you'll have to
check out the whole repository. Maybe better just to do trunk and
selected, active branches.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Joe <de...@freedomcircle.net>.
Ryan Schmidt wrote:
> The svn:eol-style property and svnadmin dump have nothing to do with one
> another. It sounds like you've just been checking in files with DOS
> linefeeds, and without any svn:eol-style property. Wouldn't matter what
> OS the server is running on; it'll just accept and store the bytes you
> send it. If you send it DOS linefeeds, it'll store those.
The files were typically created on Windows using gvim so each original
file had CR-LF end-of-lines. I was under the (perhaps mistaken)
impression that unless you specified something to the contrary,
Subversion used native eol-style, which would be CRLF on Windows, but
would "actually store the file in the repository using normalized LF EOL
markers regardless of the operating system" (Chapter 7). You seem to
imply that one *has* to specify the svn:eol-style property initially,
rather than getting the OS default. Is that correct?
Joe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Migrating repository from Windows to UNIX and eol-style
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jun 28, 2006, at 23:00, Joe wrote:
> I'm trying to migrate some source code repositories that were
> created on Windows XP to FreeBSD. The svnadmin dump and load work
> well, but in the dump file, the source files end up with visible
> CRs at the end of each file and when I checkout the individual
> files from BSD, they also contain CRs. If possible, I'd like to
> remove the CRs in the migration and not have to do a massive
> revision at either end. I've read the discussion in the Subversion
> book about svn:eol-style and I was hoping that by setting svn:eol-
> style to LF on the Windows side I could do have the svnadmin dump
> eliminate the CRs. However, it seems that propset only works on
> client files, in other words I can't do
>
> svn propset svn:eol-style LF c:/path/to/repos
>
> Any suggestions? If it makes any difference, I have svn 1.2.3 on
> Windows and 1.3.2 on FreeBSD [no, it's not a typo].
The svn:eol-style property and svnadmin dump have nothing to do with
one another. It sounds like you've just been checking in files with
DOS linefeeds, and without any svn:eol-style property. Wouldn't
matter what OS the server is running on; it'll just accept and store
the bytes you send it. If you send it DOS linefeeds, it'll store those.
If you're now wanting the files you check out to have Unix linefeeds,
then you should set the svn:eol-style property on each file for which
you want this to happen. Doesn't matter if you do this before or
after the dump/load cycle, but you'll have to do it at some point if
you want that conversion to take place. There are scripts to help
with this. See below.
Note that svn propset svn:eol-style LF c:/path/to/repos doesn't work,
because the property is relevant for individual files, not for
directories.
You must decide what types of files you want to have this linefeed
conversion performed on. You're unlikely to want it on images or
other binary files, for example, because such conversion would ruin
such files. So only set svn:eol-style on text files where you
actually want this to happen. Perhaps you want all *.html files this
way, or all *.php or *.c files or whatever. You can set up automatic
properties (auto-props) so that all new files matching this glob
expression automatically get the indicated property(ies).
Although that would take care of future files, you'll need to
manually take care of all the files you currently have in the repo.
There are a few scripts you can use to apply auto-props definitions
after the fact to files already in the repo.
Auto-props are a client-side affair; while you may have your client
configured correctly, Joe Coworker down the hall may forget. To
prevent him from mucking up your newly polished and auto-propped
repo, you can install a pre-commit hook, preventing the commit of any
file which is missing any required property, as defined by you.
For more on all of this with URLs to relevant pages, see this mail I
wrote before:
http://svn.haxx.se/users/archive-2006-01/0485.shtml
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org