You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Robert P. J. Day" <rp...@crashcourse.ca> on 2008/10/29 17:15:34 UTC
doing a massive EOL normalization for a small group of users?
currently, i have a small group all working out of the
same repo that was inherited from elsewhere, which contains
a chaotic mixture of LF and CRLF end-of-line characters on
all of the *.c, *.h and Makefile files. what is the best
(and safest) way to do a massive normalization and change
everything to the UNIX native eof of LF, across the repo
and everyone's working copies?
i don't want to have to force everyone to stop editing,
save and check back in everything they're working on because
some of the work is still in progress, so we can assume that
*some* stuff in working copies will differ from the repo --
hopefully, not a lot, and not in an unmergeable way once the
eol change is taken into account.
first, in order to get everyone's sessions in sync, should
i get them to change their .subversion/config files to enable
auto-props, then uncomment the appropriate eol-style=native
lines? i checked and some of them don't have that set, so
i was assuming that would be step one.
and after that, what? i can have everyone "dos2unix" those
files in their working copies, then i can check in a dos2unix'ed
clean version, and have everyone try to update. is that the
right recipe? thanks.
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 29, 2008, at 14:27, Robert P. J. Day wrote:
> Quoting David Weintraub:
>
>> The hook enforces your policy. The auto-properties just is a
>> convenience for the developers.
>
> right ... i had assumed that, once you set a server-side
> commit hook, the client-side auto-props for eol for any files
> covered by that commit hook is pretty much redundant, right?
Not redundant at all.
The server-side pre-commit hook will prevent the commit of files
whose properties do not match your requirements. The developer will
then have to set the required properties before trying to commit again.
The client-side auto-props will prevent the user from having to set
the required properties on new files manually and will do it for them
automatically.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by "Robert P. J. Day" <rp...@crashcourse.ca>.
Quoting David Weintraub <qa...@gmail.com>:
> The hook enforces your policy. The auto-properties just is a
> convenience for the developers.
right ... i had assumed that, once you set a server-side
commit hook, the client-side auto-props for eol for any files
covered by that commit hook is pretty much redundant, right?
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 29, 2008, at 15:30, Robert P. J. Day wrote:
> first, a question about the per-file eol-style property. let's
> imagine i have a file that is currently checked in with proper
> LF eols, and that there is no svn:eol-style property on that file
> and no commit hooks to handle eol processing.
>
> in that case, i can check out the file (it will end up in my
> working copy with proper LF eols), at which point i can mangle
> the eols with "unix2dos". after i do that, i can use hexdump
> to see the result, "file" will tell me there are CRLF terminators,
> and "svn diff" will show me the massive file-wide difference and,
> since there's no prevention, i can check it in with the CRLF
> changes, correct?
Yes, though hopefully you would never do that.
> on the other hand, let's imagine that, after i originally checked
> it out, i set its eol-style property to LF. after i do that, i
> can still mangle its eols with "unix2dos" but, even after i do
> that, "svn diff" won't now show me the differences, and "svn ci"
> won't see anything to check in, and "svn up" doesn't seem to
> update that file back to its original LF eol format. why is that?
> do all of those client-side svn commands take the eol-style property
> into account and ignore exactly those differences?
Exactly.
> it's not like
> that would surprise me, i just want to confirm that that's what's
> happening. (heck, even "svn revert" won't revert it back since,
> i'm guessing, it also realizes it's redundant. i literally have
> to delete the file and check it out again from the repo to get
> the LF version back.)
>
> so far, so good?
>
> and it occurs to me that there might be an easier way to do
> this. if someone's working copy has various changes sprinkled around
> and has a mixture of LFs and CRLFs, i can have them record their
> diffs with "svn diff" and save that output somewhere, then blow
> away their working copy and check out a cleaned version. at that
> point, they can just apply the saved diff back to their working
> copy, but only after they "dos2unix" that patch file so that it
> doesn't cause all sorts of problems. does that sound reasonable?
That sounds possible.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by "Robert P. J. Day" <rp...@crashcourse.ca>.
a couple more mind-meltingly simple questions on all this EOL
normalization, and then i'll shut up.
first, a question about the per-file eol-style property. let's
imagine i have a file that is currently checked in with proper
LF eols, and that there is no svn:eol-style property on that file
and no commit hooks to handle eol processing.
in that case, i can check out the file (it will end up in my
working copy with proper LF eols), at which point i can mangle
the eols with "unix2dos". after i do that, i can use hexdump
to see the result, "file" will tell me there are CRLF terminators,
and "svn diff" will show me the massive file-wide difference and,
since there's no prevention, i can check it in with the CRLF
changes, correct?
on the other hand, let's imagine that, after i originally checked
it out, i set its eol-style property to LF. after i do that, i
can still mangle its eols with "unix2dos" but, even after i do
that, "svn diff" won't now show me the differences, and "svn ci"
won't see anything to check in, and "svn up" doesn't seem to
update that file back to its original LF eol format. why is that?
do all of those client-side svn commands take the eol-style property
into account and ignore exactly those differences? it's not like
that would surprise me, i just want to confirm that that's what's
happening. (heck, even "svn revert" won't revert it back since,
i'm guessing, it also realizes it's redundant. i literally have
to delete the file and check it out again from the repo to get
the LF version back.)
so far, so good?
and it occurs to me that there might be an easier way to do
this. if someone's working copy has various changes sprinkled around
and has a mixture of LFs and CRLFs, i can have them record their
diffs with "svn diff" and save that output somewhere, then blow
away their working copy and check out a cleaned version. at that
point, they can just apply the saved diff back to their working
copy, but only after they "dos2unix" that patch file so that it
doesn't cause all sorts of problems. does that sound reasonable?
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by "Robert P. J. Day" <rp...@crashcourse.ca>.
Quoting David Weintraub <qa...@gmail.com>:
> I believe the auto-properties only works with new files, ...
yes, i just noticed that in the default config file:
"### Set enable-auto-props to 'yes' to enable automatic properties
### for 'svn add' and 'svn import', it defaults to 'no'."
so it clearly won't mean squat for existing files. my bad.
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by David Weintraub <qa...@gmail.com>.
On Wed, Oct 29, 2008 at 2:17 PM, Robert P. J. Day <rp...@crashcourse.ca> wrote:
> oh, i know how to fix the files *locally*. since all of my stuff
> is checked in, i can trivially check out a fresh working copy, make
> massive normalization changes, and check back in without a problem.
> that part's easy.
>
> what i'm after is what prep work i need to do for the *others*
> who all have working copies in which they're currently doing work,
> so that when *they* try to update, they're not hit with a blizzard
> of merge conflicts based on different EOLs.
>
> i've already told them to edit .subversion/config, and set up
> an eol-style of LF. and they do use a mixture of tortoise, rapidsvn
> and straight command line checkouts. so i'm trying to lock this
> down so that, once we get all this normalized, it can't get out of
> sync again based on people mixing their editor usage.
Then, all you have to do is add in the pre-commit hook to prevent
people from checking in these files unless svn:eol-style is on the
file. I believe the auto-properties only works with new files, so you
have to force users to add the svn:eol-style property to pre-existing
files before they do the commit. Otherwise, the developers will simply
mess up the files again.
The hook enforces your policy. The auto-properties just is a
convenience for the developers.
--
David Weintraub
qazwart@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by "Robert P. J. Day" <rp...@crashcourse.ca>.
Quoting Jerry McAllister <je...@msu.edu>:
> On Wed, Oct 29, 2008 at 01:15:34PM -0400, Robert P. J. Day wrote:
>
>> currently, i have a small group all working out of the
>> same repo that was inherited from elsewhere, which contains
>> a chaotic mixture of LF and CRLF end-of-line characters on
>> all of the *.c, *.h and Makefile files. what is the best
>> (and safest) way to do a massive normalization and change
>> everything to the UNIX native eof of LF, across the repo
>> and everyone's working copies?
>>
>> i don't want to have to force everyone to stop editing,
>> save and check back in everything they're working on because
>> some of the work is still in progress, so we can assume that
>> *some* stuff in working copies will differ from the repo --
>> hopefully, not a lot, and not in an unmergeable way once the
>> eol change is taken into account.
>>
>> first, in order to get everyone's sessions in sync, should
>> i get them to change their .subversion/config files to enable
>> auto-props, then uncomment the appropriate eol-style=native
>> lines? i checked and some of them don't have that set, so
>> i was assuming that would be step one.
>>
>> and after that, what? i can have everyone "dos2unix" those
>> files in their working copies, then i can check in a dos2unix'ed
>> clean version, and have everyone try to update. is that the
>> right recipe? thanks.
>
> I presume that the ones with CR are from MessyDOS/Win environment
> and what they have is CR-LF at the end of their lines and that
> the others all have just LF at the end of lines (UNIX style) and
> that none have only CR as a line ender.
that would be correct.
> In that case, just removing the CR would do the trick.
> You could use the dos2unix utility. It has been a very long
> time since I have used it, but I think it worked fine.
>
> Another one is to use tr(1) on the files. It could be used
> on every file across the board without confusing anything even
> if they didn't have any CR-s in them. Something like:
> tr -d '\r' <infile >outfile
>
> That tells it to read infile, delete all RETURNs and write to outfile.
oh, i know how to fix the files *locally*. since all of my stuff
is checked in, i can trivially check out a fresh working copy, make
massive normalization changes, and check back in without a problem.
that part's easy.
what i'm after is what prep work i need to do for the *others*
who all have working copies in which they're currently doing work,
so that when *they* try to update, they're not hit with a blizzard
of merge conflicts based on different EOLs.
i've already told them to edit .subversion/config, and set up
an eol-style of LF. and they do use a mixture of tortoise, rapidsvn
and straight command line checkouts. so i'm trying to lock this
down so that, once we get all this normalized, it can't get out of
sync again based on people mixing their editor usage.
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by Jerry McAllister <je...@msu.edu>.
On Wed, Oct 29, 2008 at 01:15:34PM -0400, Robert P. J. Day wrote:
> currently, i have a small group all working out of the
> same repo that was inherited from elsewhere, which contains
> a chaotic mixture of LF and CRLF end-of-line characters on
> all of the *.c, *.h and Makefile files. what is the best
> (and safest) way to do a massive normalization and change
> everything to the UNIX native eof of LF, across the repo
> and everyone's working copies?
>
> i don't want to have to force everyone to stop editing,
> save and check back in everything they're working on because
> some of the work is still in progress, so we can assume that
> *some* stuff in working copies will differ from the repo --
> hopefully, not a lot, and not in an unmergeable way once the
> eol change is taken into account.
>
> first, in order to get everyone's sessions in sync, should
> i get them to change their .subversion/config files to enable
> auto-props, then uncomment the appropriate eol-style=native
> lines? i checked and some of them don't have that set, so
> i was assuming that would be step one.
>
> and after that, what? i can have everyone "dos2unix" those
> files in their working copies, then i can check in a dos2unix'ed
> clean version, and have everyone try to update. is that the
> right recipe? thanks.
I presume that the ones with CR are from MessyDOS/Win environment
and what they have is CR-LF at the end of their lines and that
the others all have just LF at the end of lines (UNIX style) and
that none have only CR as a line ender.
In that case, just removing the CR would do the trick.
You could use the dos2unix utility. It has been a very long
time since I have used it, but I think it worked fine.
Another one is to use tr(1) on the files. It could be used
on every file across the board without confusing anything even
if they didn't have any CR-s in them. Something like:
tr -d '\r' <infile >outfile
That tells it to read infile, delete all RETURNs and write to outfile.
////jerry
>
> rday
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by David Weintraub <qa...@gmail.com>.
Quoting David Weintraub <qa...@gmail.com>:
> First off, add a pre-commit hook that will fail a commit if a file
> being edited is of the right type and doesn't have the required
> svn:eol-style property set to the correct value. I don't know if the
> current Python pre-commit hook can do this, but I have a Perl
> pre-commit hook that does this type of checking as well as folder
> permission checking. Let me know if you want a copy of it.
On Wed, Oct 29, 2008 at 2:38 PM, Robert P. J. Day <rp...@crashcourse.ca> wrote:
> sure, send it along, but for the moment, i'm going to hold off
> on that as my first priority is to just get the normalization done
> and get everything sync'ed. once that's done, i can add the properties
> and commit hooks at my leisure to keep everything that way.
Actually, the hook should be your first priority. Otherwise, you're
fighting a losing battle. You'll fix a bunch of scripts, and some
developer is going to check one of them out while you're busy fixing
something else, and mess it all up again.
> again, adding that property is something that can be done afterwards
> to keep things clean once they've been cleaned up. my number one
> priority is to just get there, so i'm about to give it a shot.
The svn:eol-style property will do the cleanup for you. You don't have
to modify the files themselves by fixing the line endings.
If you modify a file by changing the line endings while another
developer is attempting to modify it, they'll get a merge conflict,
and then they have to figure out the diff (which will show every line
as being modified). In the end, they'll simply check the file back in
with the bad line endings. You're back to where you started from.
If instead, you add the svn:eol-style property instead of modifying
the file's contents, developers will get a merge conflict, but the
only thing they'll have to do is add teh svn:eol-style property to
their checked out copy. No real merge is necessary. Of course, putting
the pre-commit hook in will help "remind" the developers to add this
property instead of checking in their revision that doesn't have this
property.
--
David Weintraub
qazwart@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 29, 2008, at 13:38, Robert P. J. Day wrote:
> Quoting David Weintraub:
>
>> As far as I can see, you'll have to do this and this will interrupt
>> everyone who is currently editing.
>>
>> First off, add a pre-commit hook that will fail a commit if a file
>> being edited is of the right type and doesn't have the required
>> svn:eol-style property set to the correct value. I don't know if the
>> current Python pre-commit hook can do this, but I have a Perl
>> pre-commit hook that does this type of checking as well as folder
>> permission checking. Let me know if you want a copy of it.
>
> sure, send it along, but for the moment, i'm going to hold off
> on that as my first priority is to just get the normalization done
> and get everything sync'ed. once that's done, i can add the
> properties
> and commit hooks at my leisure to keep everything that way.
You can do that, but to be sure you got everything right, you'll have
to do the normalization twice then -- a second time after you've set
the pre-commit hook. Otherwise, someone might have committed improper
line endings to a file between the time you did your first
normalization and the time you installed the pre-commit hook.
>> After that, you'll have to do a massive checkout that will checkout
>> the entire repository, and add the svn:eol-style property to the
>> file,
>> then do a checkin. This of course, has to be coordinated with
>> everyone. Some people will be editing these same files, and will
>> get a
>> commit conflict when they attempt to check them in. But, it isn't all
>> that bad. You're not modifying the file, only the property. The
>> merge,
>> although a pain (I hate it when commits fail too) is pretty simple.
>
> again, adding that property is something that can be done afterwards
> to keep things clean once they've been cleaned up. my number one
> priority is to just get there, so i'm about to give it a shot. i'm
> still curious if anyone's ever actually *done* this, and the exact
> order of the steps to make it go smoothly, including what to get
> the users to do in preparation, assuming that they simply can't
> drop everything they're doing and check in all their changes.
If you have files with consistent line endings, just the wrong line
endings, you can just set svn:eol-style to native and Subversion will
convert the line endings to LF and store it in the repository.
If you have files with inconsistent line endings, Subversion will not
convert it for you.You must convert it yourself, e.g. using tr or
dos2unix or whatever. Then you can add the svn:eol-style property.
If you just clean up the files, then at some later point add the
property, it may be too late; someone may already have committed new
incorrect line endings to the file by then.
Your strategy should be to add the svn:eol-style property to files.
If it fails because of inconsistent line endings, then fix the line
endings and immediately set the svn:eol-style property so they can't
get inconsistent again.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by "Robert P. J. Day" <rp...@crashcourse.ca>.
Quoting David Weintraub <qa...@gmail.com>:
> As far as I can see, you'll have to do this and this will interrupt
> everyone who is currently editing.
>
> First off, add a pre-commit hook that will fail a commit if a file
> being edited is of the right type and doesn't have the required
> svn:eol-style property set to the correct value. I don't know if the
> current Python pre-commit hook can do this, but I have a Perl
> pre-commit hook that does this type of checking as well as folder
> permission checking. Let me know if you want a copy of it.
sure, send it along, but for the moment, i'm going to hold off
on that as my first priority is to just get the normalization done
and get everything sync'ed. once that's done, i can add the properties
and commit hooks at my leisure to keep everything that way.
> After that, you'll have to do a massive checkout that will checkout
> the entire repository, and add the svn:eol-style property to the file,
> then do a checkin. This of course, has to be coordinated with
> everyone. Some people will be editing these same files, and will get a
> commit conflict when they attempt to check them in. But, it isn't all
> that bad. You're not modifying the file, only the property. The merge,
> although a pain (I hate it when commits fail too) is pretty simple.
again, adding that property is something that can be done afterwards
to keep things clean once they've been cleaned up. my number one
priority is to just get there, so i'm about to give it a shot. i'm
still curious if anyone's ever actually *done* this, and the exact
order of the steps to make it go smoothly, including what to get
the users to do in preparation, assuming that they simply can't
drop everything they're doing and check in all their changes.
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by "Robert P. J. Day" <rp...@crashcourse.ca>.
Quoting David Weintraub <qa...@gmail.com>:
> First off, add a pre-commit hook that will fail a commit if a file
> being edited is of the right type and doesn't have the required
> svn:eol-style property set to the correct value.
actually, at the moment, i don't have repository admin permission,
so i can't add any hooks. i can, of course, sweet talk the actual
admin into doing it for me.
rday
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: doing a massive EOL normalization for a small group of users?
Posted by David Weintraub <qa...@gmail.com>.
As far as I can see, you'll have to do this and this will interrupt
everyone who is currently editing.
First off, add a pre-commit hook that will fail a commit if a file
being edited is of the right type and doesn't have the required
svn:eol-style property set to the correct value. I don't know if the
current Python pre-commit hook can do this, but I have a Perl
pre-commit hook that does this type of checking as well as folder
permission checking. Let me know if you want a copy of it.
After that, you'll have to do a massive checkout that will checkout
the entire repository, and add the svn:eol-style property to the file,
then do a checkin. This of course, has to be coordinated with
everyone. Some people will be editing these same files, and will get a
commit conflict when they attempt to check them in. But, it isn't all
that bad. You're not modifying the file, only the property. The merge,
although a pain (I hate it when commits fail too) is pretty simple.
Then, it is up to your developers to setup their own auto-properties
when they create a new file. You can help by creating a default config
file for them to use, and give them instructions on how to install it,
but that's really the developers' responsibility.
--
David Weintraub
qazwart@gmail.com
On Wed, Oct 29, 2008 at 1:15 PM, Robert P. J. Day <rp...@crashcourse.ca> wrote:
> currently, i have a small group all working out of the
> same repo that was inherited from elsewhere, which contains
> a chaotic mixture of LF and CRLF end-of-line characters on
> all of the *.c, *.h and Makefile files. what is the best
> (and safest) way to do a massive normalization and change
> everything to the UNIX native eof of LF, across the repo
> and everyone's working copies?
>
> i don't want to have to force everyone to stop editing,
> save and check back in everything they're working on because
> some of the work is still in progress, so we can assume that
> *some* stuff in working copies will differ from the repo --
> hopefully, not a lot, and not in an unmergeable way once the
> eol change is taken into account.
>
> first, in order to get everyone's sessions in sync, should
> i get them to change their .subversion/config files to enable
> auto-props, then uncomment the appropriate eol-style=native
> lines? i checked and some of them don't have that set, so
> i was assuming that would be step one.
>
> and after that, what? i can have everyone "dos2unix" those
> files in their working copies, then i can check in a dos2unix'ed
> clean version, and have everyone try to update. is that the
> right recipe? thanks.
>
> rday
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org