You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Toby Johnson <to...@etjohnson.us> on 2004/05/18 21:29:44 UTC
VSS to SVN migration script
I have finished (sort of) cleaning up my rewritten Visual SourceSafe to
Subversion migration script. Go 'n' get it at
http://www.tobiasly.com/svn/ . Let me know if it goes horribly wrong
for anyone.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [OT] Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Jens B. Jorgensen wrote:
> Hey guys! I'm actually the original author (plagiarist? I borrowed
> heavily from cvs2svn.py) of vss2svn.py. We took the directory approach
> because sadly (at through the automation interface) history on actual
> files is incomplete. Believe me, parsing the text of ss.exe was not
> the route of choice! I was originally trying to do it through the
> automation interface as this is much easier but alas it cannot be done.
Yes, I realized early on that VSS's "API" is very incomplete and
inadequate. It seemed everything was geared around setting up hooks for
certain events, but since VSS has no server component, you must deploy
your hook scripts to all clients who use it!
By file vs. directory approach, I meant using ss.exe to get file history
vs. directory history. I'm still using ss.exe (and trying to account for
all possible outputs it gives). I actually borrowed the methodology from
the "migrate.pl" script that's floating around, but it never worked
correctly and it was easier for me to start from scratch than try to fix it.
And yes, VSS's notion of labels seems to be rather weird; it's like
labeling a file is a separate item in its history, kinda like a commit
without actually changing anything. Plus you can have labels on
projects, and sometimes those get inherited by files in that project and
sometimes they don't. In the end, it's just not worth it to try to
migrate them.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [OT] Re: VSS to SVN migration script
Posted by "Jens B. Jorgensen" <je...@tallan.com>.
Toby Johnson wrote:
>Nathan Kidd wrote:
>
>
>
>>All that info was from a directory's perspective. When doing history
>>on a directory you still get info about the files in it. When doing a
>>
>>
>
>
>
>>single file's history I get the same results as you.
>>
>>BTW, I believe your file-based approach is better. Getting the
>>current directory layout from an SS dir, and then populating with the
>>history of current files is much easier than trying to retrace through
>>
>>
Hey guys! I'm actually the original author (plagiarist? I borrowed
heavily from cvs2svn.py) of vss2svn.py. We took the directory approach
because sadly (at through the automation interface) history on actual
files is incomplete. Believe me, parsing the text of ss.exe was not the
route of choice! I was originally trying to do it through the automation
interface as this is much easier but alas it cannot be done. Here's a
quote from the script:
> # NOTE: originally this script was written to utilize the VSS Automation
> # interface so that all this disgusting parsing of command-line output
> # would not have to be done. Unfortunately, the VSS Automation API
> # is crippled in some dire and more unfortunately, unfixable ways. You
> # can get around some of the issues. Anyhow so the command-line is used
> # to get histories on the files. For a nice in-depth read of what is wrong
> # with VSS (including but not limited to why we must use the command-line
> # tool) read http://www.michaelbolton.net/testing/VSSDefects.html
>>the directory's history.
>>
>>
>>
>OK, I see what you're saying. I believe, though, that the python script
>you refer to tried to build a "dump" file for Subversion to import,
>rather than doing live commits to Subversion in the order they
>originally occured. So that's probably why the other script took the
>directory-based approach.
>
>
yep, exactly. I took that approach basically because that was the
approach cvs2svn.py took and I kinda liked it.
>So, it looks like I still need to account for "branched" and "rolled
>back", no big deal. But Steve had encountered the following output while
>
>running my script, which is completely different than anything I
>encountered:
>
>**********************
>Label: "031013-077p-TY2"
>User: Watsone Date: 13/10/03 Time: 2:03p
>Labeled
>
>***************** Version 1 *****************
>User: Admin Date: 7/10/03 Time: 5:16p
>Created
>
>The separate section about the label is new to me, so I need to see how
>it fits in with everything else.
>
>
It's been a while since I worked on vss2svn.py (and it is by no means
finished in the sense of being fully functional though it does work for
simple cases) I think I remember that VSS doesn't actually associate a
Label with a specific revision, at least not in the sense that it is
attached to it. Cool huh?!
The bottom line with vss2svn.py is that I don't have time to work on it
right now even though I'd love to see it fully working. I would
certainly be willing to spend a little time looking at patches if you
can fix/improve anything in it.
--
Jens B. Jorgensen
jens.jorgensen@tallan.com
"With a focused commitment to our clients and our people, we deliver value through customized technology solutions"
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [OT] Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Nathan Kidd wrote:
> All that info was from a directory's perspective. When doing history
> on a directory you still get info about the files in it. When doing a
> single file's history I get the same results as you.
>
> BTW, I believe your file-based approach is better. Getting the
> current directory layout from an SS dir, and then populating with the
> history of current files is much easier than trying to retrace through
> the directory's history.
>
OK, I see what you're saying. I believe, though, that the python script
you refer to tried to build a "dump" file for Subversion to import,
rather than doing live commits to Subversion in the order they
originally occured. So that's probably why the other script took the
directory-based approach.
So, it looks like I still need to account for "branched" and "rolled
back", no big deal. But Steve had encountered the following output while
running my script, which is completely different than anything I
encountered:
**********************
Label: "031013-077p-TY2"
User: Watsone Date: 13/10/03 Time: 2:03p
Labeled
***************** Version 1 *****************
User: Admin Date: 7/10/03 Time: 5:16p
Created
The separate section about the label is new to me, so I need to see how
it fits in with everything else.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [OT] Re: VSS to SVN migration script
Posted by Nathan Kidd <na...@hummingbird.com>.
Toby Johnson wrote:
>> +# Types of ss history output lines for a file object's data:
>> +#-----------------------------------------------------------
>> +
>> +#***** FileName.ext ***** ; first line
>> +#Version 229 ; revision number
> What I
> don't understand, though, is the lines with the filename embedded within
> the asterisks. Do you get this sort of output when viewing the history
> on only one file at a time (which is what my script does)? I had only
> ever seen the *version number* in the asterisks line.
All that info was from a directory's perspective. When doing history on
a directory you still get info about the files in it. When doing a
single file's history I get the same results as you.
BTW, I believe your file-based approach is better. Getting the current
directory layout from an SS dir, and then populating with the history of
current files is much easier than trying to retrace through the
directory's history.
> It would still probably help if I could see the entire output from a "ss
> HISTORY" command of a single file which has file labels, inherited
> parent directory labels, branches, and whatever else is possible. It
> helps to see not only what can be output, but the order I can expect
> them to appear in.
I think you'll find the order and syntax the same as what I documented,
except for the difference in the first couple lines. But don't just
look at the things I listed for "file object's data". The "output lines
for a directory" don't only talk about directories; they include the
history of *files* in the directory. The division is there only because
of parsing difference in the first few lines, not because one is about
directories and the other files.
-Nathan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [OT] Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Nathan Kidd wrote:
> This can be solved by using ss.exe's -o@ option to output to a file
> and then read the data from the file. No hard wrapping occurs. This
> is particularly important not just for comments: if you have really
> long paths in VSS they also will be hard-wrapped! Attached is my patch
> for vss2svn.py to do this. (vss2svn.py.patch.nohardwrap)
> ...
>
>+# Types of ss history output lines for a file object's data:
>+#-----------------------------------------------------------
>+
>+#***** FileName.ext ***** ; first line
>+#Version 229 ; revision number
>+#User: username Date: 3/12/02 Time: 3:48p ; info line, in every rev.
>+#Checked in $/basedir ;
>+#Branched ; file dup of dir rev message
>+#Rolled back ; ignore, history was purged
>+#Comment: -item 1 fixed ; always last, CR/LF not
>+#-item 2 fixed ; escaped: keep reading lines
>+
>
>
Thanks a lot Nathan, this should be a help for me. I didn't know about
the -o@ VSS option either, but heck, why not slow down the script even
more by writing all history output to a file? :) It'd be worth it to me
though not to introduce all those line breaks.
I'm not too interested in directory history comments since I just import
the entire directory structure to begin with. I can probably just ignore
the branch stuff, but the label stuff might be sorta tricky. What I
don't understand, though, is the lines with the filename embedded within
the asterisks. Do you get this sort of output when viewing the history
on only one file at a time (which is what my script does)? I had only
ever seen the *version number* in the asterisks line.
It would still probably help if I could see the entire output from a "ss
HISTORY" command of a single file which has file labels, inherited
parent directory labels, branches, and whatever else is possible. It
helps to see not only what can be output, but the order I can expect
them to appear in.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
[OT] Re: VSS to SVN migration script
Posted by Nathan Kidd <na...@hummingbird.com>.
Toby Johnson wrote:
> Could you send me the entire ss "HISTORY" command for this file that's
> giving you trouble?
[Somewhat off topic.]
A few months ago tried to convert our VSS databases using
both vss2svn.py (https://chidb01.tallan.com:4430/vss2svn/) and
migrate.pl (http://www.riseup.com/~brettw/dev/), without success. (It
seems our databases were much more complex than what most people were
successfully converting.) I spend time hacking vss2svn.py but eventually
gave up and just did a latest-version import, keeping the old VSS
database for history.
My changes to vss2svn.py weren't entirely useless however, and I notice
a few things I solved there cropping up in
http://www.tobiasly.com/svn/vss2svn.pl.
1. Pertinent to this thread, I documented all the possible vss history
outputs I could find. I'm not aware of any missing 'action types'. See
attached vss2svn.py.patch.historydoc.
2. ss.exe output redirected to STDOUT forces hard return wrapping at 80
cols., as noted in the --flatten argument to vss2svn.pl.
" --flatten : "Flatten" migrated comments so that all newlines become
spaces. If not used, extra "hard" newlines will be inserted
due to word wrapping of "ss" output; unfortunately there is
no way to distinguish between "hard" and "soft" newlines
during migration."
This can be solved by using ss.exe's -o@ option to output to a file and
then read the data from the file. No hard wrapping occurs. This is
particularly important not just for comments: if you have really long
paths in VSS they also will be hard-wrapped! Attached is my patch for
vss2svn.py to do this. (vss2svn.py.patch.nohardwrap)
I made other large changes to vss2svn.py so it understood all the
previously documented history 'action types'. It (the script, not these
particular changes) wasn't in a finished state when I had to move on
("finished" = "converting my database") and the changes are so broad I
don't think they'd be easily accepted as a patch. If someone is
interested in this then drop me a mail.
Regards,
-Nathan
Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Steve Williams wrote:
>The last section of the debug output of the script is
>
>DEBUG:(4)<**********************>
>DEBUG:(4)<Label: "031013-077p-TY2">
>DEBUG:(4)<User: Watsone Date: 13/10/03 Time: 2:03p>
>DEBUG:(4)<Labeled>
>DEBUG:(4)<>
>DEBUG:(4)<***************** Version 1 *****************>
>DEBUG:(1)<User: Admin Date: 7/10/03 Time: 5:16p>
>DEBUG:(2)<Created>
>
>Yes, that's right. VSS did not output a comment line for version 1 of the
>file.
>
>
Could you send me the entire ss "HISTORY" command for this file that's
giving you trouble?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Steve Williams <st...@kromestudios.com>.
It gets a bit further now (two files further :).
It dies at line 1476 (last line of history should laways be part of a
comment).
The last section of the dump of the ss history is
**********************
Label: "031013-077p-TY2"
User: Watsone Date: 13/10/03 Time: 2:03p
Labeled
***************** Version 1 *****************
User: Admin Date: 7/10/03 Time: 5:16p
Created
The last section of the debug output of the script is
DEBUG:(4)<**********************>
DEBUG:(4)<Label: "031013-077p-TY2">
DEBUG:(4)<User: Watsone Date: 13/10/03 Time: 2:03p>
DEBUG:(4)<Labeled>
DEBUG:(4)<>
DEBUG:(4)<***************** Version 1 *****************>
DEBUG:(1)<User: Admin Date: 7/10/03 Time: 5:16p>
DEBUG:(2)<Created>
Yes, that's right. VSS did not output a comment line for version 1 of the
file.
Sly
----- Original Message -----
From: "Toby Johnson" <to...@etjohnson.us>
To: "SVN Users" <us...@subversion.tigris.org>
Sent: Thursday, May 20, 2004 10:21 PM
Subject: Re: VSS to SVN migration script
> Toby Johnson wrote:
>
> > Ah! We never used labels in our VSS setup, so that's why I never
> > expected that "label" line. It should probably just slurp that on in
> > with the rest of the comment and attach it to each appropriate file
> > with "Label comment:" and all. It should be a rather easy fix; if
> > someone gets to it before I do then send me a patch and I'll include
> > it...
> >
> > toby
>
> Whoops, I didn't see the "Label:" line above, that's the one causing the
> problem. The "Label comment" should get included with the rest of the
> comment already. I've added a line to ignore the Label information, give
> it another try.
>
> toby
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Toby Johnson wrote:
> Ah! We never used labels in our VSS setup, so that's why I never
> expected that "label" line. It should probably just slurp that on in
> with the rest of the comment and attach it to each appropriate file
> with "Label comment:" and all. It should be a rather easy fix; if
> someone gets to it before I do then send me a patch and I'll include
> it...
>
> toby
Whoops, I didn't see the "Label:" line above, that's the one causing the
problem. The "Label comment" should get included with the rest of the
comment already. I've added a line to ignore the Label information, give
it another try.
toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Steve Williams wrote:
>We use labels extensively in VSS to simulate the functionality of branches
>in SVN. Very painful, but necessary at the moment.
>
>Note that an inherited label (a label of a parent project) outputs a history
>entry like the following...
>
>**********************
>Label: "MK Release 266"
>User: Dave Date: 2/04/04 Time: 12:02p
>Labeled
>Label comment:
>
>That was incidentally the very next entry after the one I gave earlier.
>Your Perl script seems to handle these just fine by skipping them.
> The --debug output of vss2svn prints them out as DEBUG(4).
>
>
>
We never used labels much because we were stuck at VSS database version
5 (I almost laugh as I type "VSS database"), since the Unix port of VSS
was never upgraded to version 6. So a lot of the label promotion stuff
wasn't even available for us.
And yes, anything between the "Comment:" line and the next row of
asterisks is assumed to be part of the comment, so it will all get
slurped into the checkin comment for that file or group of files.
BTW, I didn't mention this on the website, but in order to simulate
atomic commits in VSS, the script treats multiple consecutive commits
with the same date, user, and comment all as being part of the same
atomic commit when migrating into SVN.
toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Steve Williams <st...@kromestudios.com>.
We use labels extensively in VSS to simulate the functionality of branches
in SVN. Very painful, but necessary at the moment.
Note that an inherited label (a label of a parent project) outputs a history
entry like the following...
**********************
Label: "MK Release 266"
User: Dave Date: 2/04/04 Time: 12:02p
Labeled
Label comment:
That was incidentally the very next entry after the one I gave earlier.
Your Perl script seems to handle these just fine by skipping them.
The --debug output of vss2svn prints them out as DEBUG(4).
Sly
> >Anyone with Perl experience who could help with this? The vss2svn.pl
script
> >that Toby Johnson graciously supplied (thanks Toby) works great until it
> >hits one of these entries in the history.
> >
> >***************** Version 97 *****************
> >Label: "040402-095p-TY2"
> >User: Watsone Date: 2/04/04 Time: 12:59p
> >Checked in $/Ty2/Source
> >Comment: Added a new cheat that sends MissionActivate to every prop
> >Label comment:
> >
> >
> Ah! We never used labels in our VSS setup, so that's why I never
> expected that "label" line. It should probably just slurp that on in
> with the rest of the comment and attach it to each appropriate file with
> "Label comment:" and all. It should be a rather easy fix; if someone
> gets to it before I do then send me a patch and I'll include it...
>
> toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Steve Williams wrote:
>Anyone with Perl experience who could help with this? The vss2svn.pl script
>that Toby Johnson graciously supplied (thanks Toby) works great until it
>hits one of these entries in the history.
>
>***************** Version 97 *****************
>Label: "040402-095p-TY2"
>User: Watsone Date: 2/04/04 Time: 12:59p
>Checked in $/Ty2/Source
>Comment: Added a new cheat that sends MissionActivate to every prop
>Label comment:
>
>
Ah! We never used labels in our VSS setup, so that's why I never
expected that "label" line. It should probably just slurp that on in
with the rest of the comment and attach it to each appropriate file with
"Label comment:" and all. It should be a rather easy fix; if someone
gets to it before I do then send me a patch and I'll include it...
toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Steve Williams <st...@kromestudios.com>.
Anyone with Perl experience who could help with this? The vss2svn.pl script
that Toby Johnson graciously supplied (thanks Toby) works great until it
hits one of these entries in the history.
***************** Version 97 *****************
Label: "040402-095p-TY2"
User: Watsone Date: 2/04/04 Time: 12:59p
Checked in $/Ty2/Source
Comment: Added a new cheat that sends MissionActivate to every prop
Label comment:
This entry occurs when
- a file inherited a label from a parent project
- the file was modified after the label
- the label field in the file details for that version was set to the same
as the inherited label
The script falls over at line 1426. Myself, I know nothing about Perl and
no-one in this company does (not much call for Perl knowledge in game
development). All it needs to do is ignore the "Label:" line, but I have no
idea how to do that.
Sly
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: VSS to SVN migration script
Posted by Toby Johnson <to...@etjohnson.us>.
Steve Williams wrote:
>Thanks for the script. I'm looking at migrating a large repository from VSS
>to SVN in the near future. One item I was a little confused about was the
>requirement for the setdates option.
>
>Using this also requires the "pre-revprop-change" Hook Script to be set; see
>http://svnbook.red-bean.com/svnbook/ch05s02.html#svn-ch-5-sect-2.1
>
>What does it need to be set to? Does the hook have to do anything in
>particular?
>
>Sly
>
>
Hello Steve, please send questions to the mailing list so everyone can
benefit from the discussion.
It doesn't matter what the "pre-revprop-change" script is set to, as
long as it returns a non-error (zero) exit code. I just created an empty
script ("pre-revprop-change.bat" on Windows machines). You should
probably remove the script after you've completed the migration.
The point is that allowing arbitrary changes to un-versioned data (such
as the commit date) could cause loss of important information.
Subversion assumes that if you went to the trouble of creating the Hook
Script, you must understand the implications of this and will therefore
allow you to do it.
hope this helps,
toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org