You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Bicking, David (HHoldings, IT)" <Da...@thehartford.com> on 2007/08/15 15:10:14 UTC
Complex VSS Migration - dealing with moved/branched version history
Hello,
I searched for days for information and tried many options, and now am
standing at a brick wall.
I am evaluating Subversion as a replacement for VSS. This company has a
huge repository (about 10G) in which branching and sharing were rampant.
Worse, there was no plan for how projects were defined. Consequently,
in addition to the typical VSS corruptions in the data, most projects
(folders) have references to other projects somewhere deep in the
history.
I want to migrate pieces of this behemoth into SVN, but cannot. I used
vss2svn to get a dumpfile (12G) and tried to use svndumpfilter to grab
relevant pieces. Per the state of the data described in the prior
paragraph, it fails every time. I found no reference to any way to pull
a specific project "tree" and get all its history, which is quite odd.
I would think this would be crucial. Why not permit migration to pull
the original data into place? A note stating "original location is x"
should be sufficient.
Is there a way to do this? If I were to pull the whole thing into Svn,
then "copy" the project as a Tag or Branch, then dump the repository and
pull the new copy into another repository, would that work (with entire
history), or would it also fail because of its checkered past? There
must be a way to get those revisions!
*************************************************************************
This communication, including attachments, is
for the exclusive use of addressee and may contain proprietary,
confidential and/or privileged information. If you are not the intended
recipient, any use, copying, disclosure, dissemination or distribution is
strictly prohibited. If you are not the intended recipient, please notify
the sender immediately by return e-mail, delete this communication and
destroy all copies.
*************************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Complex VSS Migration - dealing with moved/branched version
history
Posted by Toby Johnson <to...@etjohnson.us>.
Bicking, David (HHoldings, IT) wrote:
> Is it safe to say that nobody has an answer or suggestion concerning my
> questions below? Nobody has any insight into the plans or perceptions of
> the Subersion developers concerning migration of projects with checkered
> histories?
>
David,
The Subversion project itself has no plans to add the sort of
"streamline" functionality you refer to. I am a vss2svn developer, not a
Subversion developer, but their typical stance on such processing of the
dump/load cycle is that it should be done outside of the svnadmin tool
itself. The dumpfile is merely that; a dump of the entire history and it
is up to users who require additional functionality to implement it.
If you have questions specific to vss2svn, again you are welcome to join
that mailing list but I can say quite assuredly that such a feature
would not be added there (unless you or someone else were to write it an
submit a patch) due to the fact that the main developers for that
project no longer have the time to devote to such development.
toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Complex VSS Migration - dealing with moved/branched version
history
Posted by "Bicking, David (HHoldings, IT)" <Da...@thehartford.com>.
> -----Original Message-----
> From: Alan Barrett [mailto:apb@cequrux.com]
> Sent: Thursday, August 23, 2007 4:03 AM
> To: users@subversion.tigris.org
> Subject: Re: Complex VSS Migration - dealing with
> moved/branched version history
>
> On Mon, 20 Aug 2007, Bicking, David (HHoldings, IT) wrote:
> > Is it safe to say that nobody has an answer or suggestion
> concerning
> > my questions below? Nobody has any insight into the plans or
> > perceptions of the Subersion developers concerning migration of
> > projects with checkered histories?
>
> As far as I can tell, the issue is:
>
-- <snip> --
>
> I'd suggest trying the subversion dumpfile utilities from
> <http://www.cs.berkeley.edu/~dsw/>. This will let you
> convert a dumpfile into a directory hierarchy, edit the
> directory hierarchy, and convert it back to a dumpfile. With
-- <snip> --
>
> --apb (Alan Barrett)
I thank you, and Toby, for your replies. I will take a look at this.
While this option is probably too labor intensive at this time, it is
good to know and might come in handy in the near future.
FYI, once I dealt with some typical VSS corruptions, I ended up with a
17G dumpfile. Even then, the load into SVN stopped around revision
28609 (out of about 36000). This is a biiiiig repository, hence, manual
edits of this kind are logistically challenging.
I decided to go with the typical "grab the Tip/Head, archive the VSS for
reference" route. I actually expect that we will have to do the same
for the commercial products we're reviewing.
--
David
*************************************************************************
This communication, including attachments, is
for the exclusive use of addressee and may contain proprietary,
confidential and/or privileged information. If you are not the intended
recipient, any use, copying, disclosure, dissemination or distribution is
strictly prohibited. If you are not the intended recipient, please notify
the sender immediately by return e-mail, delete this communication and
destroy all copies.
*************************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Complex VSS Migration - dealing with moved/branched version
history
Posted by Alan Barrett <ap...@cequrux.com>.
On Thu, 23 Aug 2007, Alan Barrett wrote:
> I'd suggest trying the subversion dumpfile utilities from
> <http://www.cs.berkeley.edu/~dsw/>.
They seem to have moved to <http://www.cubewano.org/blacktackle>.
--apb (Alan Barrett)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Complex VSS Migration - dealing with moved/branched version
history
Posted by Alan Barrett <ap...@cequrux.com>.
On Mon, 20 Aug 2007, Bicking, David (HHoldings, IT) wrote:
> Is it safe to say that nobody has an answer or suggestion concerning
> my questions below? Nobody has any insight into the plans or
> perceptions of the Subersion developers concerning migration of
> projects with checkered histories?
As far as I can tell, the issue is:
* you can use vss2svn to create an svn dumpfile that has the entire
history of your project;
* you could use "svnadmin load" to import the dumpfile, but you don't
want to import the entire project, you want only parts of it;
* you tried using svndumpfilter to extract the parts that you want, but
it fails because some of the files or directories in the part that
you want were originally copied from files in the part that you don't
want, and svndumpfilter can't deal with this;
* you wish for a way of telling svndumpfilter to deal with this, or
some other way of editing the dump file to deal with this.
I'd suggest trying the subversion dumpfile utilities from
<http://www.cs.berkeley.edu/~dsw/>. This will let you convert a
dumpfile into a directory hierarchy, edit the directory hierarchy, and
convert it back to a dumpfile. With some effort, you should be able to
find the revisions that are causing problems for svndumpfilter, manually
change the details of those revisions (say replacing the copied-from
pointer with a full-text copy of the data, and adding a note to the log
message). Then run the modified dumpfile through svndumpfilter again.
--apb (Alan Barrett)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: RE: Complex VSS Migration - dealing with moved/branched version
history
Posted by "Bicking, David (HHoldings, IT)" <Da...@thehartford.com>.
Is it safe to say that nobody has an answer or suggestion concerning my
questions below? Nobody has any insight into the plans or perceptions of
the Subersion developers concerning migration of projects with checkered
histories?
> -----Original Message-----
> From: Bicking, David (HHoldings, IT)
> [mailto:David.Bicking@thehartford.com]
> Sent: Thursday, August 16, 2007 9:06 AM
> To: users@subversion.tigris.org
> Subject: RE: Complex VSS Migration - dealing with
> moved/branched version history
>
> > -----Original Message-----
> > From: Toby Johnson [mailto:toby@etjohnson.us]
> > Subject: Re: Complex VSS Migration - dealing with moved/branched
> > version history
> >
> > FYI, vss2svn is a completely separate project from Subversion; I am
> > copying in that mailing list so please join there
> > (http://www.pumacode.org/projects/vss2svn) to discuss further. See
> > additional comments below...
> >
> > Bicking, David (HHoldings, IT) wrote:
> > > I searched for days for information and tried many options,
> > and now am
> > > standing at a brick wall.
>
> <snip>
> >
> > First off, I would take a step back and ask yourself if
> > converting and maintaining all that history is really worth
> > it. I have seen lots of people (including myself) who
> > struggled with migrating a perfect copy of their VSS history
> > only to find out they ended up using it much less than they
> > anticipated. Particularly in your case I would say it sounds
> > like a clean start may be your best option going forward; it
> > may be more inconvenient in the short term but after a few
> > months you'll probably find yourself needing that history
> > very infrequently.
> >
> > Due to the state of your VSS repo it may be better to simply
> > put VSS into read-only mode for reference and then simply
> > export the contents and import to Subversion. You'll still
> > need to deal with all those shares, since that feature does
> > not translate to Subversion, although svn:externals is close
> > enough for many purposes.
>
> I totally agree with you, and this might be the decision we
> make, but I
> need to explore all the options first.
>
> >
> > > I want to migrate pieces of this behemoth into SVN, but
> cannot. I
> > > used vss2svn to get a dumpfile (12G) and tried to use
> > svndumpfilter to
> > > grab relevant pieces. Per the state of the data described in the
> > > prior paragraph, it fails every time. I found no reference
> > to any way
> > > to pull a specific project "tree" and get all its history,
> > which is quite odd.
> > > I would think this would be crucial. Why not permit
> > migration to pull
> > > the original data into place? A note stating "original
> > location is x"
> > > should be sufficient.
> > >
> >
> > I'm not sure exactly what you're asking here. If you are
> > asking why vss2svn doesn't allow migrating only a portion of
> > the VSS hierarchy instead of the whole thing, the answer to
> > that is rather technical but in a nutshell the way VSS stores
> > its history in "physical files" makes it very difficult to
> > convert only a portion of the repo since the program must
> > essentially walk the entire VSS tree in order to resolve
> > parent-child relationships.
> >
> > Most people who want to do what you are trying to accomplish
> > do use the svndumpfilter approach; if you can provide more
> > specific info as to why that is failing then we may be able
> > to provide a better suggestion or workaround.
> >
> > toby
> >
> >
>
> I did not explain it clearly. I am not asking for vss2svn to do
> something it does not. I'm not a member of that group (yet), so I
> cannot followup there. My question was directed at the
> functionality of
> svndumpfilter.
>
> I believe a highly useful feature, especially for migration purposes,
> would be to "streamline" the data being pulled out of a dumpfile via
> svndumpfilter. By streamline, I mean to look at the root
> path selected,
> say "/src/MyProject", trace back through the history, and
> migrate (copy)
> into the specified path any versions located outside that path. Thus,
> if two years ago, I copied, shared, or whatevered a file from
> "/src/TestVersion3OfMyProject/BusinessObjects" to
> "/src/MyProject/BusinessObjects", I might want to be able to
> pull those
> revisions into the filtered dumpfile, too.
>
> This would only happen for those revisions that fall outside
> the filter
> parameters, and only if an appropriate option were set (say,
> "--streamline"). For the sake of keeping accurate history, a property
> or the revision note would be updated on those revisions to state
> something like "this revision was originally in
> /src/TestVersion3OfMyProject/BusinessObjects".
>
> It would be nice if tools like vss2svn did this, but I think
> it is more
> effective and useful to include this as a feature of Subversion. It
> allows a wider range of utility, and makes it possible for
> administrators to break up single bloated repositories into multiple,
> more focused ones. Obviously, this amounts to a "heavy copy" of those
> revisions (which now reside in multiple locations). I think at times
> this is either desireable or not a significantly bad side effect.
>
> --
> David
>
*************************************************************************
This communication, including attachments, is
for the exclusive use of addressee and may contain proprietary,
confidential and/or privileged information. If you are not the intended
recipient, any use, copying, disclosure, dissemination or distribution is
strictly prohibited. If you are not the intended recipient, please notify
the sender immediately by return e-mail, delete this communication and
destroy all copies.
*************************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Complex VSS Migration - dealing with moved/branched version
history
Posted by "Bicking, David (HHoldings, IT)" <Da...@thehartford.com>.
> -----Original Message-----
> From: Toby Johnson [mailto:toby@etjohnson.us]
> Subject: Re: Complex VSS Migration - dealing with
> moved/branched version history
>
> FYI, vss2svn is a completely separate project from
> Subversion; I am copying in that mailing list so please join there
> (http://www.pumacode.org/projects/vss2svn) to discuss
> further. See additional comments below...
>
> Bicking, David (HHoldings, IT) wrote:
> > I searched for days for information and tried many options,
> and now am
> > standing at a brick wall.
<snip>
>
> First off, I would take a step back and ask yourself if
> converting and maintaining all that history is really worth
> it. I have seen lots of people (including myself) who
> struggled with migrating a perfect copy of their VSS history
> only to find out they ended up using it much less than they
> anticipated. Particularly in your case I would say it sounds
> like a clean start may be your best option going forward; it
> may be more inconvenient in the short term but after a few
> months you'll probably find yourself needing that history
> very infrequently.
>
> Due to the state of your VSS repo it may be better to simply
> put VSS into read-only mode for reference and then simply
> export the contents and import to Subversion. You'll still
> need to deal with all those shares, since that feature does
> not translate to Subversion, although svn:externals is close
> enough for many purposes.
I totally agree with you, and this might be the decision we make, but I
need to explore all the options first.
>
> > I want to migrate pieces of this behemoth into SVN, but cannot. I
> > used vss2svn to get a dumpfile (12G) and tried to use
> svndumpfilter to
> > grab relevant pieces. Per the state of the data described in the
> > prior paragraph, it fails every time. I found no reference
> to any way
> > to pull a specific project "tree" and get all its history,
> which is quite odd.
> > I would think this would be crucial. Why not permit
> migration to pull
> > the original data into place? A note stating "original
> location is x"
> > should be sufficient.
> >
>
> I'm not sure exactly what you're asking here. If you are
> asking why vss2svn doesn't allow migrating only a portion of
> the VSS hierarchy instead of the whole thing, the answer to
> that is rather technical but in a nutshell the way VSS stores
> its history in "physical files" makes it very difficult to
> convert only a portion of the repo since the program must
> essentially walk the entire VSS tree in order to resolve
> parent-child relationships.
>
> Most people who want to do what you are trying to accomplish
> do use the svndumpfilter approach; if you can provide more
> specific info as to why that is failing then we may be able
> to provide a better suggestion or workaround.
>
> toby
>
>
I did not explain it clearly. I am not asking for vss2svn to do
something it does not. I'm not a member of that group (yet), so I
cannot followup there. My question was directed at the functionality of
svndumpfilter.
I believe a highly useful feature, especially for migration purposes,
would be to "streamline" the data being pulled out of a dumpfile via
svndumpfilter. By streamline, I mean to look at the root path selected,
say "/src/MyProject", trace back through the history, and migrate (copy)
into the specified path any versions located outside that path. Thus,
if two years ago, I copied, shared, or whatevered a file from
"/src/TestVersion3OfMyProject/BusinessObjects" to
"/src/MyProject/BusinessObjects", I might want to be able to pull those
revisions into the filtered dumpfile, too.
This would only happen for those revisions that fall outside the filter
parameters, and only if an appropriate option were set (say,
"--streamline"). For the sake of keeping accurate history, a property
or the revision note would be updated on those revisions to state
something like "this revision was originally in
/src/TestVersion3OfMyProject/BusinessObjects".
It would be nice if tools like vss2svn did this, but I think it is more
effective and useful to include this as a feature of Subversion. It
allows a wider range of utility, and makes it possible for
administrators to break up single bloated repositories into multiple,
more focused ones. Obviously, this amounts to a "heavy copy" of those
revisions (which now reside in multiple locations). I think at times
this is either desireable or not a significantly bad side effect.
--
David
*************************************************************************
This communication, including attachments, is
for the exclusive use of addressee and may contain proprietary,
confidential and/or privileged information. If you are not the intended
recipient, any use, copying, disclosure, dissemination or distribution is
strictly prohibited. If you are not the intended recipient, please notify
the sender immediately by return e-mail, delete this communication and
destroy all copies.
*************************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Complex VSS Migration - dealing with moved/branched version history
Posted by Toby Johnson <to...@etjohnson.us>.
FYI, vss2svn is a completely separate project from Subversion; I am
copying in that mailing list so please join there
(http://www.pumacode.org/projects/vss2svn) to discuss further. See
additional comments below...
Bicking, David (HHoldings, IT) wrote:
> I searched for days for information and tried many options, and now am
> standing at a brick wall.
>
> I am evaluating Subversion as a replacement for VSS. This company has a
> huge repository (about 10G) in which branching and sharing were rampant.
> Worse, there was no plan for how projects were defined. Consequently,
> in addition to the typical VSS corruptions in the data, most projects
> (folders) have references to other projects somewhere deep in the
> history.
>
First off, I would take a step back and ask yourself if converting and
maintaining all that history is really worth it. I have seen lots of
people (including myself) who struggled with migrating a perfect copy of
their VSS history only to find out they ended up using it much less than
they anticipated. Particularly in your case I would say it sounds like a
clean start may be your best option going forward; it may be more
inconvenient in the short term but after a few months you'll probably
find yourself needing that history very infrequently.
Due to the state of your VSS repo it may be better to simply put VSS
into read-only mode for reference and then simply export the contents
and import to Subversion. You'll still need to deal with all those
shares, since that feature does not translate to Subversion, although
svn:externals is close enough for many purposes.
> I want to migrate pieces of this behemoth into SVN, but cannot. I used
> vss2svn to get a dumpfile (12G) and tried to use svndumpfilter to grab
> relevant pieces. Per the state of the data described in the prior
> paragraph, it fails every time. I found no reference to any way to pull
> a specific project "tree" and get all its history, which is quite odd.
> I would think this would be crucial. Why not permit migration to pull
> the original data into place? A note stating "original location is x"
> should be sufficient.
>
I'm not sure exactly what you're asking here. If you are asking why
vss2svn doesn't allow migrating only a portion of the VSS hierarchy
instead of the whole thing, the answer to that is rather technical but
in a nutshell the way VSS stores its history in "physical files" makes
it very difficult to convert only a portion of the repo since the
program must essentially walk the entire VSS tree in order to resolve
parent-child relationships.
Most people who want to do what you are trying to accomplish do use the
svndumpfilter approach; if you can provide more specific info as to why
that is failing then we may be able to provide a better suggestion or
workaround.
toby
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org