You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mike Wilcox <mi...@trouble.org.uk> on 2003/07/09 10:00:39 UTC
cvs2svn.py in version 6411
Hi,
I ran the latest cvs2svn over our cvs repository overnight, to see how
well the branch/tag development coped ;) and came up with a problem:
Traceback (most recent call last):
File "cvs2svn.py", line 2131, in ?
main()
File "cvs2svn.py", line 2127, in main
convert(ctx, start_pass=start_pass)
File "cvs2svn.py", line 2016, in convert
_passes[i](ctx)
File "cvs2svn.py", line 1959, in pass4
c.commit(dumper, ctx, sym_tracker)
File "cvs2svn.py", line 1761, in commit
path_deleted, closed_tags, closed_branches = \
File "cvs2svn.py", line 1032, in delete_path
branches, prune)
ValueError: unpack tuple of wrong size
I don't know python, but I suspect that the problem is:
*** cvs2svn.py.6411 2003-07-09 01:08:57.000000000 +0100
--- cvs2svn.py 2003-07-09 10:21:55.000000000 +0100
***************
*** 697,703 ****
# If we can't reach the dest, then we don't need to do anything.
if not parent.has_key(component):
! return None, ()
# Otherwise continue downward, dropping breadcrumbs.
this_entry_key = parent[component]
--- 697,703 ----
# If we can't reach the dest, then we don't need to do anything.
if not parent.has_key(component):
! return None, [], []
# Otherwise continue downward, dropping breadcrumbs.
this_entry_key = parent[component]
I am in the process of testing it, but it will be another 2 hours before
it gets to that point again!
Cheers,
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by Mike Wilcox <mi...@trouble.org.uk>.
kfogel@collab.net wrote:
>
> "It's right."
Thought as much.
>
>
>> 1) When converting our CVS repository, I want to split it into 3
>> SVN repositories, which means I create the repositories first, then
>> loop for each subproject in CVS. This was also to try to keep the
>> tags & branches separate for the different projects.
>>
>> Conversion of the first project into each svn repository works
>> fine, but the subsequent ones fail because the loader, prior to
>> creating 'trunk/<proj>', tries to re-create the 'trunk' directory -
>> which now already exists.
>
>
> I'll need a small, concrete example to understand what this means, I
> think. There are too many possible interpretations. For example, I
> don't know what you mean by "Conversion of the first project into
> each svn repository works fine." At first I thought you were putting
> each project in its own repository, but now it sounds like you're
> trying to put trunk in one repos, branches in another, and tags in a
> third? Or maybe you're using the word "repository" to mean
> "subdirectory within a repository"?...
OK. I realise I explained it in a way that wasn't obvious to understand
- I think I used repository where I meant project in at least one place.
Rather than explain my aims, I can prune the problem down to a similar
example using the test CVS repository:
tools/cvs2svn/test-data/main-cvsrepos
In my terms, each subdirectory of this would be a 'project'. I want to
divide the projects across 2 SVN repositories. eg:
proj -> svn-dev
interleaved -> svn-dev
full-prune -> svn-archive
partial-prune -> svn-archive
Initial commands to do this:
$ cd tools/cvs2svn
$ mkdir tmp
$ cd tmp
$ cvs2svn.py --create -s svn-dev ../test-data/main-cvsrepos/proj
$ cvs2svn.py -s svn-dev ../test-data/main-cvsrepos/proj/interleaved
$ cvs2svn.py --create -s svn-archive \
../test-data/main-cvsrepos/full-prune
$ cvs2svn.py -s svn-archive \
../test-data/main-cvsrepos/proj/partial-prune
The 2nd & 4th cvs2svn commands fail at the start of pass 5:
----- pass 5 -----
loading cvs2svn-dump into svnrepos
<<< Started new txn, based on original revision 1
* adding path : trunk ...svn: Item already exists in filesystem
svn: file already exists: filesystem `svnrepos/db', transaction `k',
path `trunk'
*This* is the problem I was trying (and failing) to relate.
Then...
Because I know that the above commands would jumble the project contents
together inside the SVN repositories, I would have added the "--trunk"
option:
$ cvs2svn.py --create -s svn-dev --trunk trunk/proj \
../test-data/main-cvsrepos/proj
This results in:
Traceback (most recent call last):
File "../cvs2svn.py", line 2131, in ?
main()
File "../cvs2svn.py", line 2127, in main
convert(ctx, start_pass=start_pass)
File "../cvs2svn.py", line 2016, in convert
_passes[i](ctx)
File "../cvs2svn.py", line 1980, in pass4
sym_tracker.finish(dumper, ctx)
File "../cvs2svn.py", line 1599, in finish
self.fill_branch(dumper, ctx, name, [1])
File "../cvs2svn.py", line 1570, in fill_branch
self.fill_name(dumper, ctx, branch, None, jit_new_rev)
File "../cvs2svn.py", line 1544, in fill_name
is_tag, jit_new_rev)
File "../cvs2svn.py", line 1435, in copy_descend
dumper.copy_path(src_path, parent_rev, copy_dst, val)
File "../cvs2svn.py", line 905, in copy_path
entries)
File "../cvs2svn.py", line 603, in change_path
for ent in new_val.keys():
AttributeError: 'NoneType' object has no attribute 'keys'
However, I realise that doing this might have hit your limitation you
mentioned lower down.
Incidentally - if I *also* add the "--branches" option, the fault
disappears.
> Oh my goodness. I think what you're trying may be beyond cvs2svn's
> capabilities right now. I'm interested in supporting it eventually,
> but right now it's more important to fix bugs in the core
> functionality. One of the current issues is that cvs2svn needs
> documentation -- and if it had docs, they'd probably say not to try
> passing multi-component paths as arguments to --trunk, --tags, etc.
> :-)
Then I understand the limitation at least :-)
I will say that I went down the route of using one cvs2svn command per
CVS project *because* of the documentation - it warns that converting a
repository and the CVSROOT directory 'might not be what you wanted'.
But, once you do have one command per project, you really need the 3
options to keep the projects contents, branches & tags separate from
other projects.
However, there are 2 workarounds that I can think of:
1. Issue each project-oriented cvs2svn command, but between each one,
manually re-arrange the SVN repository so that the next project won't
collide, or cause the loader to die.
2. Re-arrange the cvsroot into N separate directories, moving the
projects around as required (so creating N 'new' CVS repositories). Then
issue one repository-oriented cvs2svn command per 'new' CVS repository.
There are probably more...
Cheers,
Mike
(I've not forgotten the file for the very first problem)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by kf...@collab.net.
Mike Wilcox <mi...@trouble.org.uk> writes:
> As an aside, I was a little conerned when I saw this kind of thing
> in the output:
>
> <<< Started new txn, based on original revision 38
> * adding path : tags/T_ALL_INITIAL_FILES ...COPIED... done.
> * deleting path : tags/T_ALL_INITIAL_FILES/partial-prune ... done.
> * deleting path : tags/T_ALL_INITIAL_FILES/single-files ... done.
>
> The tag T_ALL_INITIAL_FILES only seems to be used by the "project"
> proj, so, while deleting paths related to 2 of the other projects (why
> not all other projects?) isn't harmful, I did wonder why it was even
> necessary.
Because if we didn't delete them, they'd be included in the tag, which
would be incorrect :-). (In other words, the copy that created the
tag picked up a bit too much, so we have to remove the extra stuff.
cvs2svn is all about finding semi-minimal sets of copies and deletes
that will create tags and branches correctly.)
> The problem is almost certainly my understanding about the way in
> which your process works through the order a CVS repository has been
> created. If you think this is right, it's enough for you to say "its
> right".
"It's right."
> 1) When converting our CVS repository, I want to split it into 3 SVN
> repositories, which means I create the repositories first, then loop
> for each subproject in CVS. This was also to try to keep the tags &
> branches separate for the different projects.
>
> Conversion of the first project into each svn repository works fine,
> but the subsequent ones fail because the loader, prior to creating
> 'trunk/<proj>', tries to re-create the 'trunk' directory - which now
> already exists.
I'll need a small, concrete example to understand what this means, I
think. There are too many possible interpretations. For example, I
don't know what you mean by "Conversion of the first project into each
svn repository works fine." At first I thought you were putting each
project in its own repository, but now it sounds like you're trying to
put trunk in one repos, branches in another, and tags in a third? Or
maybe you're using the word "repository" to mean "subdirectory within
a repository"?...
> 2) Back to this problem... As a workaround to 1), I adjusted my
> conversion process to create the SVN repos, then loop through the CVS
> repositories, but using the --trunk, --branches and --tags options to
> control placement into something akin to the "old"
> hierarchy. Afterwards, I later move the created structures back into
> the "new" hierarchy of "trunk/<proj>" etc.
>
> It seems that the command option for --branch is causing the
> problem. This doesn't produce any tags:
>
> $ ../cvs2svn.py -s svnrepos --trunk=<proj>/trunk
> --branches=<proj>/branches --tags=<proj>/tags --no-prune
> ../cvsroot/<proj>
>
> But these do:
>
> $ ../cvs2svn.py -s svnrepos --trunk=<proj>/trunk --no-prune
> ../cvsroot/<proj>
>
> $ ../cvs2svn.py -s svnrepos --trunk=<proj>/trunk --tags=<proj>/tags
> --no-prune ../cvsroot/<proj>
>
> So, similar to your suggested command at the top, the following fails:
> ../cvs2svn.py --create -s svnrepos --trunk=test1/trunk
> --branches=test1/branches --tags=test1/tags ../test-data/main-cvsrepos/
>
> Hopefully, this will help you fix it. It's beyond my python knowledge
> (or cvs2svn knowledge) to attempt a patch.
>
> Back to you later on a test case for the original problem :-)
Oh my goodness. I think what you're trying may be beyond cvs2svn's
capabilities right now. I'm interested in supporting it eventually,
but right now it's more important to fix bugs in the core
functionality. One of the current issues is that cvs2svn needs
documentation -- and if it had docs, they'd probably say not to try
passing multi-component paths as arguments to --trunk, --tags, etc. :-)
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by Mike Wilcox <mi...@trouble.org.uk>.
kfogel@collab.net wrote:
> Mike Wilcox <mi...@trouble.org.uk> writes:
>
> No troubles expected; you should have gotten tags. When I do
>
> $ cd tools/cvs2svn
> $ mkdir tmp
> $ cd tmp
> $ ../cvs2svn.py --create -s svnrepos ../test-data/main-cvsrepos/
> $ svn log -v file://`pwd`/svnrepos
> [...]
> $
>
> ...I get plenty of tags.
The above gave me tags. As an aside, I was a little conerned when I saw
this kind of thing in the output:
<<< Started new txn, based on original revision 38
* adding path : tags/T_ALL_INITIAL_FILES ...COPIED... done.
* deleting path : tags/T_ALL_INITIAL_FILES/partial-prune ... done.
* deleting path : tags/T_ALL_INITIAL_FILES/single-files ... done.
The tag T_ALL_INITIAL_FILES only seems to be used by the "project" proj,
so, while deleting paths related to 2 of the other projects (why not all
other projects?) isn't harmful, I did wonder why it was even necessary.
The problem is almost certainly my understanding about the way in which
your process works through the order a CVS repository has been created.
If you think this is right, it's enough for you to say "its right".
> Thanks. Also, if you can find a small subset of your repository that
> demonstrates the failure to produce tags, I'm eager to see that too!
I've been trying with the repository pruned to a single file, with many
branches & tags. I can process it to get many tags sometimes, and none
at others. It seems to be related to the way I call cvs2svn, and this
brings to mind another problem to report:
1) When converting our CVS repository, I want to split it into 3 SVN
repositories, which means I create the repositories first, then loop for
each subproject in CVS. This was also to try to keep the tags & branches
separate for the different projects.
Conversion of the first project into each svn repository works fine, but
the subsequent ones fail because the loader, prior to creating
'trunk/<proj>', tries to re-create the 'trunk' directory - which now
already exists.
2) Back to this problem... As a workaround to 1), I adjusted my
conversion process to create the SVN repos, then loop through the CVS
repositories, but using the --trunk, --branches and --tags options to
control placement into something akin to the "old" hierarchy.
Afterwards, I later move the created structures back into the "new"
hierarchy of "trunk/<proj>" etc.
It seems that the command option for --branch is causing the problem.
This doesn't produce any tags:
$ ../cvs2svn.py -s svnrepos --trunk=<proj>/trunk
--branches=<proj>/branches --tags=<proj>/tags --no-prune ../cvsroot/<proj>
But these do:
$ ../cvs2svn.py -s svnrepos --trunk=<proj>/trunk --no-prune
../cvsroot/<proj>
$ ../cvs2svn.py -s svnrepos --trunk=<proj>/trunk --tags=<proj>/tags
--no-prune ../cvsroot/<proj>
So, similar to your suggested command at the top, the following fails:
../cvs2svn.py --create -s svnrepos --trunk=test1/trunk
--branches=test1/branches --tags=test1/tags ../test-data/main-cvsrepos/
Hopefully, this will help you fix it. It's beyond my python knowledge
(or cvs2svn knowledge) to attempt a patch.
Back to you later on a test case for the original problem :-)
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by kf...@collab.net.
Mike Wilcox <mi...@trouble.org.uk> writes:
> The fix certainly did enough to get the conversion through to the
> finish. It created lots of branches (which I expected), but no tags (I
> did expect a lot of these though). If you think I *should* have got
> tags out of this version, I'll try to track down why I didn't; if
> you're still expecting trouble, I won't try just yet :-)
No troubles expected; you should have gotten tags. When I do
$ cd tools/cvs2svn
$ mkdir tmp
$ cd tmp
$ ../cvs2svn.py --create -s svnrepos ../test-data/main-cvsrepos/
$ svn log -v file://`pwd`/svnrepos
[...]
$
...I get plenty of tags.
> I'll give that a go - I think I have the failure point in the logs. It
> might not be today though!
Thanks. Also, if you can find a small subset of your repository that
demonstrates the failure to produce tags, I'm eager to see that too!
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by Mike Wilcox <mi...@trouble.org.uk>.
kfogel@collab.net wrote:
>Your patch looked right on to me, I applied it.
>
The fix certainly did enough to get the conversion through to the
finish. It created lots of branches (which I expected), but no tags (I
did expect a lot of these though). If you think I *should* have got tags
out of this version, I'll try to track down why I didn't; if you're
still expecting trouble, I won't try just yet :-)
> If you have time to
>narrow down your repository to just the part that stimulated this bug,
>could you post it? (I suspect what you'd need to do is prune away
>most of the repository, leaving just the full path down to a
>particular subdirectory or two, and invoke cvs2svn on the top of the
>pruned repository, just as you invoked it on the unpruned one before.)
>
I'll give that a go - I think I have the failure point in the logs. It
might not be today though!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by kf...@collab.net.
Mike Wilcox <mi...@trouble.org.uk> writes:
> *** cvs2svn.py.6411 2003-07-09 01:08:57.000000000 +0100
> --- cvs2svn.py 2003-07-09 10:21:55.000000000 +0100
> ***************
> *** 697,703 ****
>
> # If we can't reach the dest, then we don't need to do anything.
> if not parent.has_key(component):
> ! return None, ()
>
> # Otherwise continue downward, dropping breadcrumbs.
> this_entry_key = parent[component]
> --- 697,703 ----
>
> # If we can't reach the dest, then we don't need to do anything.
> if not parent.has_key(component):
> ! return None, [], []
>
> # Otherwise continue downward, dropping breadcrumbs.
> this_entry_key = parent[component]
>
> I am in the process of testing it, but it will be another 2 hours
> before it gets to that point again!
Your patch looked right on to me, I applied it. If you have time to
narrow down your repository to just the part that stimulated this bug,
could you post it? (I suspect what you'd need to do is prune away
most of the repository, leaving just the full path down to a
particular subdirectory or two, and invoke cvs2svn on the top of the
pruned repository, just as you invoked it on the unpruned one before.)
Thanks for the fix.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: cvs2svn.py in version 6411
Posted by kf...@collab.net.
Mike Wilcox <mi...@trouble.org.uk> writes:
> I ran the latest cvs2svn over our cvs repository overnight, to see how
> well the branch/tag development coped ;) and came up with a problem:
>
> Traceback (most recent call last):
> File "cvs2svn.py", line 2131, in ?
> main()
> File "cvs2svn.py", line 2127, in main
> convert(ctx, start_pass=start_pass)
> File "cvs2svn.py", line 2016, in convert
> _passes[i](ctx)
> File "cvs2svn.py", line 1959, in pass4
> c.commit(dumper, ctx, sym_tracker)
> File "cvs2svn.py", line 1761, in commit
> path_deleted, closed_tags, closed_branches = \
> File "cvs2svn.py", line 1032, in delete_path
> branches, prune)
> ValueError: unpack tuple of wrong size
>
> I don't know python, but I suspect that the problem is:
I'll take a look now -- thanks for the patch-as-tentative-diagnosis,
that helps a lot :-).
If the problem turns out to be more complex, I'll probably ask you to
send me a small CVS repository that reproduces the problem, plus more
details (such as, what was the exact invocation of cvs2svn?). Just
giving you fair warning now... :-)
-Karl
> *** cvs2svn.py.6411 2003-07-09 01:08:57.000000000 +0100
> --- cvs2svn.py 2003-07-09 10:21:55.000000000 +0100
> ***************
> *** 697,703 ****
>
> # If we can't reach the dest, then we don't need to do anything.
> if not parent.has_key(component):
> ! return None, ()
>
> # Otherwise continue downward, dropping breadcrumbs.
> this_entry_key = parent[component]
> --- 697,703 ----
>
> # If we can't reach the dest, then we don't need to do anything.
> if not parent.has_key(component):
> ! return None, [], []
>
> # Otherwise continue downward, dropping breadcrumbs.
> this_entry_key = parent[component]
>
> I am in the process of testing it, but it will be another 2 hours
> before it gets to that point again!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org