You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Kai Bartels <ka...@picturesafe.de> on 2004/02/16 10:07:25 UTC
Re: [w] cvs2svn takes very long time to execute (days!)
Hi Martin, Chris, *!
On Fri, Feb 13, 2004 at 02:39:52PM -0500, Martin Gadbois wrote:
> I am trying to convert our CVS to Subversion, but cvs2svn takes a very long
> time to execute, and
> things seems to slow down with time. Pass 4 takes the longest.
>
> At one point I 'imported' 1 year of commits before stopping.
>
> I simply did not have the patience to complete it, mostly due to the
> slowdown.
>
> Our CVS tree is big:
> - - earliest commit is ~2000
> - - cvsroot is 1.4G, ~17700 files
> - - One tag is used per build, we are now at build 2700.
> - - A dozen or so of branches.
> - - I had to modify cvs2svn slightly to accept branches with two different
> names
I had the same probs with cvs2svn. Passes 1 to 3 went pretty fast. But when
the programms output said "commiting revision xxxx" things started to slow
down. That is, the first "commits" went real fast, but when xxxx evolved to
around 18000 to 20000, each commit needed around 20 to 30 seconds (with our
repository I reached Jan 2002 at that point, so I stopped the conversion).
Our repository holds about 2.1G but I don't think that size is the problem,
as there was a report on this list that a repository of about 10G had been
converted in 10h. I suspect, it's about the number of commits.
(BTW: Is the author of cvs2svn reading this list?)
> Is it common to take that much time? (Should I just go on a week vacation
> while it converts?)
Good idea, didn't think of that yet; I should do that! :^))
> Any hints on what takes that much time? (I profiled the script and most of
> the time was to execute
> 'co' and in enroot_names())
Is this a possible explanation for the slow down (i.e. that early commits
are fast and later are slow)?
On Fri, Feb 13, 2004 at 11:50:49AM -0800, Chris Wein wrote:
> Try creating a dumpfile only and see if that helps. I am (literally!)
> converting a 700MB repository with 15000+ revisions and its taken about
> 2 hours (so far, 90% done) on RH7.3 P4@3GHZ. Once the dumpfile is done
> then import into svn.
1) I'm still not sure if the slowing down pass I can observe is pass 4
or pass 5 (I think pass 4 is dumpfile creation!?).
2) If pass 4 is fast and 5 the slow-down: Would importing of a dumpfile
manually be faster?
regards, Kai
--
"Commerce, n. A kind of transaction in which A plunders from B the goods of
C, and for compensation B picks the pocket of D of money beloging to E."
( what is e-commerce, then? ) <Ambrose Bierce>
KBartels@picturesafe.de + www.picturesafe.de
picturesafe GmbH, Lueerstr. 3, D-30175 Hannover fon:+49 511 85620 56
++ PGP Key fingerprint 1324 98A8 3BF8 4166 05E1 82D3 E168 6849 6022 5D32 ++
-----BEGIN GEEK CODE BLOCK-----
VERSION: 3.12
GCS d- s: a C++$ UL/S++$ P++ L++ E W-- N w--- !O !M PS+ PE Y+ PGP+ t R@ tv+>-
b+>++ DI++ G e+++ h* y?
------END GEEK CODE BLOCK------
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [w] cvs2svn takes very long time to execute (days!)
Posted by Chris Wein <cw...@mobilygen.com>.
For what its worth, it took about 90 minutes to dump our repository to a
dumpfile and then about another 2 hours to load it to the repository.
The dumpfile showed 16703 commits and the dumpfile was 1.3G with the
repository having approximately 4000 files. The machine was a 3GHZ P4
Linux box with 1G of memory.
I might be that I don't have enough commits? It might be that I have
lots of RAM?
C
On Mon, 2004-02-16 at 02:07, Kai Bartels wrote:
> Hi Martin, Chris, *!
>
> On Fri, Feb 13, 2004 at 02:39:52PM -0500, Martin Gadbois wrote:
> > I am trying to convert our CVS to Subversion, but cvs2svn takes a very long
> > time to execute, and
> > things seems to slow down with time. Pass 4 takes the longest.
> >
> > At one point I 'imported' 1 year of commits before stopping.
> >
> > I simply did not have the patience to complete it, mostly due to the
> > slowdown.
> >
> > Our CVS tree is big:
> > - - earliest commit is ~2000
> > - - cvsroot is 1.4G, ~17700 files
> > - - One tag is used per build, we are now at build 2700.
> > - - A dozen or so of branches.
> > - - I had to modify cvs2svn slightly to accept branches with two different
> > names
>
> I had the same probs with cvs2svn. Passes 1 to 3 went pretty fast. But when
> the programms output said "commiting revision xxxx" things started to slow
> down. That is, the first "commits" went real fast, but when xxxx evolved to
> around 18000 to 20000, each commit needed around 20 to 30 seconds (with our
> repository I reached Jan 2002 at that point, so I stopped the conversion).
>
> Our repository holds about 2.1G but I don't think that size is the problem,
> as there was a report on this list that a repository of about 10G had been
> converted in 10h. I suspect, it's about the number of commits.
> (BTW: Is the author of cvs2svn reading this list?)
>
> > Is it common to take that much time? (Should I just go on a week vacation
> > while it converts?)
>
> Good idea, didn't think of that yet; I should do that! :^))
>
> > Any hints on what takes that much time? (I profiled the script and most of
> > the time was to execute
> > 'co' and in enroot_names())
>
> Is this a possible explanation for the slow down (i.e. that early commits
> are fast and later are slow)?
>
> On Fri, Feb 13, 2004 at 11:50:49AM -0800, Chris Wein wrote:
> > Try creating a dumpfile only and see if that helps. I am (literally!)
> > converting a 700MB repository with 15000+ revisions and its taken about
> > 2 hours (so far, 90% done) on RH7.3 P4@3GHZ. Once the dumpfile is done
> > then import into svn.
>
> 1) I'm still not sure if the slowing down pass I can observe is pass 4
> or pass 5 (I think pass 4 is dumpfile creation!?).
>
> 2) If pass 4 is fast and 5 the slow-down: Would importing of a dumpfile
> manually be faster?
>
>
> regards, Kai
>
>
> --
> "Commerce, n. A kind of transaction in which A plunders from B the goods of
> C, and for compensation B picks the pocket of D of money beloging to E."
> ( what is e-commerce, then? ) <Ambrose Bierce>
> KBartels@picturesafe.de + www.picturesafe.de
> picturesafe GmbH, Lueerstr. 3, D-30175 Hannover fon:+49 511 85620 56
> ++ PGP Key fingerprint 1324 98A8 3BF8 4166 05E1 82D3 E168 6849 6022 5D32 ++
> -----BEGIN GEEK CODE BLOCK-----
> VERSION: 3.12
> GCS d- s: a C++$ UL/S++$ P++ L++ E W-- N w--- !O !M PS+ PE Y+ PGP+ t R@ tv+>-
> b+>++ DI++ G e+++ h* y?
> ------END GEEK CODE BLOCK------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
--
Chris Wein
Software Engineer
Mobilygen Corp.
E-Mail : cwein@mobilygen.com
Phone : 408-869-4035
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [w] cvs2svn takes very long time to execute (days!)
Posted by kf...@collab.net.
Kai Bartels <ka...@picturesafe.de> writes:
> (BTW: Is the author of cvs2svn reading this list?)
Yes, all the authors read it, as far as I know.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [w] cvs2svn takes very long time to execute (days!)
Posted by kf...@collab.net.
Martin Gadbois <ma...@colubris.com> writes:
> Is that recent? I tried to import over the weekend using friday's
> SVN version of cvs2svn, but I stopped it this monday morning when I
> realized I was only in may 2001... :-)
I don't know whether the one-dumpfile-per-revision change has anything
to do with these speed problems. It was done to solve a space problem.
As for whether it's recent: 'svn log -v' is your friend :-).
(I can't recall when it happened, but svn log -v is how I'd find out.)
> | Try running with --dump-only, then create a fresh repos and svnadmin
> | load the dumpfile. Does that go faster?
>
> No it doesn't. Since I got this hint, I tried with --dump-only ... only.
>
> I will try with the latest SVN and I will try Psyco.. see if that
> makes a difference...
And the latest cvs2svn, which has gotten many improvements lately.
Thanks,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [w] cvs2svn takes very long time to execute (days!)
Posted by Martin Gadbois <ma...@colubris.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
kfogel@collab.net wrote:
|
| cvs2svn creates one dumpfile per revision now, loads that small
| dumpfile, removes it, then continues on to the next. We used to do
| one huge dumpfile, but this meant cvs2svn's peak tmp space
| requirements were very high, so we changed.
Is that recent? I tried to import over the weekend using friday's SVN version of cvs2svn, but I
stopped it this monday morning when I realized I was only in may 2001... :-)
|
| Try running with --dump-only, then create a fresh repos and svnadmin
| load the dumpfile. Does that go faster?
|
No it doesn't. Since I got this hint, I tried with --dump-only ... only.
I will try with the latest SVN and I will try Psyco.. see if that makes a difference...
- --
"Windows might take you from 0 to 60 faster, but to go to 100 you need Unix."
==============
Martin Gadbois
S/W Developper
Colubris Networks Inc.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFAMRdm9Y3/iTTCEDkRAp4CAJ9jKIn+8E9EM0uaEdHvKMEGytNQgQCgxMsX
U7Zc5aQDnvRr72X4YnCNwTs=
=MyOJ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: [w] cvs2svn takes very long time to execute (days!)
Posted by kf...@collab.net.
Kai Bartels <ka...@picturesafe.de> writes:
> Is this a possible explanation for the slow down (i.e. that early commits
> are fast and later are slow)?
No, I'm not sure what's causing this yet.
> On Fri, Feb 13, 2004 at 11:50:49AM -0800, Chris Wein wrote:
> > Try creating a dumpfile only and see if that helps. I am (literally!)
> > converting a 700MB repository with 15000+ revisions and its taken about
> > 2 hours (so far, 90% done) on RH7.3 P4@3GHZ. Once the dumpfile is done
> > then import into svn.
>
> 1) I'm still not sure if the slowing down pass I can observe is pass 4
> or pass 5 (I think pass 4 is dumpfile creation!?).
cvs2svn creates one dumpfile per revision now, loads that small
dumpfile, removes it, then continues on to the next. We used to do
one huge dumpfile, but this meant cvs2svn's peak tmp space
requirements were very high, so we changed.
> 2) If pass 4 is fast and 5 the slow-down: Would importing of a dumpfile
> manually be faster?
Try running with --dump-only, then create a fresh repos and svnadmin
load the dumpfile. Does that go faster?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org