You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Roland Dreier <ro...@topspin.com> on 2004/02/15 19:08:57 UTC
Speeding up cvs2svn (was Re: cvs2svn takes very long time to execute (days!))
Hi Martin,
I've converted an even bigger repository than yours (it created a 10GB
Subversion dump file with something like 25K revisions). I had to do
several things to get cvs2svn.py to finish in a reasonable time (my
original run looked like it would take more than a month!).
I've been meaning to send this to the svn list for a while, since it
may prove useful to others. I'd also be interested in suggestions for
further improvements.
One warning: almost everything I did is fairly Linux-specific, so it
may not help if you are running some other operating system. Also,
having a lot of RAM will help performance.
I started by looking at what was taking all the time while running
cvs2svn.py. It seemed that all the time was spent accessing the
various .db files, so I first tried creating a tmpfs filesystem (you
can use a command like
mount -t tmpfs -o size=3000M,mode=777 tmpfs /svntmp
to create the filesystem, and then run cvs2svn.py with /svntmp as its
working directory). You will need enough RAM so that the tmpfs doesn't start
swapping (or else you're back to the same old bad performance).
However, I found that this didn't help as much as I thought it would.
Surprisingly, cvs2svn.py was still only using a small fraction of the
CPU, even though it should have been almost CPU-bound (since nearly
all of the IO would be done to the tmpfs, which is in RAM). Using
strace, I found that cvs2svn.py was spending a lot of time sleeping
for 1 second using select().
I tracked this down to __memp_alloc() in the db library; when low on
memory, that function sleeps for a second to let other threads free
memory. (This sleep is present in at least versions 4.1 and 4.2 of
the db library) Of course this is completely useless in our case,
since cvs2svn.py is single threaded. I wrote an LD_PRELOAD-able
library (attached to this email) that turns 1 second select() sleeps
into NOPs. One you build select-preload.so, you can use it like
"LD_PRELOAD=<full path to select-preload.so> cvs2svn.py [cvs2svn.py
parameters]".
That helped a little, but it also made me investigate how much memory
the db library was using to cache the .db files. It turns out that db
files created using the python anydbm use a default cache size that is
far to small for the db files that cvs2svn.py creates for large
repositories (they may be 100s of MB). When I looked at the strace
output for cvs2svn.py, it seemed that the script was spending a lot of
time just shuffling blocks in and out of the db cache, without doing
much useful work.
Unfortunately there doesn't seem to be any way to specify a
non-default cache size using anydbm. Therefore I changed my copy of
cvs2svn.py to use bsddb -- in patch form:
-import anydbm
+import bsddb
I deleted the anydbm check (since I'm no longer using anydbm) and
added a cache_size variable:
# Use 500MB db file caches (this uses a lot of RAM)
cache_size = 500 * 1024 * 1024
I ran on a machine with 2GB of RAM. You may want to adjust cache_size
down if you have less. Then I changed all the anydbm.open() calls to
bsddb.hashopen() calls. For example:
- self.nodes_db = anydbm.open(self.nodes_db_file, 'n')
+ self.nodes_db = bsddb.hashopen(self.nodes_db_file, 'n', 0666, 4096, 0, 1, cache_size)
With all of these changes, things went a lot faster and the whole
conversion took only a couple of days (instead of more than a month as
I initially thought).
As a side note, I'm not sure if it's really worth using the
abstraction of anydbm in the cvs2svn.py script. Subversion proper
pretty much requires a modern BSD db library to be installed, so I
don't think it's asking too much to require the bsddb3 python package
for cvs2svn.py.
There was one more improvement that I thought of but didn't get around
to coding or trying. Pass 2 (cleaning up the revision timestamps)
took a fairly long time, and the algorithm used seems suboptimal.
Perhaps some cvs2svn hackers can tell me if the following might work
(and run faster):
- Add a new field to the pass 1 .revs output that has the resync
(if any) for each file revision. Don't generate .resync.
- In pass 2, sort the pass 1 output by log+author digest. Then
go through the output, dealing with one log+author digest at a
time, and update the time stamps.
This avoids having to create the whole resync hash table -- in my
case, the .resync file was 47 MB, and looking up commits in the hash
table seemed to take a long time.
Anyway, to summarize my findings on cvs2svn.py performance:
1. Use tmpfs for a working directory to avoid disk IO.
2. Use a LD_PRELOAD-ed library to get rid of useless sleeps in the
BSD db library.
3. Increase the db cache size to avoid shuffling blocks in and out of
cache.
4. It may be worth changing the pass 2 algorithm to increase
performance.
I'd be very interested to hear any reactions to these ideas.
Best,
Roland
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to execute (days!))
Posted by kf...@collab.net.
"C. Michael Pilato" <cm...@collab.net> writes:
> -1 on making the --in-memory version the default. Generally speaking,
> folks are more likely to run out of RAM before running out of disk.
> It would be Badness for someone who had a big repository to run their
> conversion for hours and then have it fail because they hit the
> generally much smaller of the two available boundaries. Remember,
> most OSes (with which I'm familiar, at least) have some notion of a
> ramdisk -- folks can use that today without changes to cvs2svn itself.
I have to agree with Mike here. The most important thing is that
cvs2svn work out of the box, no matter what the input. Even
performance is secondary to that.
(But, if we can find a make it perform better without more thought on
the part of the user, then we win all around.)
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to execute (days!))
Posted by "C. Michael Pilato" <cm...@collab.net>.
Tobias Ringström <to...@ringstrom.mine.nu> writes:
> I'm thinking of adding an --in-memory option. The question of what
> should be the default is interesting. If we can make the DB-version
> faster, the question is moot, of course. As it stands right now, I'd
> personally prefer to make in-memory be default, and add a --huge-repos
> options for slow on-disk processing. I wouldn't dare to do that
> without support on the dev list of course.
-1 on making the --in-memory version the default. Generally speaking,
folks are more likely to run out of RAM before running out of disk.
It would be Badness for someone who had a big repository to run their
conversion for hours and then have it fail because they hit the
generally much smaller of the two available boundaries. Remember,
most OSes (with which I'm familiar, at least) have some notion of a
ramdisk -- folks can use that today without changes to cvs2svn itself.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to
execute (days!))
Posted by Tobias Ringström <to...@ringstrom.mine.nu>.
kfogel@collab.net wrote:
> Oh-ho! The marshalling is more time-consuming than I thought, then.
> (I hadn't profiled it yet, since correctness issues are a higher
> priority to me right now, but still I admit this is surprising.)
For my test case it took the majority of the time. If you have the
time, please try the new profile-cvs2svn.py script and view the data in
kcachegrind (which btw was very easy to build from source). It's quite
illuminating.
> A thought: we could cut down the marshalling quite a bit, by making
> marshal/unmarshal behavior an optional flag to the Database class, and
> passing it as false for those database which use only Python strings
> as keys/values. The Database could still test keys/values for sanity
> before using them, assuming type tests are still cheap (!).
>
> I feel funny about using in-memory hashes. cvs2svn.py should scale
> well by default. Do you plan to automagically switch to a disk
> database if the hash count exceeds a certain magic number?
I'm thinking of adding an --in-memory option. The question of what
should be the default is interesting. If we can make the DB-version
faster, the question is moot, of course. As it stands right now, I'd
personally prefer to make in-memory be default, and add a --huge-repos
options for slow on-disk processing. I wouldn't dare to do that without
support on the dev list of course.
>>Please note that cvs2svn is not yet 1.0 material, and it still has
>>correctness bugs, i.e. the content of tags and branches can be
>>incorrect. I'm about to commit a new tool that looks for such errors.
>
> You, sir, rock my socks.
I try to. :-)
The script is a bit raw at the moment, but I hope to work more on it
soon, and I invite others to do so as well. I added it to the repos so
that the rest of you can use it to hunt for correctness bugs.
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to
execute (days!))
Posted by Tobias Ringström <to...@ringstrom.mine.nu>.
Roland Dreier wrote:
> kfogel> I feel funny about using in-memory hashes. cvs2svn.py
> kfogel> should scale well by default. Do you plan to
> kfogel> automagically switch to a disk database if the hash count
> kfogel> exceeds a certain magic number?
>
> Yes, I agree. Keeping the disk database but using a much larger cache
> gives a huge performance boost while still handling big repositories.
Sure, but you used a ram disk to store the databases, which is *a lot*
less efficient than avoiding the db altogether. Using on-disk bsddb
with a large cache makes a lot of sense of course. Here's some numbers
from my smallish test case:
Plain trunk cvs2svn : 558 s
bsddb with large cache : 347 s
In-memory hash : 110 s
This is just one repository. As soon as I'm convinced that the
in-memory code is correct, I'll post it so you can try it out yourself.
> I sort of feel that refinecvs is the "keep everything in memory" tool,
> and cvs2svn is the "handle repositories too big for memory" tool.
For clarity, it's just the repository structure that you need to keep in
memory, not the file contents.
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to execute (days!))
Posted by Roland Dreier <ro...@topspin.com>.
kfogel> I feel funny about using in-memory hashes. cvs2svn.py
kfogel> should scale well by default. Do you plan to
kfogel> automagically switch to a disk database if the hash count
kfogel> exceeds a certain magic number?
Yes, I agree. Keeping the disk database but using a much larger cache
gives a huge performance boost while still handling big repositories.
I sort of feel that refinecvs is the "keep everything in memory" tool,
and cvs2svn is the "handle repositories too big for memory" tool.
- Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to execute (days!))
Posted by kf...@collab.net.
Tobias Ringström <to...@ringstrom.mine.nu> writes:
> Interesting, but for 1-3, it would be a lot faster to avoid the DB lib
> completely and use in-memory python hashes. One of the things that
> take a long time is the marshalling done when passing data to and from
> the DB. I have some code to do this, and I hope to commit it to trunk
> in a couple of days at the latest.
Oh-ho! The marshalling is more time-consuming than I thought, then.
(I hadn't profiled it yet, since correctness issues are a higher
priority to me right now, but still I admit this is surprising.)
A thought: we could cut down the marshalling quite a bit, by making
marshal/unmarshal behavior an optional flag to the Database class, and
passing it as false for those database which use only Python strings
as keys/values. The Database could still test keys/values for sanity
before using them, assuming type tests are still cheap (!).
I feel funny about using in-memory hashes. cvs2svn.py should scale
well by default. Do you plan to automagically switch to a disk
database if the hash count exceeds a certain magic number?
> Please note that cvs2svn is not yet 1.0 material, and it still has
> correctness bugs, i.e. the content of tags and branches can be
> incorrect. I'm about to commit a new tool that looks for such errors.
You, sir, rock my socks.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to
execute (days!))
Posted by Tobias Ringström <to...@ringstrom.mine.nu>.
Roland Dreier wrote:
> Anyway, to summarize my findings on cvs2svn.py performance:
>
> 1. Use tmpfs for a working directory to avoid disk IO.
> 2. Use a LD_PRELOAD-ed library to get rid of useless sleeps in the
> BSD db library.
> 3. Increase the db cache size to avoid shuffling blocks in and out of
> cache.
> 4. It may be worth changing the pass 2 algorithm to increase
> performance.
>
> I'd be very interested to hear any reactions to these ideas.
Interesting, but for 1-3, it would be a lot faster to avoid the DB lib
completely and use in-memory python hashes. One of the things that take
a long time is the marshalling done when passing data to and from the
DB. I have some code to do this, and I hope to commit it to trunk in a
couple of days at the latest.
Please note that cvs2svn is not yet 1.0 material, and it still has
correctness bugs, i.e. the content of tags and branches can be
incorrect. I'm about to commit a new tool that looks for such errors.
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to
execute (days!))
Posted by Martin Gadbois <ma...@colubris.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Roland Dreier wrote:
| Hi Martin,
<snip>
| Anyway, to summarize my findings on cvs2svn.py performance:
|
| 1. Use tmpfs for a working directory to avoid disk IO.
| 2. Use a LD_PRELOAD-ed library to get rid of useless sleeps in the
| BSD db library.
| 3. Increase the db cache size to avoid shuffling blocks in and out of
| cache.
| 4. It may be worth changing the pass 2 algorithm to increase
| performance.
|
| I'd be very interested to hear any reactions to these ideas.
|
| Best,
| Roland
|
Wow! I knew I was not alone :-)
Thanks, I will try that out... on another machine... I have only 256M on my workstation, running X
and Mozilla... I have a 1G machine lying around somewhere... :-)
I'll repost with my progress.
- --
"Windows might take you from 0 to 60 faster, but to go to 100 you need Unix."
==============
Martin Gadbois
S/W Developper
Colubris Networks Inc.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFAMMwD9Y3/iTTCEDkRArMRAJ9+YSMyi3e3JM+CM9Nf2FFa23D/RACfcGUh
9uE1RtoEKW1LJb15brdPPBA=
=v5ca
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up cvs2svn (was Re: cvs2svn takes very long time to
execute (days!))
Posted by Martin Gadbois <ma...@colubris.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Roland Dreier wrote:
| Anyway, to summarize my findings on cvs2svn.py performance:
|
| 1. Use tmpfs for a working directory to avoid disk IO.
| 2. Use a LD_PRELOAD-ed library to get rid of useless sleeps in the
| BSD db library.
| 3. Increase the db cache size to avoid shuffling blocks in and out of
| cache.
| 4. It may be worth changing the pass 2 algorithm to increase
| performance.
|
| I'd be very interested to hear any reactions to these ideas.
|
| Best,
| Roland
|
Might I suggest to try as well Psyco:
http://psyco.sourceforge.net/
Since, as you put in your email, that the script spends most of its time handling strings, Psyco
should be able to increase the performance.
I will try it out (the 1G RAM machine here is busy) on my small workstation as soon as my
compilation of OpenOffice is done... :-)
- --
"Windows might take you from 0 to 60 faster, but to go to 100 you need Unix."
==============
Martin Gadbois
S/W Developper
Colubris Networks Inc.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFAMPqe9Y3/iTTCEDkRAvt3AKCw4aYseGqYe2ED5HCctv4+FzLtMgCgwzBM
pnPZXbaja6GpLnKndX3CmNg=
=wX3l
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org