You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Matt Blais <mb...@yummage.com> on 2003/08/12 02:12:45 UTC
Concerned about Subversion's speed
Some rough measurements of svn's speed:
On a lightly-loaded WinXP-Pro SP1 machine (P4 2.4GHz, 1GB RAM):
In a directory containing 330 files under ra-local version control, plus
about 700 other files, and no subdirs under version control:
'svn st' 24 seconds
'svn st -uv' 28 seconds
'svn up' < 1 second (no updates needed)
This is heavily CPU-bound: my hard disk light barely flickered at all.
I have also tried using TortoiseSVN with this dir, which took between 45
to 100 seconds to draw (depending on what columns are displayed, sorting
options, etc.) That's just plain not usable!
I'm assuming --hoping-- that svn is presently doing some highly
un-optimized DB access in order to be this slow.
I can hear you saying, "Why in God's name do you have over 1000 files in
a single dir?" And the answer is, of course, that I didn't design this
project, but I do work on it ;-)
So, can I count on these numbers being pared down by at least a factor
of 10 in the near future (i.e., by release-1.0)? Or should I just give
up on ever using subversion on projects that might require directories
of this size?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Steve Williams <st...@kromestudios.com>.
Try it on a 16,000+ files in over 120 folders totalling over 2GB. Then do
an add and commit on all these files over HTTP. Takes about two hours.
I didn't design the project either. :)
Sly
> I can hear you saying, "Why in God's name do you have over 1000 files
> in a single dir?" And the answer is, of course, that I didn't design
> this project, but I do work on it ;-)
>
> So, can I count on these numbers being pared down by at least a factor
> of 10 in the near future (i.e., by release-1.0)? Or should I just
> give up on ever using subversion on projects that might require
> directories of this size?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Branko Čibej <br...@xbc.nu>.
cmpilato@collab.net wrote:
>Philip Martin <ph...@codematters.co.uk> writes:
>
>
>
>>"Matt Blais" <mb...@yummage.com> writes:
>>
>>
>>
>>>OK, this is better: on my newly-checked-out dir of 330 files (none
>>>modified), 'svn st' now only takes about 2.5 seconds. And 'svn st -v'
>>>only takes 4 seconds. That, I can live with.
>>>
>>>
>>It seems that all your operations are far slower than I would expect,
>>330 files over ra_local in not very big: over 2 minutes for checkout
>>and over 2 seconds for status is slow. That's several times what I
>>would expect based on my use of Subversion on Linux. Is Subversion
>>really that slow on Windows? Is it all CPU, or is something blocking?
>>What is the code doing during all that extra time?
>>
>>
>
>It really is slow on Windows. I dunno if its OS, the way APR
>interacts with the OS, Subversion, or what. But performance is often
>noticeably worse on my 2.4 Ghz Windows box versus my 800 Mhz Linux
>laptop.
>
Yep. It's the Windows I/O subsystem and all the crap it has to carry
around. It's designed for fast throughput, but add ACL processing and
change-notification hooks etc. etc., and it bogs down.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
>For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
>
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by cm...@collab.net.
Philip Martin <ph...@codematters.co.uk> writes:
> "Matt Blais" <mb...@yummage.com> writes:
>
> > OK, this is better: on my newly-checked-out dir of 330 files (none
> > modified), 'svn st' now only takes about 2.5 seconds. And 'svn st -v'
> > only takes 4 seconds. That, I can live with.
>
> It seems that all your operations are far slower than I would expect,
> 330 files over ra_local in not very big: over 2 minutes for checkout
> and over 2 seconds for status is slow. That's several times what I
> would expect based on my use of Subversion on Linux. Is Subversion
> really that slow on Windows? Is it all CPU, or is something blocking?
> What is the code doing during all that extra time?
It really is slow on Windows. I dunno if its OS, the way APR
interacts with the OS, Subversion, or what. But performance is often
noticeably worse on my 2.4 Ghz Windows box versus my 800 Mhz Linux
laptop.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Philip Martin <ph...@codematters.co.uk>.
"Matt Blais" <mb...@yummage.com> writes:
> OK, this is better: on my newly-checked-out dir of 330 files (none
> modified), 'svn st' now only takes about 2.5 seconds. And 'svn st -v'
> only takes 4 seconds. That, I can live with.
It seems that all your operations are far slower than I would expect,
330 files over ra_local in not very big: over 2 minutes for checkout
and over 2 seconds for status is slow. That's several times what I
would expect based on my use of Subversion on Linux. Is Subversion
really that slow on Windows? Is it all CPU, or is something blocking?
What is the code doing during all that extra time?
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Ben Collins-Sussman <su...@collab.net>.
"Matt Blais" <mb...@yummage.com> writes:
> OK, this is better: on my newly-checked-out dir of 330 files (none
> modified), 'svn st' now only takes about 2.5 seconds. And 'svn st -v'
> only takes 4 seconds. That, I can live with.
>
> After touching about half the files, 'svn st' takes 14 seconds. So that
> does seem to be the issue.
>
> This is good news. However, I would expect that after 'svn up', the
> 'status' processing time should go back down to the same as for the
> freshly-checked-out dir -- but it does NOT: it stays high. I should
> think that could be fixed...
Definitely, it could certainly be fixed. When 'svn up' crawls the
working copy, and it sees that a file is *not* changed, yet has a
different timestamp than the one in the entries file, it could tweak
the entries file.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Philip Martin <ph...@codematters.co.uk>.
kfogel@collab.net writes:
>> This is good news. However, I would expect that after 'svn up', the
>> 'status' processing time should go back down to the same as for the
>> freshly-checked-out dir -- but it does NOT: it stays high. I should
>> think that could be fixed...
>
> You know, I think this might be
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=791
I have a patch that does something like that.
It's not been committed because it had a few kinks. One problem is how
to trigger the repair process. To modify the working copy a write
lock is required, so 'svn status' can't really do it. Oddly 'svn up'
didn't appear to trigger it either, only things like 'svn commit' and
'svn revert' would effect the repair. Also, I think the user should
be able to repair the working copy without otherwise modifying it,
which means that even if 'svn up' did the job, some harmless command
would need to do it as well. I experimented with adding some code to
'svn cleanup' to do the job.
The prop-time was another problem, my patch repaired that as well, but
I wasn't sure what should happen. While the text-time can get out of
date due to perfectly "legitimate" user actions, that doesn't apply to
prop-time since the properties can only be modified by Subversion. I
wasn't sure if silently repairing prop-time was a good idea.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by kf...@collab.net.
"Matt Blais" <mb...@yummage.com> writes:
> OK, this is better: on my newly-checked-out dir of 330 files (none
> modified), 'svn st' now only takes about 2.5 seconds. And 'svn st -v'
> only takes 4 seconds. That, I can live with.
>
> After touching about half the files, 'svn st' takes 14 seconds. So that
> does seem to be the issue.
>
> This is good news. However, I would expect that after 'svn up', the
> 'status' processing time should go back down to the same as for the
> freshly-checked-out dir -- but it does NOT: it stays high. I should
> think that could be fixed...
You know, I think this might be
http://subversion.tigris.org/issues/show_bug.cgi?id=791
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Concerned about Subversion's speed
Posted by Matt Blais <mb...@yummage.com>.
OK, this is better: on my newly-checked-out dir of 330 files (none
modified), 'svn st' now only takes about 2.5 seconds. And 'svn st -v'
only takes 4 seconds. That, I can live with.
After touching about half the files, 'svn st' takes 14 seconds. So that
does seem to be the issue.
This is good news. However, I would expect that after 'svn up', the
'status' processing time should go back down to the same as for the
freshly-checked-out dir -- but it does NOT: it stays high. I should
think that could be fixed...
> -----Original Message-----
> From: sussman@collab.net [mailto:sussman@collab.net]
> Sent: Tuesday, August 12, 2003 1:25 PM
> To: Matt Blais
> Cc: 'Philip Martin'; dev@subversion.tigris.org
> Subject: Re: Concerned about Subversion's speed
>
>
> "Matt Blais" <mb...@yummage.com> writes:
>
> > The modification timestamps on *ALL* the checked-out files are
> > different from what's in the repos because they are ALL set to the
> > time when the files were checked out.
>
> Yes, but then that 'now' timestamp is stored in each file's
> <entry> within .svn/entries. It's a quick way for 'svn st' (and other
> commands) to know that a file has *not* changed. CVS does
> the same trick.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Ben Collins-Sussman <su...@collab.net>.
"Matt Blais" <mb...@yummage.com> writes:
> The modification timestamps on *ALL* the checked-out files are different
> from what's in the repos because they are ALL set to the time when the
> files were checked out.
Yes, but then that 'now' timestamp is stored in each file's <entry>
within .svn/entries. It's a quick way for 'svn st' (and other
commands) to know that a file has *not* changed. CVS does the same trick.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Concerned about Subversion's speed
Posted by Matt Blais <mb...@yummage.com>.
Philip Martin [mailto:pm@localhost] wrote:
> I find it hard to believe that the OS is responsible
> for the huge difference, and since 'svn st' doesn't access
> the repository at all, I suspect the problem is something to
> do wth your particular working copy.
I just checked out a new wc from the repos. Checking out these 330
files (5.2MB) took 2 minutes and 20 seconds with svn version 0.26.0
(r6550) compiled Aug 1 2003, 07:35:50
> Unless you do some profiling or measuring it may prove
> difficult to identify the problem. Taking a wild guess, is
> it possible that something has changed the last-modified
> timestamps of the versioned files? Do the last-modified
> times of the versioned files match the text-time values
> stored in the .svn/entries file?
The modification timestamps on *ALL* the checked-out files are different
from what's in the repos because they are ALL set to the time when the
files were checked out.
Does that affect the time to process the 'status' command? And should
'checkout' be resetting all the timestamps to now?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by Philip Martin <ph...@codematters.co.uk>.
"Matt Blais" <mb...@yummage.com> writes:
> On a lightly-loaded WinXP-Pro SP1 machine (P4 2.4GHz, 1GB RAM):
>
> In a directory containing 330 files under ra-local version control, plus
> about 700 other files, and no subdirs under version control:
>
> 'svn st' 24 seconds
> 'svn st -uv' 28 seconds
> 'svn up' < 1 second (no updates needed)
I run Linux on considerably less powerful hardware than your machine,
and running 'svn st' in a directory with 330 versioned files and 700
unversioned files takes less than a second. I find it hard to believe
that the OS is responsible for the huge difference, and since 'svn st'
doesn't access the repository at all, I suspect the problem is
something to do wth your particular working copy.
Unless you do some profiling or measuring it may prove difficult to
identify the problem. Taking a wild guess, is it possible that
something has changed the last-modified timestamps of the versioned
files? Do the last-modified times of the versioned files match the
text-time values stored in the .svn/entries file?
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by kf...@collab.net.
"Matt Blais" <mb...@yummage.com> writes:
> It might take me a little while to set up this test, as I do not have
> any local CVS repositories on my machine, and I'm no CVS expert either
> (but I will try).
Thank you [in advance].
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Concerned about Subversion's speed
Posted by Matt Blais <mb...@yummage.com>.
> How do these numbers compare with similar operations in CVS
> (say, run 'cvs status' on the same tree in CVS)?
It might take me a little while to set up this test, as I do not have
any local CVS repositories on my machine, and I'm no CVS expert either
(but I will try).
--Matt
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Concerned about Subversion's speed
Posted by kf...@collab.net.
Sander Striker's organizing some scalability test cases right now; I'm
betting this scenario is going to make it onto the list :-).
"Matt Blais" <mb...@yummage.com> writes:
> Some rough measurements of svn's speed:
>
> On a lightly-loaded WinXP-Pro SP1 machine (P4 2.4GHz, 1GB RAM):
>
> In a directory containing 330 files under ra-local version control, plus
> about 700 other files, and no subdirs under version control:
>
> 'svn st' 24 seconds
> 'svn st -uv' 28 seconds
> 'svn up' < 1 second (no updates needed)
>
> This is heavily CPU-bound: my hard disk light barely flickered at all.
How do these numbers compare with similar operations in CVS (say, run
'cvs status' on the same tree in CVS)?
> So, can I count on these numbers being pared down by at least a factor
> of 10 in the near future (i.e., by release-1.0)? Or should I just give
> up on ever using subversion on projects that might require directories
> of this size?
We're trying to identify and fix scalability problems before 1.0. How
we prioritize them will depend partly on the total list, and partly on
how it compares with CVS.
Thanks for the report,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org