You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Paul Koning <pk...@equallogic.com> on 2005/09/13 20:16:48 UTC
More performance
Related, perhaps, to the checkout performance problem...
I had a working directory for trunk. Decided to switch it to a
branch.
Did the "svn switch" command. That changed about 20 files (out of
about 71,000 total). It took 30 minutes, which is just a hair longer
than the checkout took.
Given that "switch" is supposed to be a shortcut, and there was so
little difference between trunk and branch, that's surprising.
paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: More performance
Posted by Paul Koning <pk...@equallogic.com>.
>>>>> "Ben" == Ben Collins-Sussman <su...@collab.net> writes:
Ben> On Sep 13, 2005, at 3:16 PM, Paul Koning wrote:
>> Related, perhaps, to the checkout performance problem...
>>
>> I had a working directory for trunk. Decided to switch it to a
>> branch.
>>
>> Did the "svn switch" command. That changed about 20 files (out of
>> about 71,000 total). It took 30 minutes, which is just a hair
>> longer than the checkout took.
>>
>> Given that "switch" is supposed to be a shortcut, and there was so
>> little difference between trunk and branch, that's surprising.
Ben> What this means is that 1% of the clock time was spent
Ben> installing new files, and 99% of the clock time was spent
Ben> rewriting the .svn/entries files for 71,000 files. That's a
Ben> whole lot of xml parsing and rewriting.
Ben> I expect it will take almost as long to run 'svn update', 'svn
Ben> status', or 'svn commit'. These are all situations where you're
Ben> telling the client to do a walk over a gigantic working copy.
It's clear that some of my troubles were from a flaky client.
I repeated the tests on a different system (also NetBSD). It behaves
far more sensibly.
These ones I ran with svnserve instead of Apache for repository
access. The 71k file repository (which CVS checked out in 10 minutes)
took 20 minutes for checkout. Switch from trunk to a branch (the one
I complained about earlier) took 8:54. And status (of the entire
working directory) took 3:53.
This is very encouraging. Thanks everyone for the help, and sorry for
bringing in red hardware herrings.
Just for grins, I'm going to do another switch, this time switching to
http as the method. We'll see if that makes a significant difference
in timing.
paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: More performance
Posted by Christopher Ness <ch...@nesser.org>.
On Thu, 2005-09-15 at 09:04 -0400, Paul Koning wrote:
> >>>>> "Christopher" == Christopher Ness <ch...@nesser.org> writes:
<snip />
> Christopher> Have you looked at the network usage of SVN vs. CVS?
> Christopher> Checkouts should obviously be about the same, but other
> Christopher> commands like update I would expect you to see a
> Christopher> difference.
>
> I was expecting that too, from the book. But then Ben commented that
> "svn switch" was slow because it had that big tree to traverse, and I
> should expect other commands to suffer likewise. Frankly, that
switch was an extreme example. Most commands do walk the entire tree in
the working copy which takes time but do not modify the accounting
information for each file. If your client machines are under powered
but on a fast network CVS might still be a better solution for you.
If you have people distributed around the world on slow networks though,
they will not be happy with your choice. SVN trades off bandwidth for
hard drive space and client CPU cycles.
> doesn't make much sense, because if it were an unavoidable consequence
> of the repository size, CVS should be just as slow, and it isn't.
SVN takes cycles on the clients machine to describe the minimal set of
changes to the server instead of having the server blast entire files
back to the client.
This takes time before even talking to the server. CVS - as I
understand it - just brute forces it's changes back to the client.
> Christopher> How many developers do you know of that work on a
> Christopher> project with > 70,000 files and > 1.5Gb of code? I
> Christopher> don't know of too many.
<snip />
> The repository is what it is because it contains all the pieces that
> go into our product -- which is a rather large embedded system. A
> piece of it is NetBSD, so that brings in the whole source tree for
> that, which is big already. (Consider that a large application like
> gcc is just one subtree underneath NetBSD.) And then there is a whole
> lot of other imported stuff, and all the code we wrote.
You were able to break your product down into components in the above
paragraph. Likely some developers work across component lines, but I'd
think the NetBSD guys work in that code base, the other imported stuff
and "all the code you wrote".
Working copies are cheap, you can have as many as you'd like. That is
instead of one holistic working copy of trunk or a branch.
Of course builds would be an exception that need a full working copy or
export.
> We're currently using CVS. My task is to find a replacement for CVS
> that works better. If Subversion does work better with our
> repository, then it's a leading candidate. If Subversion isn't
> capable of dealing effectively with repositories that size, then it is
> not a candidate. Or at the best it would be a candidate only if there
> isn't anything else, i.e., we have no choice but to spend the
> engineering effort to redesign our repository, and development
> procedures, and build scripts, etc...
>
> Yes, if we were to start over we probably should have constructed
> things differently. But we didn't, and changing is not cheap or
> simple.
Fair enough.
I simply am observing that you are abusing the tool and giving it a poor
performance report by not following best practices.
Perhaps you should talk to the KDE developers and see what they have to
say about subversion and orders of scaling.
Cheers,
Chris
--
PGP Public Key: http://www.nesser.org/pgp-key/
09:29:30 up 1:23, 2 users, load average: 0.10, 0.39, 0.81
Re: More performance
Posted by Paul Koning <pk...@equallogic.com>.
>>>>> "Christopher" == Christopher Ness <ch...@nesser.org> writes:
Christopher> On Wed, 2005-09-14 at 09:19 -0400, Paul Koning wrote:
Ben> I expect it will take almost as long to run 'svn update', 'svn
Ben> status', or 'svn commit'. These are all situations where you're
Ben> telling the client to do a walk over a gigantic working copy.
>> That's depressing. If true, it would mean that Subversion across
>> the board is 3x slower than CVS.
Christopher> That blanket statement is not true. What does diff run
Christopher> as with local modifications in both CVS and SVN?
Christopher> Have you looked at the network usage of SVN vs. CVS?
Christopher> Checkouts should obviously be about the same, but other
Christopher> commands like update I would expect you to see a
Christopher> difference.
I was expecting that too, from the book. But then Ben commented that
"svn switch" was slow because it had that big tree to traverse, and I
should expect other commands to suffer likewise. Frankly, that
doesn't make much sense, because if it were an unavoidable consequence
of the repository size, CVS should be just as slow, and it isn't.
Christopher> How many developers do you know of that work on a
Christopher> project with > 70,000 files and > 1.5Gb of code? I
Christopher> don't know of too many.
Christopher> Perhaps you should use a normal sized repository. For
Christopher> example lets look at subversion itself which is a
Christopher> non-trivial product:
That's nice, but Subversion is just a single application.
The repository is what it is because it contains all the pieces that
go into our product -- which is a rather large embedded system. A
piece of it is NetBSD, so that brings in the whole source tree for
that, which is big already. (Consider that a large application like
gcc is just one subtree underneath NetBSD.) And then there is a whole
lot of other imported stuff, and all the code we wrote.
We're currently using CVS. My task is to find a replacement for CVS
that works better. If Subversion does work better with our
repository, then it's a leading candidate. If Subversion isn't
capable of dealing effectively with repositories that size, then it is
not a candidate. Or at the best it would be a candidate only if there
isn't anything else, i.e., we have no choice but to spend the
engineering effort to redesign our repository, and development
procedures, and build scripts, etc...
Yes, if we were to start over we probably should have constructed
things differently. But we didn't, and changing is not cheap or
simple.
paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: More performance
Posted by Paul Koning <pk...@equallogic.com>.
>>>>> "Hari" == Hari Kodungallur <hk...@gmail.com> writes:
Hari> I would also like to add here the speed of branching. With a
Hari> project of this size, i would think there would be many
Hari> branches and tags. For a 70,000 file project the comparison
Hari> would be: less than a second for SVN vs. several minutes (if
Hari> not hours) for CVS
15-20 minutes for CVS. Yes indeed, and that is the main reason why we
want a better solution. I'm really hoping that SVN will offer
acceptable performance with the the regular development cycle commands
(status, diff, update, commit) -- better than CVS preferably, similar
to CVS worst case. And of course it has to be reliable, which at this
point remains to be seen. Given those requirements, then the vastly
better tag performance is a major win, since we do several of those
each day.
paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: More performance
Posted by Hari Kodungallur <hk...@gmail.com>.
On 9/14/05, Christopher Ness <ch...@nesser.org> wrote:
> On Wed, 2005-09-14 at 09:19 -0400, Paul Koning wrote:
> > Ben> I expect it will take almost as long to run 'svn update', 'svn
> > Ben> status', or 'svn commit'. These are all situations where you're
> > Ben> telling the client to do a walk over a gigantic working copy.
> >
> > That's depressing. If true, it would mean that Subversion across the
> > board is 3x slower than CVS.
>
> That blanket statement is not true. What does diff run as with local
> modifications in both CVS and SVN?
>
> Have you looked at the network usage of SVN vs. CVS? Checkouts should
> obviously be about the same, but other commands like update I would
> expect you to see a difference.
>
> How many developers do you know of that work on a project with > 70,000
> files and > 1.5Gb of code? I don't know of too many.
>
I would also like to add here the speed of branching. With a project
of this size, i would think there would be many branches and tags. For
a 70,000 file project the comparison would be: less than a second for
SVN vs. several minutes (if not hours) for CVS
--
rgds,
-Hari
SpikeSource Inc.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: More performance
Posted by Christopher Ness <ch...@nesser.org>.
On Wed, 2005-09-14 at 09:19 -0400, Paul Koning wrote:
> Ben> I expect it will take almost as long to run 'svn update', 'svn
> Ben> status', or 'svn commit'. These are all situations where you're
> Ben> telling the client to do a walk over a gigantic working copy.
>
> That's depressing. If true, it would mean that Subversion across the
> board is 3x slower than CVS.
That blanket statement is not true. What does diff run as with local
modifications in both CVS and SVN?
Have you looked at the network usage of SVN vs. CVS? Checkouts should
obviously be about the same, but other commands like update I would
expect you to see a difference.
How many developers do you know of that work on a project with > 70,000
files and > 1.5Gb of code? I don't know of too many.
Perhaps you should use a normal sized repository. For example lets look
at subversion itself which is a non-trivial product:
[nesscg@heidrun grep-subversion]$ svn info
Path: .
URL: http://svn.collab.net/repos/svn/trunk
<snip />
[nesscg@heidrun grep-subversion]$ svnversion
16132
[nesscg@heidrun grep-subversion]$ du -sh
69M .
Since this is a working copy, that means SIZE = REPORTED_SIZE/2 and get
~35M of source code. And there is a website in there too, as well as
documentation and tools to support SVN.
Perhaps your projects are as big as the tests you are running, but you
*should* always use the smallest possible subset as a working copy.
Cheers,
Chris
--
PGP Public Key: http://www.nesser.org/pgp-key/
23:02:38 up 2 days, 6:02, 2 users, load average: 0.21, 0.22, 0.34
Re: More performance
Posted by Paul Koning <pk...@equallogic.com>.
>>>>> "Ben" == Ben Collins-Sussman <su...@collab.net> writes:
Ben> On Sep 13, 2005, at 3:16 PM, Paul Koning wrote:
>> Related, perhaps, to the checkout performance problem...
>>
>> I had a working directory for trunk. Decided to switch it to a
>> branch.
>>
>> Did the "svn switch" command. That changed about 20 files (out of
>> about 71,000 total). It took 30 minutes, which is just a hair
>> longer than the checkout took.
>>
>> Given that "switch" is supposed to be a shortcut, and there was so
>> little difference between trunk and branch, that's surprising.
Ben> What this means is that 1% of the clock time was spent
Ben> installing new files, and 99% of the clock time was spent
Ben> rewriting the .svn/entries files for 71,000 files. That's a
Ben> whole lot of xml parsing and rewriting.
Ben> I expect it will take almost as long to run 'svn update', 'svn
Ben> status', or 'svn commit'. These are all situations where you're
Ben> telling the client to do a walk over a gigantic working copy.
That's depressing. If true, it would mean that Subversion across the
board is 3x slower than CVS.
It seems strange that a switch should take the same amount of time as
a checkout, when the checkout has to move across 1.6 GB of data, not
just update XML files. And it should just be one update per
directory, since the URL is common to the directory in the "entries"
file.
paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: More performance
Posted by Ben Collins-Sussman <su...@collab.net>.
On Sep 13, 2005, at 3:16 PM, Paul Koning wrote:
> Related, perhaps, to the checkout performance problem...
>
> I had a working directory for trunk. Decided to switch it to a
> branch.
>
> Did the "svn switch" command. That changed about 20 files (out of
> about 71,000 total). It took 30 minutes, which is just a hair longer
> than the checkout took.
>
> Given that "switch" is supposed to be a shortcut, and there was so
> little difference between trunk and branch, that's surprising.
What this means is that 1% of the clock time was spent installing new
files, and 99% of the clock time was spent rewriting the .svn/entries
files for 71,000 files. That's a whole lot of xml parsing and
rewriting.
I expect it will take almost as long to run 'svn update', 'svn
status', or 'svn commit'. These are all situations where you're
telling the client to do a walk over a gigantic working copy.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org