svnbench charts interpretation

[Neels Hofmeyr <ne...@elego.de>]

Hi all,

I think it's about time for another quick review of the current charts
for libsvn_wc performance. The current charts compare trunk to 1.7.0.
If you see a green bar, trunk was measured to be faster than 1.7.0.
The last dozen test runs are always shown at
http://svn-qavm.apache.org/charts/
and updated each monday night with a new test run.

*commit*: Looking at the overall totals, right at the top of
http://svn-qavm.apache.org/charts/compare_1.7.0_trunk@last12.svg ,
I notice performance dips in r1408153, r1418963 and r1426845. Looking
for a cause, all three lead back to 'svn commit': the commit
performance dips in all three are the only ones significant enough on
the seconds side (looking on the right side, we see an average of +2.99,
+.83 and +.74 seconds per commit, respectively). Furthermore, these
dips mostly come from the evenly spread WC test (5x5) and a little bit
from the very deep WC (100x1). Note: the +2.99, +.83 and +.74 seconds
averages above are the averages over all three WC topologies tested.
Isolating the evenly spread WC timings (5x5), they really are +9.11,
+2.45 and +2.14 seconds for each commit, so it's a more dramatic loss
of speed than the overall totals suggest. To see for yourself:
http://svn-qavm.apache.org/charts/compare_1.7.0_trunk@last12,5x5.svg
http://svn-qavm.apache.org/charts/compare_1.7.0_trunk@last12,1x100.svg
http://svn-qavm.apache.org/charts/compare_1.7.0_trunk@last12,100x1.svg

But at the end of the day, all these performance dips were measured
on only one monday. Either they were freak measurements or the code
responsible for the perf loss was improved within the same week.

Back to the overall averages,
http://svn-qavm.apache.org/charts/compare_1.7.0_trunk@last12.svg

*svn copy* seems to have wild fluctuations. On second glimpse, though,
you may notice that 'copy' takes only a quarter of a second on average,
and the fluctuation is within 0.1 seconds -- probably the OS.

*svn info* is consistently faster than in 1.7.0. That's nice. You
may think it is 5 seconds faster on average, but looking at the other
charts, you can see that the very flat (1x100) and very deep (100x1)
WCs are roughly the same as 1.7.0, while the evenly spread WC of 5x5
becomes almost 15 seconds and almost 90% faster. Wow. (These 15
seconds are averaged down with the other WC topologies and show in the
overall timings as 5 seconds. Don't let the statistics fool ya.)

*checkout* is consistently not-quite-but-(optimistically)-almost-twice
as fast as 1.7.0, but only the evenly spread WC of 5x5 accounts for a
gain of more than a second per checkout. In the 1x100 and 100x1 cases,
checkout is anyway done within 0.2 seconds, so half of that is not
felt by the user, really. Also, most commit waiting time will usually
come from the network, not measured here.

*merge* also is pretty consistently faster by a quarter to a third,
again taking the longest time (and consequently gaining the most speed)
in the 5x5 case. I'd say "woot!" if I didn't know that most of the
merge slowness people usually complain about is probably coming from the
network layers and not from libsvn_wc anyway. So just a hopeful: "nice!"

I'd have more to say if the right side of charts, showing change
in seconds, had larger numbers. Most 'svn' commands are done in the
blink of an eye, and: they still are. So that's good.

All in all we can say that libsvn_wc takes roughly a 1/3rd less time
to do its job than in 1.7.0. Most speed gain, by ~1/3rd, is seen in
evenly spread WCs (5x5), while very deep WCs are faster by about 1/5th,
and very flat WCs only get faster by less than an 1/8th. And again I
must emphasize that these tests deliberately try to neutralize the
network layer to measure only the libsvn_wc performance.

Thank you for your attention. :)
~Neels

Re: svnbench charts interpretation

["C. Michael Pilato" <cm...@collab.net>]

On 01/21/2013 02:13 PM, C. Michael Pilato wrote:
> Neels, thanks for the write-up.  I wonder... should we dump this information
> into a Wiki page?

Sorry -- I wasn't clear in the least.  I mean, should we have a wiki page
about how to interpret the svnbench charts in general.  (Obviously, specific
details about specific runs thereof don't make sense for a wiki page.)

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Enterprise Cloud Development

Re: svnbench charts interpretation

["C. Michael Pilato" <cm...@collab.net>]

On 01/21/2013 02:13 PM, C. Michael Pilato wrote:
> Neels, thanks for the write-up.  I wonder... should we dump this information
> into a Wiki page?

Sorry -- I wasn't clear in the last.  I mean, should we have a wiki page
about how to interpret the svnbench charts in general.  (Obviously, specific
details about specific runs thereof don't make sense for a wiki page.)

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Enterprise Cloud Development

Re: svnbench charts interpretation

["C. Michael Pilato" <cm...@collab.net>]

Neels, thanks for the write-up.  I wonder... should we dump this information
into a Wiki page?

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Enterprise Cloud Development

RE: svnbench charts interpretation

[Bert Huijben <be...@qqmail.nl>]

> -----Original Message-----
> From: Neels Hofmeyr [mailto:neels@elego.de]
> Sent: maandag 21 januari 2013 17:19
> To: dev@subversion.apache.org
> Subject: svnbench charts interpretation

> *merge* also is pretty consistently faster by a quarter to a third,
> again taking the longest time (and consequently gaining the most speed)
> in the 5x5 case. I'd say "woot!" if I didn't know that most of the
> merge slowness people usually complain about is probably coming from the
> network layers and not from libsvn_wc anyway. So just a hopeful: "nice!"

The amount of network IO is also reduced for 1.8.
We used to retrieve top much information for every not changed directory,
which we now identify correctly as not relevant.

But this is mostly in the 'diff' part of the merge, not the 'merge' code
itself.

	Bert