Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Johan Corveleyn <jc...@gmail.com>]

On Thu, Apr 14, 2011 at 3:33 PM, Neels Hofmeyr <ne...@elego.de> wrote:
> Hi all,
>
> yesterday's and today's benchmarks look particularly good (listed
> below). I haven't paid attention to commits, but it seems some pretty
> good perf commits have happened recently (except for 'svn delete').


> All in all, once we clarified why 'svn delete' is as slow (and possibly
> choose not to worry about it), I'd suggest that trunk's performance is
> release worthy when compared to 1.6.x.

Hi Neels,

Great work, and good to see those numbers. But ... hold on a minute
before jumping to conclusions. IIUC you only tested one type of
platform: *nix with local disk (on a (powerful) VM, and on a (slightly
less powerful) "real machine"). It gives a good indication, but it's
definitely not enough to declare trunk release worthy
performance-wise.

For all we know there could be a massive regression on Windows or
MacOS, or on NTFS specifically (like the locking in WC-1, which was
orders of magnitude slower on Windows/NTFS than on Linux), or, as
already hinted a few times by Philip, on network-mounted filesystems:
NFS, CIFS, ...

So, to get a better picture, your test-suite should be run on a
variety of platforms (others can help, I guess, like with Mark's
perf-tests). Can your test-suite be run "as is" by others on Windows
for example? Maybe you could already run it yourself on an NFS-mounted
volume?

I think we need to at least test [ *nix | Windows | MacOS ] x [ local
disk | remote disk ] (ok maybe *nix is too coarse-grained, as is local
disk etc..., but you get the picture).

Cheers,
-- 
Johan

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Johan Corveleyn <jc...@gmail.com>]

On Fri, Apr 15, 2011 at 10:22 AM, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
>> Sent: donderdag 14 april 2011 23:23
>> To: Neels Hofmeyr
>> Cc: dev
>> Subject: Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28
>>
>> On Thu, Apr 14, 2011 at 3:33 PM, Neels Hofmeyr <ne...@elego.de> wrote:
>> > Hi all,
>> >
>> > yesterday's and today's benchmarks look particularly good (listed
>> > below). I haven't paid attention to commits, but it seems some pretty
>> > good perf commits have happened recently (except for 'svn delete').
>>
>>
>> > All in all, once we clarified why 'svn delete' is as slow (and possibly
>> > choose not to worry about it), I'd suggest that trunk's performance is
>> > release worthy when compared to 1.6.x.
>>
>> Hi Neels,
>>
>> Great work, and good to see those numbers. But ... hold on a minute
>> before jumping to conclusions. IIUC you only tested one type of
>> platform: *nix with local disk (on a (powerful) VM, and on a (slightly
>> less powerful) "real machine"). It gives a good indication, but it's
>> definitely not enough to declare trunk release worthy
>> performance-wise.
>>
>> For all we know there could be a massive regression on Windows or
>> MacOS, or on NTFS specifically (like the locking in WC-1, which was
>> orders of magnitude slower on Windows/NTFS than on Linux), or, as
>> already hinted a few times by Philip, on network-mounted filesystems:
>> NFS, CIFS, ...
>
> Most of my recent performance improvements were the result of profiling
> Subversion on Windows 7 / NTFS (ramdrive and harddisk scenarios). Looking at
> the results a I don't expect a real performance regression here:)

Great

> But please provide more numbers. We can't run on all configurations
> ourselves...

I have (with Mark's benchmarks), and I will continue to do so (was
thinking of running Mark's benchmarks with a CIFS-mounted working
copy). If Neels' benchmarks also work on Windows, I'll try running
them too. But honestly, I also don't have much tuits to spare
currently, so don't expect too much on the short term.

OTOH: maybe you're speaking to the mailinglist-public in general, not
just to me :).

> The move to a single database reduced the number of differences between
> operating systems.
> On Windows just removing that lock problem you mentioned makes such a huge
> difference that comparing performance in extremely large working copy
> scenarios won't produce usable results. (1.7 is orders faster when you would
> have seen that problem before)
>
>
> Other changes in 1.7 should improve performance more in real world cases,
> than you would see in our test runs.
> With 1.6 (and earlier) updating a working copy with tree conflicts most
> likely created a mixed revision working copy, which would slow down the next
> update (while dropping the result). With 1.7 the update will just complete
> so the next update (after you resolved the conflicts) can use the most
> efficient update mode.
>
>
> Another performance aspect that is very hard to test is a hot-cache vs
> cold-cache scenario. With 1.7 your disk cache will usually just proactively
> cache the entire SQLite database whenever you access it, while 1.6 had to
> gather all the entries and properties files.  I see a huge (positive)
> difference here, for large working copies.

That all sounds great.

To be honest: I don't expect much platform-specific performance
regressions either (at least not with local disks), because there is
so much testing/developing/profiling going on on Windows now. But I
was more speaking out of principle: don't assume it's performant on
all platforms just because it's performant on one single flavor.
That's exactly the attitude that makes it possible to miss huge
platform-specific performance problems like with the WC-1 locking
code.

Luckily, there seems to be a good amount of attention to usage on
Windows these days, and devs working/focusing also on Windows, so I
don't expect any disasters. But still, the principle :)...

(note: I don't want to start any religious wars. Windows isn't my
favorite OS either, but I (have to) use it at work, and there happen
to be tons of svn users running Windows, so ...).

> Using NFS and CIFS for working copies was never a primary use case, but all
> the extreme cases reported earlier have been resolved and I expect that most
> of the recent performance improvements had even more effect on these network
> filesystems than on local disks.
>
> But in this area I would love to see some real numbers.
> (I would assume that network disks are usually in the same category as the
> huge-working copy+cold-cache local scenarios as the local disk cache can't
> cache all data)

Yes. I understand NFS/CIFS working copies are not the primary use
case, but I do think they are being used more and more often. In a lot
of corporate environments, people have homedirs on a remote fileserver
(NAS/SAN/whatever/... with snapshots, backups, failover, ...). Be it
on *nix, with NFS mounts, or on Windows, with CIFS mounts. All the
important data is supposed to be put on those volumes, so that the
data is safe, and it can be mounted from anywhere ... So I just see
this happening more and more.

Theoretically, I can see that with WC-NG things should become faster
in those situations (reducing the lots-of-small-files-I/O of the lock
files, entries file, ...). So if the extreme cases that were reported
earlier have been fixed, it should be ok. I agree it would be good to
have some more numbers here ...

Cheers,
-- 
Johan

RE: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Bert Huijben <be...@qqmail.nl>]

> -----Original Message-----
> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
> Sent: donderdag 14 april 2011 23:23
> To: Neels Hofmeyr
> Cc: dev
> Subject: Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28
> 
> On Thu, Apr 14, 2011 at 3:33 PM, Neels Hofmeyr <ne...@elego.de> wrote:
> > Hi all,
> >
> > yesterday's and today's benchmarks look particularly good (listed
> > below). I haven't paid attention to commits, but it seems some pretty
> > good perf commits have happened recently (except for 'svn delete').
> 
> 
> > All in all, once we clarified why 'svn delete' is as slow (and possibly
> > choose not to worry about it), I'd suggest that trunk's performance is
> > release worthy when compared to 1.6.x.
> 
> Hi Neels,
> 
> Great work, and good to see those numbers. But ... hold on a minute
> before jumping to conclusions. IIUC you only tested one type of
> platform: *nix with local disk (on a (powerful) VM, and on a (slightly
> less powerful) "real machine"). It gives a good indication, but it's
> definitely not enough to declare trunk release worthy
> performance-wise.
> 
> For all we know there could be a massive regression on Windows or
> MacOS, or on NTFS specifically (like the locking in WC-1, which was
> orders of magnitude slower on Windows/NTFS than on Linux), or, as
> already hinted a few times by Philip, on network-mounted filesystems:
> NFS, CIFS, ...

Most of my recent performance improvements were the result of profiling
Subversion on Windows 7 / NTFS (ramdrive and harddisk scenarios). Looking at
the results a I don't expect a real performance regression here:)

But please provide more numbers. We can't run on all configurations
ourselves...


The move to a single database reduced the number of differences between
operating systems.
On Windows just removing that lock problem you mentioned makes such a huge
difference that comparing performance in extremely large working copy
scenarios won't produce usable results. (1.7 is orders faster when you would
have seen that problem before)


Other changes in 1.7 should improve performance more in real world cases,
than you would see in our test runs. 
With 1.6 (and earlier) updating a working copy with tree conflicts most
likely created a mixed revision working copy, which would slow down the next
update (while dropping the result). With 1.7 the update will just complete
so the next update (after you resolved the conflicts) can use the most
efficient update mode.


Another performance aspect that is very hard to test is a hot-cache vs
cold-cache scenario. With 1.7 your disk cache will usually just proactively
cache the entire SQLite database whenever you access it, while 1.6 had to
gather all the entries and properties files.  I see a huge (positive)
difference here, for large working copies.


Using NFS and CIFS for working copies was never a primary use case, but all
the extreme cases reported earlier have been resolved and I expect that most
of the recent performance improvements had even more effect on these network
filesystems than on local disks.

But in this area I would love to see some real numbers.
(I would assume that network disks are usually in the same category as the
huge-working copy+cold-cache local scenarios as the local disk cache can't
cache all data)

> So, to get a better picture, your test-suite should be run on a
> variety of platforms (others can help, I guess, like with Mark's
> perf-tests). Can your test-suite be run "as is" by others on Windows
> for example? Maybe you could already run it yourself on an NFS-mounted
> volume?
> 
> I think we need to at least test [ *nix | Windows | MacOS ] x [ local
> disk | remote disk ] (ok maybe *nix is too coarse-grained, as is local
> disk etc..., but you get the picture).

	Bert

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Gavin Baumanis <ga...@thespidernet.com>]

Hi Neels,

I have a brand new (2 week old) MacBook Pro with 4 GB RAM.
I am more than happy to run your scripts if you like.

On the same machine I have 1.6.16 (and trunk r1092145) installed - so I can do a comparison run of both.

And if you think it is worthy of another comparison run, I have a 3 year old MacBook (non-Pro in flavour) that could also have the same runs.

Beau.


On 15/04/2011, at 9:25 AM, Neels Hofmeyr wrote:

> On Thu, 2011-04-14 at 23:22 +0200, Johan Corveleyn wrote:
>> Hi Neels,
>> 
>> Great work, and good to see those numbers. But ... hold on a minute
>> before jumping to conclusions. IIUC you only tested one type of
>> platform: *nix with local disk (on a (powerful) VM, and on a (slightly
>> less powerful) "real machine"). It gives a good indication, but it's
>> definitely not enough to declare trunk release worthy
>> performance-wise.
> 
> I'm aware of that and am usually very careful to place "I'd say" and
> "IMHO" all around my humble "conclusions" ;)
> 
> As-is, my scripts definitely won't run on Windows. The core is python,
> but there's a thin layer of bash around it that calls the python with
> the desired settings and filenames, so the bash stuff needs to be looked
> at and translated to BAT ...or python probably.
> It could work on OSX without changes, AFAIK.
> Frankly, I won't personally go into testing on Windows or a WC on an NFS
> mount, because, *IMHO*, both Win and WC-on-NFS are misguided, and I
> personally can't be bothered. This is where people using those things
> and whose jobs depend on svn 1.7 doing well on them should get involved,
> as far as I'm concerned.
> 
> Johan is right to highlight the limitations. And I can add a large
> portion of limitations inherent to benchmarks like these (hardware
> particulars, time measurement inaccuracies, simulation validity,
> statistics, interpretation, selective perception, chance,... ). Only the
> actual release 1.7.0 will bring all the real issues to the surface.
> 
> And yet, the conclusion I find from my past and present test runs is:
> the performance I have seen in my setup a while back has definitely been
> a show-stopper. And now I don't see those particular show-stoppers
> anymore. So, yes, I should probably be more clear instead of saying
> "release-worthy" -- I meant it rather negatively defined, as in: my
> personal perf show-stoppers have vanished. (Thanks for poking)
> 
> So, does anyone do Windows? Do you speak bash, python and BAT? Want to
> know about perf on your setup? I'll gladly send you my scripts and
> explain anything you want to know about it.
> 
> Oh, and about the time factors getting much better on the VM -- that's
> not just because the VM is faster than my home machines, since then
> 1.6.x would also be faster on the VM, resulting in similar time factors
> as on other hardware. I don't know in detail why we're seeing this, but
> it must be something like, maybe SQlite has some magic that enhances
> performance when -- I don't know -- the CPU bus is very wide, or disk
> cache is fast, or something like that, boosting fast hardware even more.
> It must be something that svn 1.6 can't benefit from.
> 
> Enough talk :)
> 
> ~Neels
> 
> 
> 
>> 
>> For all we know there could be a massive regression on Windows or
>> MacOS, or on NTFS specifically (like the locking in WC-1, which was
>> orders of magnitude slower on Windows/NTFS than on Linux), or, as
>> already hinted a few times by Philip, on network-mounted filesystems:
>> NFS, CIFS, ...
>> 
>> So, to get a better picture, your test-suite should be run on a
>> variety of platforms (others can help, I guess, like with Mark's
>> perf-tests). Can your test-suite be run "as is" by others on Windows
>> for example? Maybe you could already run it yourself on an NFS-mounted
>> volume?
>> 
>> I think we need to at least test [ *nix | Windows | MacOS ] x [ local
>> disk | remote disk ] (ok maybe *nix is too coarse-grained, as is local
>> disk etc..., but you get the picture).
>> 
>> Cheers,
> 
> 

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Greg Stein <gs...@gmail.com>]

On Mon, Apr 18, 2011 at 10:50, Neels Hofmeyr <ne...@elego.de> wrote:
> On Fri, 2011-04-15 at 03:18 -0400, Greg Stein wrote:
>> On Fri, Apr 15, 2011 at 00:13, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>> > Just add them to public version control?
>>
>> Already done. See tools/dev/benchmarks/suite1/
>>
>> Looking at the scripts (along with a bit of work that I did), it seems
>> quite easy to incorporate the 'run' script directly into
>> 'benchmark.py', and (thus) make it more portable to Windows devs.
>
> Thanks, Greg, for your tweaks! I've just merged mine and am now using
> tools/dev/benchmarks/suite1/ instead of the elego repos.

Sweet.

> BTW, what's in the repos in suite1/ is now *exactly* what is run on the
> VM, no tweaks, just 'svn co'. (because I've added two ifs conditional to
> my username, for the svn-bins path and the email address...)
> Oh, but the scripts on the VM are not automatically updated, so if you
> tweaked and want to see the changes in the weekly mail, do ping me.

Will do. I'm going to fold much of 'run' into the .py script.
Eventually, 'run' should just be a simple way to type the N parameters
necessary to the benchmark script, which will then make it cake to
translate to a .BAT script for Windows.

> The benchmark will now run once a week on Monday morning and send
> results directly to dev@. Any objections?

Sounds great. If it starts to piss people off, then we can always
disable it. We could send it to notifications@, as an alternative.

Cheers,
-g

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Neels Hofmeyr <ne...@elego.de>]

On Tue, 2011-05-10 at 11:41 +0200, Neels Hofmeyr wrote:
> On Mon, 2011-04-18 at 16:50 +0200, Neels Hofmeyr wrote:
> > The benchmark will now run once a week on Monday morning and send
> > results directly to dev@. Any objections?
> 
> It was supposed to, but I just actively noted that I haven't seen any
> for quite a while. I'm now fixing stuff I broke in the scripts.

ts, crontab's shell didn't have the USER variable set, that was all. I
never knew.

~Neels

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Neels Hofmeyr <ne...@elego.de>]

On Mon, 2011-04-18 at 16:50 +0200, Neels Hofmeyr wrote:
> The benchmark will now run once a week on Monday morning and send
> results directly to dev@. Any objections?

It was supposed to, but I just actively noted that I haven't seen any
for quite a while. I'm now fixing stuff I broke in the scripts.

~Neels

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Hyrum K Wright <hy...@hyrumwright.org>]

On Mon, Apr 18, 2011 at 9:50 AM, Neels Hofmeyr <ne...@elego.de> wrote:
> On Fri, 2011-04-15 at 03:18 -0400, Greg Stein wrote:
>> On Fri, Apr 15, 2011 at 00:13, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>> > Just add them to public version control?
>>
>> Already done. See tools/dev/benchmarks/suite1/
>>
>> Looking at the scripts (along with a bit of work that I did), it seems
>> quite easy to incorporate the 'run' script directly into
>> 'benchmark.py', and (thus) make it more portable to Windows devs.
>
> Thanks, Greg, for your tweaks! I've just merged mine and am now using
> tools/dev/benchmarks/suite1/ instead of the elego repos.
>
> BTW, what's in the repos in suite1/ is now *exactly* what is run on the
> VM, no tweaks, just 'svn co'. (because I've added two ifs conditional to
> my username, for the svn-bins path and the email address...)
> Oh, but the scripts on the VM are not automatically updated, so if you
> tweaked and want to see the changes in the weekly mail, do ping me.
>
> The benchmark will now run once a week on Monday morning and send
> results directly to dev@. Any objections?

What would be *really* nifty is if there was some kind of database
somewhere which captured all this information (for both this VM as
well as others who are running the tests).  That way, somebody else
could do some analysis on it over time.
</wishful-thinking>

-Hyrum

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Neels Hofmeyr <ne...@elego.de>]

On Fri, 2011-04-15 at 03:18 -0400, Greg Stein wrote:
> On Fri, Apr 15, 2011 at 00:13, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> > Just add them to public version control?
> 
> Already done. See tools/dev/benchmarks/suite1/
> 
> Looking at the scripts (along with a bit of work that I did), it seems
> quite easy to incorporate the 'run' script directly into
> 'benchmark.py', and (thus) make it more portable to Windows devs.

Thanks, Greg, for your tweaks! I've just merged mine and am now using
tools/dev/benchmarks/suite1/ instead of the elego repos.

BTW, what's in the repos in suite1/ is now *exactly* what is run on the
VM, no tweaks, just 'svn co'. (because I've added two ifs conditional to
my username, for the svn-bins path and the email address...)
Oh, but the scripts on the VM are not automatically updated, so if you
tweaked and want to see the changes in the weekly mail, do ping me.

The benchmark will now run once a week on Monday morning and send
results directly to dev@. Any objections?

~Neels

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Greg Stein <gs...@gmail.com>]

On Fri, Apr 15, 2011 at 00:13, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> On Fri, 15 Apr 2011 01:25 +0200, "Neels Hofmeyr" <ne...@elego.de> wrote:
>> So, does anyone do Windows? Do you speak bash, python and BAT? Want to
>> know about perf on your setup? I'll gladly send you my scripts
>
> Just add them to public version control?

Already done. See tools/dev/benchmarks/suite1/

Looking at the scripts (along with a bit of work that I did), it seems
quite easy to incorporate the 'run' script directly into
'benchmark.py', and (thus) make it more portable to Windows devs.

Cheers,
-g

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Daniel Shahaf <d....@daniel.shahaf.name>]

On Fri, 15 Apr 2011 01:25 +0200, "Neels Hofmeyr" <ne...@elego.de> wrote:
> So, does anyone do Windows? Do you speak bash, python and BAT? Want to
> know about perf on your setup? I'll gladly send you my scripts 

Just add them to public version control?

> and explain anything you want to know about it.

Re: [svnbench] Revision: 1091976 compiled Apr 14 2011, 00:21:28

[Neels Hofmeyr <ne...@elego.de>]

On Thu, 2011-04-14 at 23:22 +0200, Johan Corveleyn wrote:
> Hi Neels,
> 
> Great work, and good to see those numbers. But ... hold on a minute
> before jumping to conclusions. IIUC you only tested one type of
> platform: *nix with local disk (on a (powerful) VM, and on a (slightly
> less powerful) "real machine"). It gives a good indication, but it's
> definitely not enough to declare trunk release worthy
> performance-wise.

I'm aware of that and am usually very careful to place "I'd say" and
"IMHO" all around my humble "conclusions" ;)

As-is, my scripts definitely won't run on Windows. The core is python,
but there's a thin layer of bash around it that calls the python with
the desired settings and filenames, so the bash stuff needs to be looked
at and translated to BAT ...or python probably.
It could work on OSX without changes, AFAIK.
Frankly, I won't personally go into testing on Windows or a WC on an NFS
mount, because, *IMHO*, both Win and WC-on-NFS are misguided, and I
personally can't be bothered. This is where people using those things
and whose jobs depend on svn 1.7 doing well on them should get involved,
as far as I'm concerned.

Johan is right to highlight the limitations. And I can add a large
portion of limitations inherent to benchmarks like these (hardware
particulars, time measurement inaccuracies, simulation validity,
statistics, interpretation, selective perception, chance,... ). Only the
actual release 1.7.0 will bring all the real issues to the surface.

And yet, the conclusion I find from my past and present test runs is:
the performance I have seen in my setup a while back has definitely been
a show-stopper. And now I don't see those particular show-stoppers
anymore. So, yes, I should probably be more clear instead of saying
"release-worthy" -- I meant it rather negatively defined, as in: my
personal perf show-stoppers have vanished. (Thanks for poking)

So, does anyone do Windows? Do you speak bash, python and BAT? Want to
know about perf on your setup? I'll gladly send you my scripts and
explain anything you want to know about it.

Oh, and about the time factors getting much better on the VM -- that's
not just because the VM is faster than my home machines, since then
1.6.x would also be faster on the VM, resulting in similar time factors
as on other hardware. I don't know in detail why we're seeing this, but
it must be something like, maybe SQlite has some magic that enhances
performance when -- I don't know -- the CPU bus is very wide, or disk
cache is fast, or something like that, boosting fast hardware even more.
It must be something that svn 1.6 can't benefit from.

Enough talk :)

~Neels



> 
> For all we know there could be a massive regression on Windows or
> MacOS, or on NTFS specifically (like the locking in WC-1, which was
> orders of magnitude slower on Windows/NTFS than on Linux), or, as
> already hinted a few times by Philip, on network-mounted filesystems:
> NFS, CIFS, ...
> 
> So, to get a better picture, your test-suite should be run on a
> variety of platforms (others can help, I guess, like with Mark's
> perf-tests). Can your test-suite be run "as is" by others on Windows
> for example? Maybe you could already run it yourself on an NFS-mounted
> volume?
> 
> I think we need to at least test [ *nix | Windows | MacOS ] x [ local
> disk | remote disk ] (ok maybe *nix is too coarse-grained, as is local
> disk etc..., but you get the picture).
> 
> Cheers,