You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Matthew Bentham <mj...@artvps.com> on 2010/02/10 11:57:35 UTC
performance on cygwin
Hi,
The summary is that I'm motivated to help with work on the performance
of Windows subversion clients, because the speed of 'svn update' is not
great for us on large working copies, and we keep having to re-arrange
our source to work around that.
At work I've used Subversion happily for many years. There are
perennial discussions in the office about the performance of the windows
clients. Several times in the past we have had to carefully prune the
project, especially external modules imported into our code, in order to
keep acceptable performance of 'svn update'. The Linux clients have
always been fine.
I've been hearing rumours of performance improvements to Windows svn
clients with the 'wc-ng' work. A couple of weeks ago I decided to have
a go with subversion trunk to see what it would make of our repository.
I've compiled it, compared the performance with subversion 1.6 from
cygwin, and done some profiling with Intel VTune. The results follow, I
hope they are interesting.
Cygwin session, svn executable from cygwin. OS is Vista 64:
$ uname -a
CYGWIN_NT-6.0-WOW64 brahe 1.7.1(0.218/5/3) 2009-12-07 11:48 i686 Cygwin
$ svn --version
svn, version 1.6.9 (r901367)
compiled Jan 21 2010, 16:47:22
[...]
$ cd avir-cyg
$ svn info
Path: .
URL: svn://morley/repos/trunk/avir
Repository Root: svn://morley/repos
[...]
$ time svn update
At revision 36888.
real 0m19.865s
user 0m0.250s
sys 0m1.468s
$
This is roughly what we've come to expect. TortoiseSVN built with
subversion 1.6.9 has similar performance over an update that has no changes.
So I compiled subversion trunk, r907984:
$ cd ../avir-newsvn
$ /cygdrive/e/subversions/subversion/installdir/bin/svn.exe --version
svn, version 1.7.0 (dev build)
compiled Feb 8 2010, 16:21:13
[...]
$ /cygdrive/e/subversions/subversion/installdir/bin/svn.exe info
Path: .
URL: svn://morley/repos/trunk/avir
Repository Root: svn://morley/repos
$ time /cygdrive/e/subversions/subversion/installdir/bin/svn.exe update
At revision 36891.
real 1m18.007s
user 0m6.609s
sys 0m11.124s
This is disappointing, my build of trunk is a lot slower than cygwin's 1.6.
Rebuilt with -pie option to gcc (required by VTune instrumented
profile), ran a call graph profiling session in VTune. Without dumping
the whole data, I'll try to describe some interesting points.
The profiling run took 172s to do an update with no changes.
The majority (96s) was spent under "svn_wc_crawl_revisions5", another
important routine was "svn_wc__adm_open_anchor_in_context" with 34.6s.
Both ultimately spend almost all of their time in "svn_sqlite__step".
Total time spent under "svn_sqlite__step" was 95s, all spent in
"sqlite3_step".
The other important sqlite routine was "sqlite3_exec", in which 15.6s
was spent.
Total time spent in "Kernel32.dll,CloseHandle" was 32.2s, 31.7s of that
was via calls from sqlite routines.
Total time spent in "Kernel32.dll,CreateFileW" was 75.3s, all via sqlite
routines.
So it looks like the old 'Windows file handling is expensive' issue is
still there.
Digging a bit further, it looks like almost all of the time that sqlite
is spending in opening and closing files is in two related routines,
"sqlite3PagerClose" and "sqlite3PagerBegin". Looking at the sqlite
source, it seems that these are related to journalling.
This document:
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html#transaction-wrappers
advises using BEGIN_TRANSACTION and END_TRANSACTION to reduce the number
of times that the journal file is opened. These are used in
svn_sqlite__with_transaction, which is used for all sqlite accesses made
under "svn_wc_crawl_revisions5", but not at all for sqlite access under
"svn_wc__adm_open_anchor_in_context", so maybe there is some
optimisation opportunity there.
However, the majority of time is spent under
"svn_sqlite__with_transaction" so how can we speed that up. We could
either increase the size of the transactions (reducing the calls to
BEGIN/END_TRANSACTION) or make journalling faster.
This document:
http://www.sqlite.org/pragma.html
Says that when the 'locking_mode' is set to EXCLUSIVE, "a small number
of filesystem operations are saved by optimizations enabled in this mode".
Inserting "PRAGMA locking_mode=EXCLUSIVE" into svn_sqlite__open gives:
$ time /cygdrive/e/subversions/subversion/profinstalldir/bin/svn.exe update
At revision 36894.
real 0m47.669s
user 0m6.203s
sys 0m7.593s
So considerably faster than the 1m18s that we saw before, but still
nowhere near as fast as subversion 1.6.
That's where I am for now, I'm going to continue noodling with it but I
hope these results are interesting. If anyone has anything that they
would like me to try I would love to help.
All the best,
Matthew Bentham
ART VPS Ltd.
Re: performance on cygwin
Posted by Matthew Bentham <mj...@artvps.com>.
stsp@stsp.name wrote:
> On Wed, Feb 10, 2010 at 11:57:35AM +0000, Matthew Bentham wrote:
>> I've been hearing rumours of performance improvements to Windows svn
>> clients with the 'wc-ng' work. A couple of weeks ago I decided to
>> have a go with subversion trunk to see what it would make of our
>> repository. I've compiled it, compared the performance with
>> subversion 1.6 from cygwin, and done some profiling with Intel
>> VTune. The results follow, I hope they are interesting.
>
> Thanks for profiling the code. This is quite interesting and
> I think the results are in line with our expectations.
>
> We're aware that trunk is currently slower than 1.6.x.
> The major reason for the current performance problems in trunk is that
> the .svn directories have not been centralised yet. Once there is only
> a single .svn directory at the root of the working copy, we expect a
> large speed-up since only a single sqlite database will be used.
> It would be great if you could profile the code again once we have
> reached that goal. If you want to help us reach that goal faster,
> please join the wc-ng development efffort :)
OK! I'll track changes for now, and I'm here if anyone wants me to try
out something specific.
> The current non-performance of trunk has been discussed before,
> e.g. see http://svn.haxx.se/dev/archive-2009-09/0189.shtml
> and related messages.
Thanks, somehow I missed that thread when reading the archives before
posting.
Just for fun I tried building with "PRAGMA journal_mode=MEMORY", the
result is:
$ time /cygdrive/e/subversions/subversion/profinstalldir/bin/svn.exe update
At revision 36896.
real 0m17.685s
user 0m6.000s
sys 0m7.109s
From my original post, running subversion 1.6:
> $ time svn update
> At revision 36888.
>
> real 0m19.865s
> user 0m0.250s
> sys 0m1.468s
> $
Which I think is pretty conclusive evidence (if any more were needed)
that the performance problems will be solved by reducing the database IO
operations.
All the best,
Matthew
ART VPS
Re: performance on cygwin
Posted by Stefan Sperling <st...@elego.de>.
On Wed, Feb 10, 2010 at 11:57:35AM +0000, Matthew Bentham wrote:
> Hi,
>
> The summary is that I'm motivated to help with work on the
> performance of Windows subversion clients, because the speed of 'svn
> update' is not great for us on large working copies, and we keep
> having to re-arrange our source to work around that.
>
> At work I've used Subversion happily for many years. There are
> perennial discussions in the office about the performance of the
> windows clients. Several times in the past we have had to carefully
> prune the project, especially external modules imported into our
> code, in order to keep acceptable performance of 'svn update'. The
> Linux clients have always been fine.
>
> I've been hearing rumours of performance improvements to Windows svn
> clients with the 'wc-ng' work. A couple of weeks ago I decided to
> have a go with subversion trunk to see what it would make of our
> repository. I've compiled it, compared the performance with
> subversion 1.6 from cygwin, and done some profiling with Intel
> VTune. The results follow, I hope they are interesting.
Thanks for profiling the code. This is quite interesting and
I think the results are in line with our expectations.
We're aware that trunk is currently slower than 1.6.x.
The major reason for the current performance problems in trunk is that
the .svn directories have not been centralised yet. Once there is only
a single .svn directory at the root of the working copy, we expect a
large speed-up since only a single sqlite database will be used.
It would be great if you could profile the code again once we have
reached that goal. If you want to help us reach that goal faster,
please join the wc-ng development efffort :)
The current non-performance of trunk has been discussed before,
e.g. see http://svn.haxx.se/dev/archive-2009-09/0189.shtml
and related messages.
Thanks,
Stefan