You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Philip Martin <ph...@wandisco.com> on 2015/10/15 16:09:30 UTC
NFS performance regression in 1.9
There is a performance regression for checkout on NFS with 1.9. I'm
using 1.7.x as my dataset, a 95MB checkout, with localhost as my
Subversion server, so NFS is almost the only network traffic. I'm
comparing 1.7 patched to enable exclusive locking, with 1.8 and 1.9 both
with exclusive locking enabled:
1.7: 73s
1.8: 66s
1.9: 90s
For comparison a local checkout is less than 2s. I've tried both
http:// and svn:// and get similar results.
So 1.8 is better than patched 1.7 but 1.9 is much slower than both.
Using "strace -c" I can see 1.9 generally makes slightly fewer system
calls apart from chmod() and ftruncate(). The extra chmod() calls are a
result of a new feature in 1.9: the pristine text-bases are read-only.
The ftruncate() calls are due to r1659426:
Use journal_mode = TRUNCATE for sqlite [1] instead of default DELETE value.
It makes svn checkout about 10% faster on Windows at expense of one empty
file in .svn. Create/delete file is relatively expensive operation on
Windows, especially if indexes/virus scanners are enabled.
These two changes explain the 1.9 regression, if I patch them back then
1.9 is as good as, or better than, 1.8.
We can't fix the chmod() problem: the new feature has a cost. We also
don't want to give up a 10% Windows gain. The TRUNCATE change is the
one that has the biggest effect on NFS performance so I propose to
continue with TRUNCATE as default but have exclusive locking mode switch
back to DELETE:
Index: ../src-1.9/subversion/libsvn_wc/wc-queries.sql
===================================================================
--- ../src-1.9/subversion/libsvn_wc/wc-queries.sql (revision 1708752)
+++ ../src-1.9/subversion/libsvn_wc/wc-queries.sql (working copy)
@@ -1286,7 +1286,8 @@
OR (wc_id = ?1 AND IS_STRICT_DESCENDANT_OF(local_relpath, ?2))
-- STMT_PRAGMA_LOCKING_MODE
-PRAGMA locking_mode = exclusive
+PRAGMA locking_mode = exclusive;
+PRAGMA journal_mode = DELETE
/* ------------------------------------------------------------------------- */
This change brings the 1.9 checkout time down to 69s, so only 5% slower
than 1.8 rather than 36%.
--
Philip Martin
WANdisco
Re: NFS performance regression in 1.9
Posted by Philip Martin <ph...@wandisco.com>.
Julian Foad <ju...@gmail.com> writes:
> Test time in seconds (two runs) Svn version
> 739, 731 1.8.15
> 1068, 1071 1.9.3
> 802, 805 1.9.x+r1710167
That corresponds to my figures as far as I recall. The reason 1.9 is
slower than 1.8 is that 1.9 makes the pristines read-only and the extra
chmods() more than cancel out any wc.db improvements.
--
Philip Martin
WANdisco
Re: NFS performance regression in 1.9
Posted by Julian Foad <ju...@gmail.com>.
On 8 January 2016, Julian Foad wrote:
> This commit was
>
> http://svn.apache.org/r1710167
>
> The backport proposal was not approved in time for 1.9.3. Due to
> customer pressure I'm now testing & reviewing it.
+1, approved.
On my system, the total test times for 'exclusive locking' mode were:
Test time in seconds (two runs) Svn version
739, 731 1.8.15
1068, 1071 1.9.3
802, 805 1.9.x+r1710167
(WC on a Linux-based NAS server, NFSv3 over 100M ethernet, repository
and everything else on Linux client with SSD.)
With the WC on the local SSD, the patch causes an increase in test
times in 'exclusive locking' mode:
Test time in seconds (two runs) Svn version
81, 80 1.8.15
85, 84 1.9.3
92, 89 1.9.x+r1710167
But that's not the use case this mode is intended for, so I suppose that is OK.
- Julian
Re: NFS performance regression in 1.9
Posted by Julian Foad <ju...@gmail.com>.
Philip Martin wrote:
> I've committed this change and proposed it for 1.9. It only affects
> people using exclusive-locking mode working copies and I don't believe
> there are any compatibility problems. It goes some way to restoring the
> 1.9 performance on remote disks.
This commit was
http://svn.apache.org/r1710167
The backport proposal was not approved in time for 1.9.3. Due to
customer pressure I'm now testing & reviewing it.
- Julian
Re: NFS performance regression in 1.9
Posted by Philip Martin <ph...@wandisco.com>.
Branko Čibej <br...@apache.org> writes:
> On 16.10.2015 13:14, Philip Martin wrote:
>> Philip Martin <ph...@wandisco.com> writes:
>>
>>> "Bert Huijben" <be...@qqmail.nl> writes:
>>>
>>>> I'm not able to answer all that, but I do know that your change will
>>>> slow Subversion down on Samba shares as used from Windows systems with
>>>> that flag. And once one user used it in that way it will stay slow
>>>> because the journal mode is stored in the database.
>>>>
>>>> I don't think just touching a working copy with a client should ever
>>>> have such an effect on a working copy.
>>> The journal mode is set to TRUNCATE every time we open the database,
>>> that doesn't change with my patch. Are you saying that TRUNCATE will be
>>> ignored after some previous connection uses DELETE?
>> The documentation explicitly states that journal_mode=WAL stays in
>> effect for later connections, but doesn't say the same for the other
>> modes:
>
> I think that means "for later concurrent connections to the database",
> meaning until the last process closes it. That would make sense, because
> you can't combine WAL mode with either truncate or delete at the same time.
I've committed this change and proposed it for 1.9. It only affects
people using exclusive-locking mode working copies and I don't believe
there are any compatibility problems. It goes some way to restoring the
1.9 performance on remote disks.
--
Philip Martin
WANdisco
Re: NFS performance regression in 1.9
Posted by Branko Čibej <br...@apache.org>.
On 16.10.2015 13:14, Philip Martin wrote:
> Philip Martin <ph...@wandisco.com> writes:
>
>> "Bert Huijben" <be...@qqmail.nl> writes:
>>
>>> I'm not able to answer all that, but I do know that your change will
>>> slow Subversion down on Samba shares as used from Windows systems with
>>> that flag. And once one user used it in that way it will stay slow
>>> because the journal mode is stored in the database.
>>>
>>> I don't think just touching a working copy with a client should ever
>>> have such an effect on a working copy.
>> The journal mode is set to TRUNCATE every time we open the database,
>> that doesn't change with my patch. Are you saying that TRUNCATE will be
>> ignored after some previous connection uses DELETE?
> The documentation explicitly states that journal_mode=WAL stays in
> effect for later connections, but doesn't say the same for the other
> modes:
I think that means "for later concurrent connections to the database",
meaning until the last process closes it. That would make sense, because
you can't combine WAL mode with either truncate or delete at the same time.
-- Brane
Re: NFS performance regression in 1.9
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> "Bert Huijben" <be...@qqmail.nl> writes:
>
>> I'm not able to answer all that, but I do know that your change will
>> slow Subversion down on Samba shares as used from Windows systems with
>> that flag. And once one user used it in that way it will stay slow
>> because the journal mode is stored in the database.
>>
>> I don't think just touching a working copy with a client should ever
>> have such an effect on a working copy.
>
> The journal mode is set to TRUNCATE every time we open the database,
> that doesn't change with my patch. Are you saying that TRUNCATE will be
> ignored after some previous connection uses DELETE?
The documentation explicitly states that journal_mode=WAL stays in
effect for later connections, but doesn't say the same for the other
modes:
https://sqlite.org/pragma.html#pragma_journal_mode
and that is what I see. journal_mode=delete applies unless we set
truncate and the truncate is temporary:
$ svnadmin create repo
$ svn co file://`pwd`/repo wc
$ sqlite3 wc/.svn/wc.db "pragma journal_mode"
delete
$ sqlite3 wc/.svn/wc.db "pragma journal_mode=truncate"
truncate
$ sqlite3 wc/.svn/wc.db "pragma journal_mode"
delete
$ svn st wc
$ sqlite3 wc/.svn/wc.db "pragma journal_mode"
delete
$ svn st wc --config-option config:working-copy:exclusive-locking=true
$ sqlite3 wc/.svn/wc.db "pragma journal_mode"
delete
--
Philip Martin
WANdisco
Re: NFS performance regression in 1.9
Posted by Philip Martin <ph...@wandisco.com>.
"Bert Huijben" <be...@qqmail.nl> writes:
> I'm not able to answer all that, but I do know that your change will
> slow Subversion down on Samba shares as used from Windows systems with
> that flag. And once one user used it in that way it will stay slow
> because the journal mode is stored in the database.
>
> I don't think just touching a working copy with a client should ever
> have such an effect on a working copy.
The journal mode is set to TRUNCATE every time we open the database,
that doesn't change with my patch. Are you saying that TRUNCATE will be
ignored after some previous connection uses DELETE? The documentation
doesn't appear to support you:
https://sqlite.org/pragma.html#pragma_journal_mode
This pragma queries or sets the journal mode for databases associated
with the current database connection.
Ivan's original change r1659426 stated:
The TRUNCATE and DELETE journal modes are compatible, so different
Subversion clients with different journal mode should just work with same
working copy without problems.
--
Philip Martin
WANdisco
RE: NFS performance regression in 1.9
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Philip Martin [mailto:philip.martin@wandisco.com]
> Sent: donderdag 15 oktober 2015 19:40
> To: Branko Čibej <br...@apache.org>
> Cc: dev@subversion.apache.org
> Subject: Re: NFS performance regression in 1.9
>
> Branko Čibej <br...@apache.org> writes:
>
> > And in another thread (on IRC, I think) we talked about not recommending
> > NFS because it's not reliable given our requirement for atomic renames.
>
> Lots of people, including Subversion developers, use working copies on
> NFS successfully. It may not be perfect but it works well enough in
> lots of cases. Pointing out potential pitfalls is fine, any sort of
> blanket ban would be odd. Repositories on NFS might be more vulnerable.
>
> I think the usual problem for NFS rename is that while the underlying
> filesystem implements an atomic rename the mechanism that reports back
> to the client may fail. Such an error may cause the client to exit with
> an error but the working copy itself is probably OK.
>
> All filesystems can have bugs. How likely is NFS to fail for a typical
> user of Subversion working copies on NFS? Is NFS less reliable than a
> local filesystem that uses LVM+mdadm+ZFS/BtrFS? Is NFS less reliable
> than the firmware in a h/w RAID card?
I'm not able to answer all that, but I do know that your change will slow Subversion down on Samba shares as used from Windows systems with that flag. And once one user used it in that way it will stay slow because the journal mode is stored in the database.
I don't think just touching a working copy with a client should ever have such an effect on a working copy.
Bert
Re: NFS performance regression in 1.9
Posted by Philip Martin <ph...@wandisco.com>.
Branko Čibej <br...@apache.org> writes:
> And in another thread (on IRC, I think) we talked about not recommending
> NFS because it's not reliable given our requirement for atomic renames.
Lots of people, including Subversion developers, use working copies on
NFS successfully. It may not be perfect but it works well enough in
lots of cases. Pointing out potential pitfalls is fine, any sort of
blanket ban would be odd. Repositories on NFS might be more vulnerable.
I think the usual problem for NFS rename is that while the underlying
filesystem implements an atomic rename the mechanism that reports back
to the client may fail. Such an error may cause the client to exit with
an error but the working copy itself is probably OK.
All filesystems can have bugs. How likely is NFS to fail for a typical
user of Subversion working copies on NFS? Is NFS less reliable than a
local filesystem that uses LVM+mdadm+ZFS/BtrFS? Is NFS less reliable
than the firmware in a h/w RAID card?
--
Philip Martin
WANdisco
Re: NFS performance regression in 1.9
Posted by Branko Čibej <br...@apache.org>.
On 15.10.2015 16:09, Philip Martin wrote:
> There is a performance regression for checkout on NFS with 1.9. I'm
> using 1.7.x as my dataset, a 95MB checkout, with localhost as my
> Subversion server, so NFS is almost the only network traffic. I'm
> comparing 1.7 patched to enable exclusive locking, with 1.8 and 1.9 both
> with exclusive locking enabled:
>
> 1.7: 73s
> 1.8: 66s
> 1.9: 90s
>
> For comparison a local checkout is less than 2s. I've tried both
> http:// and svn:// and get similar results.
>
> So 1.8 is better than patched 1.7 but 1.9 is much slower than both.
>
> Using "strace -c" I can see 1.9 generally makes slightly fewer system
> calls apart from chmod() and ftruncate(). The extra chmod() calls are a
> result of a new feature in 1.9: the pristine text-bases are read-only.
> The ftruncate() calls are due to r1659426:
>
> Use journal_mode = TRUNCATE for sqlite [1] instead of default DELETE value.
> It makes svn checkout about 10% faster on Windows at expense of one empty
> file in .svn. Create/delete file is relatively expensive operation on
> Windows, especially if indexes/virus scanners are enabled.
>
> These two changes explain the 1.9 regression, if I patch them back then
> 1.9 is as good as, or better than, 1.8.
>
> We can't fix the chmod() problem: the new feature has a cost. We also
> don't want to give up a 10% Windows gain. The TRUNCATE change is the
> one that has the biggest effect on NFS performance so I propose to
> continue with TRUNCATE as default but have exclusive locking mode switch
> back to DELETE:
>
> Index: ../src-1.9/subversion/libsvn_wc/wc-queries.sql
> ===================================================================
> --- ../src-1.9/subversion/libsvn_wc/wc-queries.sql (revision 1708752)
> +++ ../src-1.9/subversion/libsvn_wc/wc-queries.sql (working copy)
> @@ -1286,7 +1286,8 @@
> OR (wc_id = ?1 AND IS_STRICT_DESCENDANT_OF(local_relpath, ?2))
>
> -- STMT_PRAGMA_LOCKING_MODE
> -PRAGMA locking_mode = exclusive
> +PRAGMA locking_mode = exclusive;
> +PRAGMA journal_mode = DELETE
>
> /* ------------------------------------------------------------------------- */
>
> This change brings the 1.9 checkout time down to 69s, so only 5% slower
> than 1.8 rather than 36%.
And in another thread (on IRC, I think) we talked about not recommending
NFS because it's not reliable given our requirement for atomic renames.
-- Brane