You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mattias Engdegård <ma...@bredband.net> on 2012/07/08 18:47:15 UTC
Measured: btrfs COW and sqlite exclusive locking
Out of curiosity, I modified the svn client to use the btrfs facility
for fast file copies (copy-on-write, also known as reflink), to see
what benefits it might possibly give in terms of performance and disk
space. While at it, I also measured the benefits of the sqlite
exclusive locking that everybody have been taking about lately, to see
if it really makes a difference on fast file systems.
Using the btrfs COW mechanism will make the copied files share the
data on disk in a transparent way, and is implemented in cp(1) from
the GNU coreutils (the --reflink option). Since the bulk of a working
tree typically consists of duplicated files, this seems very attractive.
The changes were straightforward; please tell me if you want to see
the patches.
All tests were done against a dedicated svnserve on another machine.
The tree is a typical medium-sized source tree: 27000 files and
directories, 263 MiB in total (.svn metadata and pristine storage not
included).
The client is based on a fairly recent 1.7.x (r1357273). The server
runs 1.6.16.
Times for checking out a fresh tree, in seconds:
default exclusive locking
btrfs 26.4 19.1
btrfs + COW 28.8 21.3
ext4 28.7 19.4
tmpfs 16.7 13.9
nfs 743.2 143.2
The biggest surprise was that COW actually made the checkout slightly
slower, even though no "true" file copies were made. I'm not sure how
to explain this---perhaps everything is already in cache so the copies
aren't very expensive, or the COW operations are somehow
synchronising? More research is clearly needed. Free-standing tests on
large files show that COW copies are much faster than moving the bits
using read/write.
However, the disk usage (as reported by df(1)) was almost halved with
COW, as expected:
disk usage (MiB)
btrfs 573
btrfs + COW 361
These savings could very well be worth the performance penalties.
Also interesting were the gains of exclusive sqlite locking even on
fast local file systems. Not only does NFS go from unusably slow to
barely tolerable, even on tmpfs do we see substantial improvements. No
question about it: for the next production svn build here, I'm turning
on exclusive locking by default!
Times for a null update (hot cache):
default exclusive locking
btrfs 0.71 0.68
ext4 0.68 0.65
tmpfs 0.68 0.65
nfs 9.0 2.3
Times for a slightly less null update (1000 revisions, roughly as many
files changed):
default exclusive locking
btrfs 3.2 2.7
btrfs + COW 4.0 3.5
ext4 2.8 2.6
tmpfs 2.6 2.5
nfs 73.7 15.3
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Mattias Engdegård <ma...@bredband.net>.
10 jul 2012 kl. 22.38 skrev Peter Samuelson:
> Whatever it is in the data path that makes it slower, it probably is
> not a fair comparison. Even though we normally don't want to do our
> own fdatasync(), it is fair to consider the additional I/O load that
> is
> generated by Subversion operations. That the I/O might happen in the
> background after an svn operation finishes does not make it "free"
> from
> the user's standpoint.
I re-ran the checkout test followed by an immediate sync in a simple
attempt to test your hypothesis, but the difference remained:
standard exclusive locking
btrfs 33.1 26.0
btrfs + COW 36.4 27.2
Even so, I still believe using COW where available would be a sensible
move, for the disk space savings if nothing else.
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Peter Samuelson <pe...@p12n.org>.
[Mattias Engdegård]
> The biggest surprise was that COW actually made the checkout slightly
> slower, even though no "true" file copies were made. I'm not sure how
> to explain this---perhaps everything is already in cache so the
> copies aren't very expensive, or the COW operations are somehow
> synchronising?
Whatever it is in the data path that makes it slower, it probably is
not a fair comparison. Even though we normally don't want to do our
own fdatasync(), it is fair to consider the additional I/O load that is
generated by Subversion operations. That the I/O might happen in the
background after an svn operation finishes does not make it "free" from
the user's standpoint.
And of course, this load ought to be significantly lower with COW. It
should be measurable if you run 'sync' from the shell in strategic
places during the test runs. Though even this may not give the whole
picture, if a test run involves creating and deleting a lot of repos
and wcs. (Deleting a repos or a wc before it ever has the chance to
hit the disk platters is typical of testsuites, but rather _atypical_
of real world usage.)
Peter
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Branko Čibej <br...@wandisco.com>.
On 08.07.2012 17:47, Mattias Engdegård wrote:
> The biggest surprise was that COW actually made the checkout slightly
> slower, even though no "true" file copies were made. I'm not sure how
> to explain this---perhaps everything is already in cache so the copies
> aren't very expensive, or the COW operations are somehow synchronising?
Copy-on-Write requires some metadata and processing overhead compared to
just-plain copying, which obviously makes it slower. It's a time vs.
space optimization.
-- Brane
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Mattias Engdegård <ma...@bredband.net>.
9 jul 2012 kl. 18.08 skrev Daniel Shahaf:
> Thanks for clarifying. I'm not familiar enough with the library to
> have much more to say, but I thought I would point out svn_tristate_t
> (cf your maybe_t).
Thank you, I didn't know about svn_tristate_t. I'll be sure to use it
in any production version of the patch, something I would not mind
putting together if there is general interest and agreement about the
utility and soundness of the basic idea.
Windows programmers may want to investigate the win32 call CopyFile().
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Daniel Shahaf <da...@elego.de>.
Mattias Engdegård wrote on Mon, Jul 09, 2012 at 09:40:48 +0200:
> 8 jul 2012 kl. 19.26 skrev Daniel Shahaf:
>
> >Yes, please specify what the client used COW for. Was it for
> >populating the working tree files from the pristine store? Was it for
> >something else?
>
> It was only intended for that, but it's possible that the copying
> code is used for other purposes as well. The changes look bigger
> than they really are, because I had to reorganise some code (in
> workqueue.c) in order to expose a plain file copy that was hiding
> inside.
Thanks for clarifying. I'm not familiar enough with the library to
have much more to say, but I thought I would point out svn_tristate_t
(cf your maybe_t).
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Mattias Engdegård <ma...@bredband.net>.
8 jul 2012 kl. 19.26 skrev Daniel Shahaf:
> Yes, please specify what the client used COW for. Was it for
> populating the working tree files from the pristine store? Was it for
> something else?
It was only intended for that, but it's possible that the copying code
is used for other purposes as well. The changes look bigger than they
really are, because I had to reorganise some code (in workqueue.c) in
order to expose a plain file copy that was hiding inside.
Index: subversion/libsvn_subr/io.c
===================================================================
--- subversion/libsvn_subr/io.c (revision 1357273)
+++ subversion/libsvn_subr/io.c (working copy)
@@ -29,6 +29,8 @@
#include <unistd.h>
#endif
+#include <sys/ioctl.h>
+
#ifndef APR_STATUS_IS_EPERM
#include <errno.h>
#ifdef EPERM
@@ -706,7 +708,14 @@
return SVN_NO_ERROR;
}
+#define BTRFS_IOCTL_MAGIC 0x94
+#define BTRFS_IOC_CLONE _IOW(BTRFS_IOCTL_MAGIC, 9, int)
+static int
+cow_copy_file(int dest_fd, int src_fd)
+{
+ return ioctl(dest_fd, BTRFS_IOC_CLONE, src_fd);
+}
/*** Creating, copying and appending files. ***/
@@ -768,6 +777,13 @@
const char *dst_tmp;
svn_error_t *err;
+ typedef enum { No, Unknown, Yes } maybe_t;
+
+ static maybe_t want_cow_copy = Unknown;
+ static maybe_t cow_copy_available = Unknown;
+ svn_boolean_t did_cow_copy;
+
+
/* ### NOTE: sometimes src == dst. In this case, because we copy
to a
### temporary file, and then rename over the top of the
destination,
### the net result is resetting the permissions on src/dst.
@@ -796,8 +812,42 @@
svn_dirent_dirname(dst, pool),
svn_io_file_del_none, pool, pool));
- apr_err = copy_contents(from_file, to_file, pool);
+ if (want_cow_copy == Unknown)
+ want_cow_copy = getenv("SVN_COW_COPY") ? Yes : No;
+ did_cow_copy = FALSE;
+ if (want_cow_copy == Yes && cow_copy_available != No)
+ {
+ apr_os_file_t from_fd;
+ apr_os_file_t to_fd;
+ apr_os_file_get(&from_fd, from_file);
+ apr_os_file_get(&to_fd, to_file);
+ int r = cow_copy_file(to_fd, from_fd);
+ if (r == 0)
+ {
+ did_cow_copy = TRUE;
+ if (cow_copy_available == Unknown)
+ fprintf(stderr, "svn: using cow copy\n");
+ cow_copy_available = Yes;
+ apr_err = 0;
+ }
+ else
+ {
+ if (errno == ENOTTY)
+ {
+ cow_copy_available = No;
+ fprintf(stderr, "svn: cow copy not available here (%s)
\n", src);
+ }
+ else
+ {
+ apr_err = apr_get_os_error();
+ SVN_ERR_ASSERT(apr_err);
+ }
+ }
+ }
+ if (!did_cow_copy)
+ apr_err = copy_contents(from_file, to_file, pool);
+
if (apr_err)
{
err = svn_error_wrap_apr(apr_err, _("Can't copy '%s' to '%s'"),
Index: subversion/libsvn_wc/workqueue.c
===================================================================
--- subversion/libsvn_wc/workqueue.c (revision 1357273)
+++ subversion/libsvn_wc/workqueue.c (working copy)
@@ -692,9 +692,6 @@
scratch_pool,
scratch_pool));
}
- SVN_ERR(svn_stream_open_readonly(&src_stream, source_abspath,
- scratch_pool, scratch_pool));
-
/* Fetch all the translation bits. */
SVN_ERR(svn_wc__get_translate_info(&style, &eol,
&keywords,
@@ -708,6 +705,9 @@
SVN_ERR(svn_subst_create_specialfile(&dst_stream, local_abspath,
scratch_pool,
scratch_pool));
+ SVN_ERR(svn_stream_open_readonly(&src_stream, source_abspath,
+ scratch_pool, scratch_pool));
+
/* Copy the "repository normal" form of the special file into
the
special stream. */
SVN_ERR(svn_stream_copy3(src_stream, dst_stream,
@@ -718,68 +718,55 @@
return SVN_NO_ERROR;
}
+ /* Where is the Right Place to put a temp file in this working
copy? */
+ SVN_ERR(svn_wc__db_temp_wcroot_tempdir(&temp_dir_abspath,
+ db, wcroot_abspath,
+ scratch_pool, scratch_pool));
+
if (svn_subst_translation_required(style, eol, keywords,
FALSE /* special */,
TRUE /* force_eol_check */))
{
+ SVN_ERR(svn_stream_open_readonly(&src_stream, source_abspath,
+ scratch_pool, scratch_pool));
+
/* Wrap it in a translating (expanding) stream. */
src_stream = svn_subst_stream_translated(src_stream, eol,
TRUE /* repair */,
keywords,
TRUE /* expand */,
scratch_pool);
- }
- /* Where is the Right Place to put a temp file in this working
copy? */
- SVN_ERR(svn_wc__db_temp_wcroot_tempdir(&temp_dir_abspath,
- db, wcroot_abspath,
- scratch_pool, scratch_pool));
+ /* Translate to a temporary file. We don't want the user seeing
a partial
+ file, nor let them muck with it while we translate. We may
also need to
+ get its TRANSLATED_SIZE before the user can monkey it. */
+ SVN_ERR(svn_stream_open_unique(&dst_stream, &dst_abspath,
+ temp_dir_abspath,
+ svn_io_file_del_none,
+ scratch_pool, scratch_pool));
+
+ /* Copy from the source to the dest, translating as we go. This
will also
+ close both streams. */
+ SVN_ERR(svn_stream_copy3(src_stream, dst_stream,
+ cancel_func, cancel_baton,
+ scratch_pool));
- /* Translate to a temporary file. We don't want the user seeing a
partial
- file, nor let them muck with it while we translate. We may also
need to
- get its TRANSLATED_SIZE before the user can monkey it. */
- SVN_ERR(svn_stream_open_unique(&dst_stream, &dst_abspath,
- temp_dir_abspath,
- svn_io_file_del_none,
- scratch_pool, scratch_pool));
+ /* All done. Move the file into place. */
+ SVN_ERR(svn_io_file_rename(dst_abspath, local_abspath,
scratch_pool));
+ }
+ else
+ {
+ /* No translation required - do a fast file-to-file copy. */
- /* Copy from the source to the dest, translating as we go. This
will also
- close both streams. */
- SVN_ERR(svn_stream_copy3(src_stream, dst_stream,
- cancel_func, cancel_baton,
- scratch_pool));
+ SVN_ERR(svn_io_copy_file(source_abspath, local_abspath,
+ FALSE, scratch_pool));
+ }
- /* All done. Move the file into place. */
+ /* FIXME: The old "tolerant" code that would make missing
directories if
+ the rename failed has been ditched. If we want to keep it, it
could be
+ done prior to the copy instead. (It never worked for "special"
files
+ anyway.) */
- {
- svn_error_t *err;
-
- err = svn_io_file_rename(dst_abspath, local_abspath, scratch_pool);
-
- /* With a single db we might want to install files in a missing
directory.
- Simply trying this scenario on error won't do any harm and at
least
- one user reported this problem on IRC. */
- if (err && APR_STATUS_IS_ENOENT(err->apr_err))
- {
- svn_error_t *err2;
-
- err2 =
svn_io_make_dir_recursively(svn_dirent_dirname(local_abspath,
-
scratch_pool),
- scratch_pool);
-
- if (err2)
- /* Creating directory didn't work: Return all errors */
- return svn_error_trace(svn_error_compose_create(err, err2));
- else
- /* We could create a directory: retry install */
- svn_error_clear(err);
-
- SVN_ERR(svn_io_file_rename(dst_abspath, local_abspath,
scratch_pool));
- }
- else
- SVN_ERR(err);
- }
-
/* Tweak the on-disk file according to its properties. */
if (props
&& (apr_hash_get(props, SVN_PROP_NEEDS_LOCK,
APR_HASH_KEY_STRING)
Re: Measured: btrfs COW and sqlite exclusive locking
Posted by Daniel Shahaf <da...@elego.de>.
Mattias Engdegård wrote on Sun, Jul 08, 2012 at 18:47:15 +0200:
> Out of curiosity, I modified the svn client to use the btrfs
> facility for fast file copies (copy-on-write, also known as
> reflink), to see what benefits it might possibly give in terms of
> performance and disk space. While at it, I also measured the
> benefits of the sqlite exclusive locking that everybody have been
> taking about lately, to see if it really makes a difference on fast
> file systems.
>
> Using the btrfs COW mechanism will make the copied files share the
> data on disk in a transparent way, and is implemented in cp(1) from
> the GNU coreutils (the --reflink option). Since the bulk of a
> working tree typically consists of duplicated files, this seems very
> attractive.
>
> The changes were straightforward; please tell me if you want to see
> the patches.
Yes, please specify what the client used COW for. Was it for
populating the working tree files from the pristine store? Was it for
something else?