You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by st...@apache.org on 2014/03/07 22:35:55 UTC
svn commit: r1575418 - in /subversion/trunk/subversion/libsvn_fs_fs: fs.h
fs_fs.c fs_fs.h pack.c structure
Author: stefan2
Date: Fri Mar 7 21:35:54 2014
New Revision: 1575418
URL: http://svn.apache.org/r1575418
Log:
Enable FSFS format 7 repositories to be packed without completely
blocking commits. We simply introduce a separate lock file for
'svnadmin pack' and take out the global write lock for packing
revprops and switching the shard to "packed" state.
Most of the run time is spent building the revision pack file
and does not require any synchronization with writers.
Packing large shards no longer causes noticable delays to committers.
This is important for format 7 because the data reorganization may
involes significantly more I/O than the append-only process used
with previous formats. We can now pack during 'svnadmin load'
without delaying the process.
* subversion/libsvn_fs_fs/fs.h
(PATH_PACK_LOCK_FILE): Declare the new lock file name.
(SVN_FS_FS__MIN_PACK_LOCK_FORMAT): Define when that is available.
* subversion/libsvn_fs_fs/fs_fs.h
(svn_fs_fs__get_lock_on_filesystem): Publish previously static function.
* subversion/libsvn_fs_fs/fs_fs.c
(get_lock_on_filesystem): Rename to ...
(svn_fs_fs__get_lock_on_filesystem): ... this.
(with_some_lock_file): Update caller.
* subversion/libsvn_fs_fs/pack.c
(pack_baton): Extend by adding all previous parameters of the pack_*
functions as members. All pack_* function now use the
baton for their parameter passing.
(synced_pack_shard): Extracted from pack_shard; handles the revprop
packing and shard pack switch-over. This will
be executed while holding the global write lock.
(pack_shard): Wraps the above and adds the parts that don't require
the global write lock such as packing the rev data.
(pack_body): Use the baton to pass parameters along.
(svn_fs_fs__pack): When available, acquire only the pack-lock.
* subversion/libsvn_fs_fs/structure
(Layout of the FS directory): Document the new lock file and update
the doc for the global write-lock.
Modified:
subversion/trunk/subversion/libsvn_fs_fs/fs.h
subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c
subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h
subversion/trunk/subversion/libsvn_fs_fs/pack.c
subversion/trunk/subversion/libsvn_fs_fs/structure
Modified: subversion/trunk/subversion/libsvn_fs_fs/fs.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/fs.h?rev=1575418&r1=1575417&r2=1575418&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/fs.h (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/fs.h Fri Mar 7 21:35:54 2014
@@ -55,6 +55,7 @@ extern "C" {
#define PATH_UUID "uuid" /* Contains UUID */
#define PATH_CURRENT "current" /* Youngest revision */
#define PATH_LOCK_FILE "write-lock" /* Revision lock file */
+#define PATH_PACK_LOCK_FILE "pack-lock" /* Pack lock file */
#define PATH_REVS_DIR "revs" /* Directory of revisions */
#define PATH_REVPROPS_DIR "revprops" /* Directory of revprops */
#define PATH_TXNS_DIR "transactions" /* Directory of transactions */
@@ -170,6 +171,9 @@ extern "C" {
/* The minimum format number that supports packed revprops. */
#define SVN_FS_FS__MIN_LOG_ADDRESSING_FORMAT 7
+/* Minimum format number that providing a separate lock file for pack ops */
+#define SVN_FS_FS__MIN_PACK_LOCK_FORMAT 7
+
/* Minimum format number that stores mergeinfo-mode flag in changed paths */
#define SVN_FS_FS__MIN_MERGEINFO_IN_CHANGES_FORMAT 7
Modified: subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c?rev=1575418&r1=1575417&r2=1575418&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c Fri Mar 7 21:35:54 2014
@@ -114,9 +114,9 @@ path_lock(svn_fs_t *fs, apr_pool_t *pool
/* Get a lock on empty file LOCK_FILENAME, creating it in POOL. */
-static svn_error_t *
-get_lock_on_filesystem(const char *lock_filename,
- apr_pool_t *pool)
+svn_error_t *
+svn_fs_fs__get_lock_on_filesystem(const char *lock_filename,
+ apr_pool_t *pool)
{
svn_error_t *err = svn_io_file_lock2(lock_filename, TRUE, FALSE, pool);
@@ -160,7 +160,8 @@ with_some_lock_file(svn_fs_t *fs,
apr_pool_t *pool)
{
apr_pool_t *subpool = svn_pool_create(pool);
- svn_error_t *err = get_lock_on_filesystem(lock_filename, subpool);
+ svn_error_t *err = svn_fs_fs__get_lock_on_filesystem(lock_filename,
+ subpool);
if (!err)
{
Modified: subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h?rev=1575418&r1=1575417&r2=1575418&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h Fri Mar 7 21:35:54 2014
@@ -136,6 +136,12 @@ svn_fs_fs__write_format(svn_fs_t *fs,
svn_boolean_t overwrite,
apr_pool_t *pool);
+/* Get a lock on empty file LOCK_FILENAME, creating it in POOL;
+ releasing the lock upon POOL cleanup. */
+svn_error_t *
+svn_fs_fs__get_lock_on_filesystem(const char *lock_filename,
+ apr_pool_t *pool);
+
/* Obtain a write lock on the filesystem FS in a subpool of POOL, call
BODY with BATON and that subpool, destroy the subpool (releasing the write
lock) and return what BODY returned. */
Modified: subversion/trunk/subversion/libsvn_fs_fs/pack.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/pack.c?rev=1575418&r1=1575417&r2=1575418&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/pack.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/pack.c Fri Mar 7 21:35:54 2014
@@ -1777,105 +1777,96 @@ pack_rev_shard(svn_fs_t *fs,
return SVN_NO_ERROR;
}
-/* In the file system at FS_PATH, pack the SHARD in REVS_DIR and
- * REVPROPS_DIR containing exactly MAX_FILES_PER_DIR revisions, using POOL
- * for allocations. REVPROPS_DIR will be NULL if revprop packing is not
- * supported. COMPRESSION_LEVEL and MAX_PACK_SIZE will be ignored in that
- * case.
- *
- * CANCEL_FUNC and CANCEL_BATON are what you think they are; similarly
- * NOTIFY_FUNC and NOTIFY_BATON.
- *
- * If for some reason we detect a partial packing already performed, we
- * remove the pack file and start again.
- */
-static svn_error_t *
-pack_shard(const char *revs_dir,
- const char *revsprops_dir,
- svn_fs_t *fs,
- apr_int64_t shard,
- int max_files_per_dir,
- apr_off_t max_pack_size,
- int compression_level,
- svn_fs_pack_notify_t notify_func,
- void *notify_baton,
- svn_cancel_func_t cancel_func,
- void *cancel_baton,
- apr_pool_t *pool)
+/* Baton struct used by pack_body(), pack_shard() and synced_pack_shard().
+ These calls are nested and for every level additional fields will be
+ available. */
+struct pack_baton
{
- fs_fs_data_t *ffd = fs->fsap_data;
- const char *rev_shard_path, *rev_pack_file_dir;
- const char *revprops_shard_path, *revprops_pack_file_dir;
+ /* Valid when entering pack_body(). */
+ svn_fs_t *fs;
+ svn_fs_pack_notify_t notify_func;
+ void *notify_baton;
+ svn_cancel_func_t cancel_func;
+ void *cancel_baton;
- /* Notify caller we're starting to pack this shard. */
- if (notify_func)
- SVN_ERR(notify_func(notify_baton, shard, svn_fs_pack_notify_start,
- pool));
+ /* Additional entries valid when entering pack_shard(). */
+ const char *revs_dir;
+ const char *revsprops_dir;
+ apr_int64_t shard;
- /* Some useful paths. */
- rev_pack_file_dir = svn_dirent_join(revs_dir,
- apr_psprintf(pool,
- "%" APR_INT64_T_FMT PATH_EXT_PACKED_SHARD,
- shard),
- pool);
- rev_shard_path = svn_dirent_join(revs_dir,
- apr_psprintf(pool, "%" APR_INT64_T_FMT, shard),
- pool);
+ /* Additional entries valid when entering synced_pack_shard(). */
+ const char *rev_shard_path;
+};
- /* pack the revision content */
- SVN_ERR(pack_rev_shard(fs, rev_pack_file_dir, rev_shard_path,
- shard, max_files_per_dir, DEFAULT_MAX_MEM,
- cancel_func, cancel_baton, pool));
+
+/* Part of the pack process that requires global (write) synchronization.
+ * We pack the revision properties of the shard described by BATON and
+ * In the file system at FS_PATH, pack the SHARD in REVS_DIR and replace
+ * the non-packed revprop & rev shard folder(s) with the packed ones.
+ * The packed rev folder has been created prior to calling this function.
+ */
+static svn_error_t *
+synced_pack_shard(void *baton,
+ apr_pool_t *pool)
+{
+ struct pack_baton *pb = baton;
+ fs_fs_data_t *ffd = pb->fs->fsap_data;
+ const char *revprops_shard_path, *revprops_pack_file_dir;
/* if enabled, pack the revprops in an equivalent way */
- if (revsprops_dir)
+ if (pb->revsprops_dir)
{
- revprops_pack_file_dir = svn_dirent_join(revsprops_dir,
+ revprops_pack_file_dir = svn_dirent_join(pb->revsprops_dir,
apr_psprintf(pool,
"%" APR_INT64_T_FMT PATH_EXT_PACKED_SHARD,
- shard),
+ pb->shard),
pool);
- revprops_shard_path = svn_dirent_join(revsprops_dir,
- apr_psprintf(pool, "%" APR_INT64_T_FMT, shard),
- pool);
+ revprops_shard_path = svn_dirent_join(pb->revsprops_dir,
+ apr_psprintf(pool, "%" APR_INT64_T_FMT, pb->shard),
+ pool);
SVN_ERR(svn_fs_fs__pack_revprops_shard(revprops_pack_file_dir,
revprops_shard_path,
- shard, max_files_per_dir,
- (int)(0.9 * max_pack_size),
- compression_level,
- cancel_func, cancel_baton,
+ pb->shard,
+ ffd->max_files_per_dir,
+ (int)(0.9*ffd->revprop_pack_size),
+ ffd->compress_packed_revprops
+ ? SVN__COMPRESSION_ZLIB_DEFAULT
+ : SVN__COMPRESSION_NONE,
+ pb->cancel_func,
+ pb->cancel_baton,
pool));
}
/* Update the min-unpacked-rev file to reflect our newly packed shard. */
- SVN_ERR(svn_fs_fs__write_min_unpacked_rev(fs,
- (svn_revnum_t)((shard + 1) * max_files_per_dir),
- pool));
- ffd->min_unpacked_rev = (svn_revnum_t)((shard + 1) * max_files_per_dir);
+ SVN_ERR(svn_fs_fs__write_min_unpacked_rev(pb->fs,
+ (svn_revnum_t)((pb->shard + 1) * ffd->max_files_per_dir),
+ pool));
+ ffd->min_unpacked_rev
+ = (svn_revnum_t)((pb->shard + 1) * ffd->max_files_per_dir);
/* Finally, remove the existing shard directories.
* For revprops, clean up older obsolete shards as well as they might
* have been left over from an interrupted FS upgrade. */
- SVN_ERR(svn_io_remove_dir2(rev_shard_path, TRUE,
- cancel_func, cancel_baton, pool));
- if (revsprops_dir)
+ SVN_ERR(svn_io_remove_dir2(pb->rev_shard_path, TRUE,
+ pb->cancel_func, pb->cancel_baton, pool));
+ if (pb->revsprops_dir)
{
svn_node_kind_t kind = svn_node_dir;
- apr_int64_t to_cleanup = shard;
+ apr_int64_t to_cleanup = pb->shard;
do
{
SVN_ERR(svn_fs_fs__delete_revprops_shard(revprops_shard_path,
to_cleanup,
- max_files_per_dir,
- cancel_func,
- cancel_baton,
+ ffd->max_files_per_dir,
+ pb->cancel_func,
+ pb->cancel_baton,
pool));
/* If the previous shard exists, clean it up as well.
Don't try to clean up shard 0 as it we can't tell quickly
whether it actually needs cleaning up. */
- revprops_shard_path = svn_dirent_join(revsprops_dir,
+ revprops_shard_path = svn_dirent_join(pb->revsprops_dir,
apr_psprintf(pool, "%" APR_INT64_T_FMT, --to_cleanup),
pool);
SVN_ERR(svn_io_check_path(revprops_shard_path, &kind, pool));
@@ -1883,23 +1874,60 @@ pack_shard(const char *revs_dir,
while (kind == svn_node_dir && to_cleanup > 0);
}
- /* Notify caller we're starting to pack this shard. */
- if (notify_func)
- SVN_ERR(notify_func(notify_baton, shard, svn_fs_pack_notify_end,
- pool));
-
return SVN_NO_ERROR;
}
-struct pack_baton
+/* Pack the shard described by BATON.
+ *
+ * If for some reason we detect a partial packing already performed,
+ * we remove the pack file and start again.
+ */
+static svn_error_t *
+pack_shard(struct pack_baton *baton,
+ apr_pool_t *pool)
{
- svn_fs_t *fs;
- svn_fs_pack_notify_t notify_func;
- void *notify_baton;
- svn_cancel_func_t cancel_func;
- void *cancel_baton;
-};
+ fs_fs_data_t *ffd = baton->fs->fsap_data;
+ const char *rev_pack_file_dir;
+
+ /* Notify caller we're starting to pack this shard. */
+ if (baton->notify_func)
+ SVN_ERR(baton->notify_func(baton->notify_baton, baton->shard,
+ svn_fs_pack_notify_start, pool));
+ /* Some useful paths. */
+ rev_pack_file_dir = svn_dirent_join(baton->revs_dir,
+ apr_psprintf(pool,
+ "%" APR_INT64_T_FMT PATH_EXT_PACKED_SHARD,
+ baton->shard),
+ pool);
+ baton->rev_shard_path = svn_dirent_join(baton->revs_dir,
+ apr_psprintf(pool,
+ "%" APR_INT64_T_FMT,
+ baton->shard),
+ pool);
+
+ /* pack the revision content */
+ SVN_ERR(pack_rev_shard(baton->fs, rev_pack_file_dir, baton->rev_shard_path,
+ baton->shard, ffd->max_files_per_dir,
+ DEFAULT_MAX_MEM, baton->cancel_func,
+ baton->cancel_baton, pool));
+
+ /* For newer repo formats, we only acquired the pack lock so far.
+ Before modifying the repo state by switching over to the packed
+ data, we need to acquire the global (write) lock. */
+ if (ffd->format >= SVN_FS_FS__MIN_PACK_LOCK_FORMAT)
+ SVN_ERR(svn_fs_fs__with_write_lock(baton->fs, synced_pack_shard, baton,
+ pool));
+ else
+ SVN_ERR(synced_pack_shard(baton, pool));
+
+ /* Notify caller we're starting to pack this shard. */
+ if (baton->notify_func)
+ SVN_ERR(baton->notify_func(baton->notify_baton, baton->shard,
+ svn_fs_pack_notify_end, pool));
+
+ return SVN_NO_ERROR;
+}
/* The work-horse for svn_fs_fs__pack, called with the FS write lock.
This implements the svn_fs_fs__with_write_lock() 'body' callback
@@ -1922,11 +1950,8 @@ pack_body(void *baton,
struct pack_baton *pb = baton;
fs_fs_data_t *ffd = pb->fs->fsap_data;
apr_int64_t completed_shards;
- apr_int64_t i;
svn_revnum_t youngest;
apr_pool_t *iterpool;
- const char *rev_data_path;
- const char *revprops_data_path = NULL;
/* If the repository isn't a new enough format, we don't support packing.
Return a friendly error to that effect. */
@@ -1949,29 +1974,22 @@ pack_body(void *baton,
if (ffd->min_unpacked_rev == (completed_shards * ffd->max_files_per_dir))
return SVN_NO_ERROR;
- rev_data_path = svn_dirent_join(pb->fs->path, PATH_REVS_DIR, pool);
+ pb->revs_dir = svn_dirent_join(pb->fs->path, PATH_REVS_DIR, pool);
if (ffd->format >= SVN_FS_FS__MIN_PACKED_REVPROP_FORMAT)
- revprops_data_path = svn_dirent_join(pb->fs->path, PATH_REVPROPS_DIR,
- pool);
+ pb->revsprops_dir = svn_dirent_join(pb->fs->path, PATH_REVPROPS_DIR,
+ pool);
iterpool = svn_pool_create(pool);
- for (i = ffd->min_unpacked_rev / ffd->max_files_per_dir;
- i < completed_shards;
- i++)
+ for (pb->shard = ffd->min_unpacked_rev / ffd->max_files_per_dir;
+ pb->shard < completed_shards;
+ pb->shard++)
{
svn_pool_clear(iterpool);
if (pb->cancel_func)
SVN_ERR(pb->cancel_func(pb->cancel_baton));
- SVN_ERR(pack_shard(rev_data_path, revprops_data_path,
- pb->fs, i, ffd->max_files_per_dir,
- ffd->revprop_pack_size,
- ffd->compress_packed_revprops
- ? SVN__COMPRESSION_ZLIB_DEFAULT
- : SVN__COMPRESSION_NONE,
- pb->notify_func, pb->notify_baton,
- pb->cancel_func, pb->cancel_baton, iterpool));
+ SVN_ERR(pack_shard(pb, iterpool));
}
svn_pool_destroy(iterpool);
@@ -1987,10 +2005,33 @@ svn_fs_fs__pack(svn_fs_t *fs,
apr_pool_t *pool)
{
struct pack_baton pb = { 0 };
+ fs_fs_data_t *ffd = fs->fsap_data;
+ svn_error_t *err;
+
pb.fs = fs;
pb.notify_func = notify_func;
pb.notify_baton = notify_baton;
pb.cancel_func = cancel_func;
pb.cancel_baton = cancel_baton;
- return svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
+
+ if (ffd->format >= SVN_FS_FS__MIN_PACK_LOCK_FORMAT)
+ {
+ /* Newer repositories provide a pack operation specific lock.
+ Acquire it to prevent concurrent packs. */
+ apr_pool_t *subpool = svn_pool_create(pool);
+ const char *lock_path = svn_dirent_join(fs->path, PATH_PACK_LOCK_FILE,
+ subpool);
+ err = svn_fs_fs__get_lock_on_filesystem(lock_path, subpool);
+ if (!err)
+ err = pack_body(&pb, subpool);
+
+ svn_pool_destroy(subpool);
+ }
+ else
+ {
+ /* Use the global write lock for older repos. */
+ err = svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
+ }
+
+ return svn_error_trace(err);
}
Modified: subversion/trunk/subversion/libsvn_fs_fs/structure
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/structure?rev=1575418&r1=1575417&r2=1575418&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/structure (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/structure Fri Mar 7 21:35:54 2014
@@ -60,6 +60,7 @@ repository) is:
current File specifying current revision and next node/copy id
fs-type File identifying this filesystem as an FSFS filesystem
write-lock Empty file, locked to serialise writers
+ pack-lock Empty file, locked to serialise 'svnadmin pack' (f. 7+)
txn-current-lock Empty file, locked to serialise 'txn-current'
uuid File containing the UUID of the repository
format File containing the format number of this filesystem
@@ -86,9 +87,19 @@ The "write-lock" file is an empty file w
final stage of a commit and unlocked after the new "current" file has
been moved into place to indicate that a new revision is present. It
is also locked during a revprop propchange while the revprop file is
-read in, mutated, and written out again. Note that readers are never
-blocked by any operation - writers must ensure that the filesystem is
-always in a consistent state.
+read in, mutated, and written out again. Furthermore, it will be used
+to serialize the repository structure changes during 'svnadmin pack'
+(see also next section). Note that readers are never blocked by any
+operation - writers must ensure that the filesystem is always in a
+consistent state.
+
+The "pack-lock" file is an empty file which is locked before an 'svnadmin
+pack' operation commences. Thus, only one process may attempt to modify
+the repository structure at a time while other processes may still read
+and write (commit) to the repository during most of the pack procedure.
+It is only available with format 7 and newer repositories. Older formats
+use the global write-lock instead which disables commits completely
+for the duration of the pack process.
The "txn-current" file is a file with a single line of text that
contains only a base-36 number. The current value will be used in the
Re: svn commit: r1575418 - in /subversion/trunk/subversion/libsvn_fs_fs:
fs.h fs_fs.c fs_fs.h pack.c structure
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Wed, Mar 12, 2014 at 9:49 AM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On 11 March 2014 21:56, Stefan Fuhrmann <st...@wandisco.com>
> wrote:
> > On Tue, Mar 11, 2014 at 1:23 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> >>
> >> On 8 March 2014 01:35, <st...@apache.org> wrote:
> >> > Author: stefan2
> >> > Date: Fri Mar 7 21:35:54 2014
> >> > New Revision: 1575418
> >> >
> >> > URL: http://svn.apache.org/r1575418
> >> > Log:
> >> > Enable FSFS format 7 repositories to be packed without completely
> >> > blocking commits. We simply introduce a separate lock file for
> >> > 'svnadmin pack' and take out the global write lock for packing
> >> > revprops and switching the shard to "packed" state.
> >> >
> >> > Most of the run time is spent building the revision pack file
> >> > and does not require any synchronization with writers.
> >> >
> >> [...]
> >>
> >> > @@ -1987,10 +2005,33 @@ svn_fs_fs__pack(svn_fs_t *fs,
> >> > apr_pool_t *pool)
> >> > {
> >> > struct pack_baton pb = { 0 };
> >> > + fs_fs_data_t *ffd = fs->fsap_data;
> >> > + svn_error_t *err;
> >> > +
> >> > pb.fs = fs;
> >> > pb.notify_func = notify_func;
> >> > pb.notify_baton = notify_baton;
> >> > pb.cancel_func = cancel_func;
> >> > pb.cancel_baton = cancel_baton;
> >> > - return svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
> >> > +
> >> > + if (ffd->format >= SVN_FS_FS__MIN_PACK_LOCK_FORMAT)
> >> > + {
> >> > + /* Newer repositories provide a pack operation specific lock.
> >> > + Acquire it to prevent concurrent packs. */
> >> > + apr_pool_t *subpool = svn_pool_create(pool);
> >> What is the reason for using subpool here? Could you please document
> >> it if any, otherwise it looks very confusing.
> >
> >
> > Done in r1576427.
> >
> Thanks! It's clearer now. But I have suggestions for this code:
> 1. It seems mutex is missed for POSIX platform where file lock is
> per-process, not per-thread. Like we
> fs_fs_shared_data_t->fs_write_lock
>
You are right. Although it is unlikely for an application
to do attempt multi-threaded packs ... ;)
> 2. I think the code will be clearer with explicit
> svn_fs_fs__with_pack_lock() function. In this case mutexes and subpool
> will be abstracted from pack logic, which is good imho.
>
Both fixed in r1578176.
-- Stefan^2.
Re: svn commit: r1575418 - in /subversion/trunk/subversion/libsvn_fs_fs:
fs.h fs_fs.c fs_fs.h pack.c structure
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 11 March 2014 21:56, Stefan Fuhrmann <st...@wandisco.com> wrote:
> On Tue, Mar 11, 2014 at 1:23 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>
>> On 8 March 2014 01:35, <st...@apache.org> wrote:
>> > Author: stefan2
>> > Date: Fri Mar 7 21:35:54 2014
>> > New Revision: 1575418
>> >
>> > URL: http://svn.apache.org/r1575418
>> > Log:
>> > Enable FSFS format 7 repositories to be packed without completely
>> > blocking commits. We simply introduce a separate lock file for
>> > 'svnadmin pack' and take out the global write lock for packing
>> > revprops and switching the shard to "packed" state.
>> >
>> > Most of the run time is spent building the revision pack file
>> > and does not require any synchronization with writers.
>> >
>> [...]
>>
>> > @@ -1987,10 +2005,33 @@ svn_fs_fs__pack(svn_fs_t *fs,
>> > apr_pool_t *pool)
>> > {
>> > struct pack_baton pb = { 0 };
>> > + fs_fs_data_t *ffd = fs->fsap_data;
>> > + svn_error_t *err;
>> > +
>> > pb.fs = fs;
>> > pb.notify_func = notify_func;
>> > pb.notify_baton = notify_baton;
>> > pb.cancel_func = cancel_func;
>> > pb.cancel_baton = cancel_baton;
>> > - return svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
>> > +
>> > + if (ffd->format >= SVN_FS_FS__MIN_PACK_LOCK_FORMAT)
>> > + {
>> > + /* Newer repositories provide a pack operation specific lock.
>> > + Acquire it to prevent concurrent packs. */
>> > + apr_pool_t *subpool = svn_pool_create(pool);
>> What is the reason for using subpool here? Could you please document
>> it if any, otherwise it looks very confusing.
>
>
> Done in r1576427.
>
Thanks! It's clearer now. But I have suggestions for this code:
1. It seems mutex is missed for POSIX platform where file lock is
per-process, not per-thread. Like we
fs_fs_shared_data_t->fs_write_lock
2. I think the code will be clearer with explicit
svn_fs_fs__with_pack_lock() function. In this case mutexes and subpool
will be abstracted from pack logic, which is good imho.
--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com
Re: svn commit: r1575418 - in /subversion/trunk/subversion/libsvn_fs_fs:
fs.h fs_fs.c fs_fs.h pack.c structure
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Tue, Mar 11, 2014 at 1:23 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On 8 March 2014 01:35, <st...@apache.org> wrote:
> > Author: stefan2
> > Date: Fri Mar 7 21:35:54 2014
> > New Revision: 1575418
> >
> > URL: http://svn.apache.org/r1575418
> > Log:
> > Enable FSFS format 7 repositories to be packed without completely
> > blocking commits. We simply introduce a separate lock file for
> > 'svnadmin pack' and take out the global write lock for packing
> > revprops and switching the shard to "packed" state.
> >
> > Most of the run time is spent building the revision pack file
> > and does not require any synchronization with writers.
> >
> [...]
>
> > @@ -1987,10 +2005,33 @@ svn_fs_fs__pack(svn_fs_t *fs,
> > apr_pool_t *pool)
> > {
> > struct pack_baton pb = { 0 };
> > + fs_fs_data_t *ffd = fs->fsap_data;
> > + svn_error_t *err;
> > +
> > pb.fs = fs;
> > pb.notify_func = notify_func;
> > pb.notify_baton = notify_baton;
> > pb.cancel_func = cancel_func;
> > pb.cancel_baton = cancel_baton;
> > - return svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
> > +
> > + if (ffd->format >= SVN_FS_FS__MIN_PACK_LOCK_FORMAT)
> > + {
> > + /* Newer repositories provide a pack operation specific lock.
> > + Acquire it to prevent concurrent packs. */
> > + apr_pool_t *subpool = svn_pool_create(pool);
> What is the reason for using subpool here? Could you please document
> it if any, otherwise it looks very confusing.
>
Done in r1576427.
> > + const char *lock_path = svn_dirent_join(fs->path,
> PATH_PACK_LOCK_FILE,
> > + subpool);
> > + err = svn_fs_fs__get_lock_on_filesystem(lock_path, subpool);
> > + if (!err)
> > + err = pack_body(&pb, subpool);
> > +
> > + svn_pool_destroy(subpool);
> > + }
> > + else
> > + {
> > + /* Use the global write lock for older repos. */
> > + err = svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
> > + }
> > +
> > + return svn_error_trace(err);
> > }
>
Thanks for the feedback!
-- Stefan^2.
Re: svn commit: r1575418 - in /subversion/trunk/subversion/libsvn_fs_fs:
fs.h fs_fs.c fs_fs.h pack.c structure
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 8 March 2014 01:35, <st...@apache.org> wrote:
> Author: stefan2
> Date: Fri Mar 7 21:35:54 2014
> New Revision: 1575418
>
> URL: http://svn.apache.org/r1575418
> Log:
> Enable FSFS format 7 repositories to be packed without completely
> blocking commits. We simply introduce a separate lock file for
> 'svnadmin pack' and take out the global write lock for packing
> revprops and switching the shard to "packed" state.
>
> Most of the run time is spent building the revision pack file
> and does not require any synchronization with writers.
>
[...]
> @@ -1987,10 +2005,33 @@ svn_fs_fs__pack(svn_fs_t *fs,
> apr_pool_t *pool)
> {
> struct pack_baton pb = { 0 };
> + fs_fs_data_t *ffd = fs->fsap_data;
> + svn_error_t *err;
> +
> pb.fs = fs;
> pb.notify_func = notify_func;
> pb.notify_baton = notify_baton;
> pb.cancel_func = cancel_func;
> pb.cancel_baton = cancel_baton;
> - return svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
> +
> + if (ffd->format >= SVN_FS_FS__MIN_PACK_LOCK_FORMAT)
> + {
> + /* Newer repositories provide a pack operation specific lock.
> + Acquire it to prevent concurrent packs. */
> + apr_pool_t *subpool = svn_pool_create(pool);
What is the reason for using subpool here? Could you please document
it if any, otherwise it looks very confusing.
> + const char *lock_path = svn_dirent_join(fs->path, PATH_PACK_LOCK_FILE,
> + subpool);
> + err = svn_fs_fs__get_lock_on_filesystem(lock_path, subpool);
> + if (!err)
> + err = pack_body(&pb, subpool);
> +
> + svn_pool_destroy(subpool);
> + }
> + else
> + {
> + /* Use the global write lock for older repos. */
> + err = svn_fs_fs__with_write_lock(fs, pack_body, &pb, pool);
> + }
> +
> + return svn_error_trace(err);
> }
>
--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com