You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by da...@apache.org on 2011/07/07 16:21:08 UTC
svn commit: r1143841 - in /subversion/branches/revprop-packing:
BRANCH-README subversion/libsvn_fs_fs/structure
Author: danielsh
Date: Thu Jul 7 14:21:08 2011
New Revision: 1143841
URL: http://svn.apache.org/viewvc?rev=1143841&view=rev
Log:
On the 'revprop-sharding' branch:
Begin documenting the design plan.
* BRANCH-README: Point to 'structure'.
* subversion/libsvn_fs_fs/structure:
Document the directory and file structure.
TBD: locking, readers/writers algorithm.
Modified:
subversion/branches/revprop-packing/BRANCH-README
subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure
Modified: subversion/branches/revprop-packing/BRANCH-README
URL: http://svn.apache.org/viewvc/subversion/branches/revprop-packing/BRANCH-README?rev=1143841&r1=1143840&r2=1143841&view=diff
==============================================================================
--- subversion/branches/revprop-packing/BRANCH-README (original)
+++ subversion/branches/revprop-packing/BRANCH-README Thu Jul 7 14:21:08 2011
@@ -1,3 +1,9 @@
-This branch exists to reimplement the revprop-packing without SQLite. Several
-strategies have been discussed on the mailing list here:
+This branch exists to reimplement the revprop-packing in FSFS f5 without
+SQLite. Several strategies have been discussed on the mailing list here:
http://svn.haxx.se/dev/archive-2011-07/0134.shtml
+
+The concrete design proposal has been patched into the 'structure' file; you
+may review it via
+
+ svn diff ^/subversion/trunk/subversion/libsvn_fs_fs/structure@r1143816 \
+ ^/subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure
Modified: subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure
URL: http://svn.apache.org/viewvc/subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure?rev=1143841&r1=1143840&r2=1143841&view=diff
==============================================================================
--- subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure (original)
+++ subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure Thu Jul 7 14:21:08 2011
@@ -40,7 +40,9 @@ repository) is:
revprops/ Subdirectory containing rev-props
<shard>/ Shard directory, if sharding is in use (see below)
<revnum> File containing rev-props for <revnum>
- revprops.db SQLite database of the packed revision properties
+ <shard>.revpack/ Pack directory, if the repo has been packed (see below)
+ revpack Revprops pack file, contains inline manifest (see below)
+ lock Revprops shard lock file (see below)
transactions/ Subdirectory containing transactions
<txnid>.txn/ Directory containing transaction <txnid>
txn-protorevs/ Subdirectory containing transaction proto-revision files
@@ -63,8 +65,8 @@ repository) is:
min-unpacked-revprop File containing the oldest revision of unpacked revprop
rep-cache.db SQLite database mapping rep checksums to locations
-Files in the revprops directory are in the hash dump format used by
-svn_hash_write.
+Non-packed files in the revprops directory are in the hash dump format used by
+svn_hash_write.
The format of the "current" file is:
@@ -171,6 +173,10 @@ Revision changed paths list:
Format 1-3: Does not contain the node's kind.
Format 4-5: Contains the node's kind.
+Revprop packing:
+ Format 1-4: revprops are not packed
+ Format 5: revprops are packed
+
# Incomplete list. See SVN_FS_FS__MIN_*_FORMAT
@@ -225,13 +231,27 @@ Packing revision properties
---------------------------
Revprop shards may be packed too. The packing process moves all revprops from
-a the oldest unpacked shard to the SQLite database "revprops/revprops.db". The
-sequel documents the schema of FSFS format 5, which corresponds to schema
-version USER_VERSION=1:
-
-The database maps revision numbers to serialized "proplist" skels. Every
-packed revision must have a mapping entry (a table row), even if it maps the
-revnum to an empty skel.
+each completed shard S to a single pack file. As a special-case, r0 revprops
+are never packed, in consideration of performance of 'svnsync' and other
+consumers that propedit r0.
+
+The pack+manifest file consists of:
+
+* A 32-bit sequence number
+* A list of offsets, one for each revision's hunk in the pack file.
+* The concatenated hunks for all revisions in the shard.
+
+The sequence number and the offsets are stored as ASCII decimal, followed by
+newline. They are space-padded from the left such that the least-significant
+digit is on the tenth byte of the line for the sequence number, and on the
+twentieth byte of the line for the offsets. Each revprop's hunk takes the same
+format as an unpacked revprop file: namely, it is a svn_hash_write()-serialized
+revprops hash.
+
+The manifest for the zeroth shard contains the literal ASCII text "zerothshard"
+(with no following newline) in place of the offset for the zeroth revision of
+the shard.
+
Node-revision IDs
-----------------
Re: svn commit: r1143841 - in /subversion/branches/revprop-packing:
BRANCH-README subversion/libsvn_fs_fs/structure
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Jul 7, 2011 at 13:54, Peter Samuelson <pe...@p12n.org> wrote:
>
> [Daniel Shahaf]
>> The only format 5 repositories we have right now 'in the wild' are
>> 1.7-dev code. revprops.db code was never included in a release.
>
> Well unless you count the alphas. But we _do_ warn people not to
> assume anything about them, right? Also, it's not like everyone who
> downloaded our alphas necessarily ran 'svnadmin upgrade' anyway.
While we do warn them, I don't see a need to break their repositories
on them. We just don't support the format that they used. "use alphaN
to dump your repos, then load it into alphaN+1".
They may not have run 'svnadmin upgrade', but they may have created a
new repository that they'd like to keep.
>> If people prefer calling the new format f6 and having a tools/ script
>> that bumps f5 to f6, worksforme. (and by this suggestion f5 would
>> never be supported by released code)
>
> Ah, indeed, f5 has nothing else new in it except revprop packing.
> Indeed, integers are cheap: probably best to roll trunk back to f4 and
> roll the revprop-packing branch up to f6. Code to update from f5 in
> 'svnadmin upgrade' is optional.
That was my thinking, yeah.
Cheers,
-g
Re: svn commit: r1143841 - in
/subversion/branches/revprop-packing: BRANCH-README
subversion/libsvn_fs_fs/structure
Posted by Peter Samuelson <pe...@p12n.org>.
[Daniel Shahaf]
> The only format 5 repositories we have right now 'in the wild' are
> 1.7-dev code. revprops.db code was never included in a release.
Well unless you count the alphas. But we _do_ warn people not to
assume anything about them, right? Also, it's not like everyone who
downloaded our alphas necessarily ran 'svnadmin upgrade' anyway.
> If people prefer calling the new format f6 and having a tools/ script
> that bumps f5 to f6, worksforme. (and by this suggestion f5 would
> never be supported by released code)
Ah, indeed, f5 has nothing else new in it except revprop packing.
Indeed, integers are cheap: probably best to roll trunk back to f4 and
roll the revprop-packing branch up to f6. Code to update from f5 in
'svnadmin upgrade' is optional.
--
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/
Re: svn commit: r1143841 - in
/subversion/branches/revprop-packing: BRANCH-README
subversion/libsvn_fs_fs/structure
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Greg Stein wrote on Thu, Jul 07, 2011 at 13:22:40 -0400:
> On Thu, Jul 7, 2011 at 10:21, <da...@apache.org> wrote:
> >...
> > +++ subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure Thu Jul 7 14:21:08 2011
> > @@ -40,7 +40,9 @@ repository) is:
> > revprops/ Subdirectory containing rev-props
> > <shard>/ Shard directory, if sharding is in use (see below)
> > <revnum> File containing rev-props for <revnum>
> > - revprops.db SQLite database of the packed revision properties
> > + <shard>.revpack/ Pack directory, if the repo has been packed (see below)
> > + revpack Revprops pack file, contains inline manifest (see below)
> > + lock Revprops shard lock file (see below)
>
> I didn't follow the lead-up discussion to this branch, but will the
> new code automatically manage the transition from the .db file to the
> new format? IOW, we have "format 5" repositories in the wild now. Do
> we have to burn that number and move to f6? Can f5 mean "we'll see the
> old .db and delete it" ... etc
>
The only format 5 repositories we have right now 'in the wild' are
1.7-dev code. revprops.db code was never included in a release.
If people prefer calling the new format f6 and having a tools/ script
that bumps f5 to f6, worksforme. (and by this suggestion f5 would never
be supported by released code)
> >...
>
> Cheers,
> -g
Re: svn commit: r1143841 - in /subversion/branches/revprop-packing:
BRANCH-README subversion/libsvn_fs_fs/structure
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Jul 7, 2011 at 10:21, <da...@apache.org> wrote:
>...
> +++ subversion/branches/revprop-packing/subversion/libsvn_fs_fs/structure Thu Jul 7 14:21:08 2011
> @@ -40,7 +40,9 @@ repository) is:
> revprops/ Subdirectory containing rev-props
> <shard>/ Shard directory, if sharding is in use (see below)
> <revnum> File containing rev-props for <revnum>
> - revprops.db SQLite database of the packed revision properties
> + <shard>.revpack/ Pack directory, if the repo has been packed (see below)
> + revpack Revprops pack file, contains inline manifest (see below)
> + lock Revprops shard lock file (see below)
I didn't follow the lead-up discussion to this branch, but will the
new code automatically manage the transition from the .db file to the
new format? IOW, we have "format 5" repositories in the wild now. Do
we have to burn that number and move to f6? Can f5 mean "we'll see the
old .db and delete it" ... etc
>...
Cheers,
-g