You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Johan Corveleyn <jc...@gmail.com> on 2016/01/27 13:08:16 UTC
Re: svn commit: r1705646 - /subversion/trunk/subversion/libsvn_fs_fs/pack.c
On Mon, Sep 28, 2015 at 12:16 PM, <st...@apache.org> wrote:
> Author: stefan2
> Date: Mon Sep 28 10:16:12 2015
> New Revision: 1705646
>
> URL: http://svn.apache.org/viewvc?rev=1705646&view=rev
> Log:
> Tune reorg strategy during the FSFS format7 packs such that it favors
> checkout-style tree walks now.
>
> Since r1703237, following the log history no longer requires frequent
> access to directory data but mainly relies on noderev predecessor chain.
> Therefore, it is no longer necessary to tightly pack directories in a
> separate part of the pack file. With this patch, they are now placed
> with the file contents and can be processed by a quasi-linear scan
> instead of reading from two sections per pack.
>
> * subversion/libsvn_fs_fs/pack.c
> (compare_dir_entries_format7): Adapt reporting strategy - process dirs
> at the same time as files now.
> (compare_is_dir): No longer needed.
> (sort_reps): No longer distinguish between file and dir reps but only
> paths and delta chains when determining reprentation order.
Hi Stefan,
In this post-1.9.x-branch commit (and a couple of subsequent
pack-related commits) you changed the pack layout for FSFS format7, to
make it more efficient for exports, checkouts, ...
I'm wondering, when 1.10 comes out, how will I be able to benefit from
this improved pack layout? Supposing I've already dump/load-ed with
1.9 in FSFS7, and packed it.
Does this require a format bump (and another dump/load)? Or do you
plan on providing a way to "repack" (or simply unpack and pack) a
packed repository? Currently there is no unpack or repack command ...
--
Johan
Re: svn commit: r1705646 -
/subversion/trunk/subversion/libsvn_fs_fs/pack.c
Posted by Stefan Fuhrmann <st...@apache.org>.
On 28.01.2016 01:10, Johan Corveleyn wrote:
> On Wed, Jan 27, 2016 at 1:54 PM, Stefan Fuhrmann <eq...@web.de> wrote:
>> On 27.01.2016 13:08, Johan Corveleyn wrote:
>>>
>>> Does this require a format bump (and another dump/load)? Or do you
>>> plan on providing a way to "repack" (or simply unpack and pack) a
>>> packed repository? Currently there is no unpack or repack command ...
>>
>> This is no format change; the beauty of format 7 is
>> that we can learn and tweak strategies as we go without
>> a format bump. OTOH, there is not much potential for
>> further improvement right now. FSFS is limited by its
>> fixed sharing scheme.
>>
>> I have no *plans* for unpack/repack but the thought
>> has occurred to me. It should not be hard to do as
>> an offline operation, maybe alongside a reshard command.
>> If you want to write an 'svnadmin unpack', I'd support
>> you in that effort.
> Interesting :-). I've put it on my radar, but not sure if I'll ever
> get to it. Life keeps getting in the way ;-).
>
> Would be a nice bite-sized task though ... if someone else wants to
> have a go, feel free. Maybe it's something to put on the
> http://subversion.apache.org/ideas.html page?
>
Good idea. Done in r1727706.
-- Stefan^2.
Re: svn commit: r1705646 - /subversion/trunk/subversion/libsvn_fs_fs/pack.c
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Jan 27, 2016 at 1:54 PM, Stefan Fuhrmann <eq...@web.de> wrote:
> On 27.01.2016 13:08, Johan Corveleyn wrote:
>>
>> On Mon, Sep 28, 2015 at 12:16 PM, <st...@apache.org> wrote:
>>>
>>> Author: stefan2
>>> Date: Mon Sep 28 10:16:12 2015
>>> New Revision: 1705646
>>>
>>> URL: http://svn.apache.org/viewvc?rev=1705646&view=rev
>>> Log:
>>> Tune reorg strategy during the FSFS format7 packs such that it favors
>>> checkout-style tree walks now.
>>>
>>> Since r1703237, following the log history no longer requires frequent
>>> access to directory data but mainly relies on noderev predecessor chain.
>>> Therefore, it is no longer necessary to tightly pack directories in a
>>> separate part of the pack file. With this patch, they are now placed
>>> with the file contents and can be processed by a quasi-linear scan
>>> instead of reading from two sections per pack.
>>>
>>> * subversion/libsvn_fs_fs/pack.c
>>> (compare_dir_entries_format7): Adapt reporting strategy - process dirs
>>> at the same time as files now.
>>> (compare_is_dir): No longer needed.
>>> (sort_reps): No longer distinguish between file and dir reps but only
>>> paths and delta chains when determining reprentation
>>> order.
>>
>>
>> Hi Stefan,
>>
>> In this post-1.9.x-branch commit (and a couple of subsequent
>> pack-related commits) you changed the pack layout for FSFS format7, to
>> make it more efficient for exports, checkouts, ...
>
>
> The benefit is not massive and very much depends on
> project vs. repository size. 20% or so for SVN.
>
>> I'm wondering, when 1.10 comes out, how will I be able to benefit from
>> this improved pack layout? Supposing I've already dump/load-ed with
>> 1.9 in FSFS7, and packed it.
>
>
> Future commits will use the new strategy. So, speed
> will slightly go up over time.
>
>> Does this require a format bump (and another dump/load)? Or do you
>> plan on providing a way to "repack" (or simply unpack and pack) a
>> packed repository? Currently there is no unpack or repack command ...
>
>
> This is no format change; the beauty of format 7 is
> that we can learn and tweak strategies as we go without
> a format bump. OTOH, there is not much potential for
> further improvement right now. FSFS is limited by its
> fixed sharing scheme.
>
> I have no *plans* for unpack/repack but the thought
> has occurred to me. It should not be hard to do as
> an offline operation, maybe alongside a reshard command.
> If you want to write an 'svnadmin unpack', I'd support
> you in that effort.
Interesting :-). I've put it on my radar, but not sure if I'll ever
get to it. Life keeps getting in the way ;-).
Would be a nice bite-sized task though ... if someone else wants to
have a go, feel free. Maybe it's something to put on the
http://subversion.apache.org/ideas.html page?
--
Johan
Re: svn commit: r1705646 - /subversion/trunk/subversion/libsvn_fs_fs/pack.c
Posted by Stefan Fuhrmann <eq...@web.de>.
On 27.01.2016 13:08, Johan Corveleyn wrote:
> On Mon, Sep 28, 2015 at 12:16 PM, <st...@apache.org> wrote:
>> Author: stefan2
>> Date: Mon Sep 28 10:16:12 2015
>> New Revision: 1705646
>>
>> URL: http://svn.apache.org/viewvc?rev=1705646&view=rev
>> Log:
>> Tune reorg strategy during the FSFS format7 packs such that it favors
>> checkout-style tree walks now.
>>
>> Since r1703237, following the log history no longer requires frequent
>> access to directory data but mainly relies on noderev predecessor chain.
>> Therefore, it is no longer necessary to tightly pack directories in a
>> separate part of the pack file. With this patch, they are now placed
>> with the file contents and can be processed by a quasi-linear scan
>> instead of reading from two sections per pack.
>>
>> * subversion/libsvn_fs_fs/pack.c
>> (compare_dir_entries_format7): Adapt reporting strategy - process dirs
>> at the same time as files now.
>> (compare_is_dir): No longer needed.
>> (sort_reps): No longer distinguish between file and dir reps but only
>> paths and delta chains when determining reprentation order.
>
> Hi Stefan,
>
> In this post-1.9.x-branch commit (and a couple of subsequent
> pack-related commits) you changed the pack layout for FSFS format7, to
> make it more efficient for exports, checkouts, ...
The benefit is not massive and very much depends on
project vs. repository size. 20% or so for SVN.
> I'm wondering, when 1.10 comes out, how will I be able to benefit from
> this improved pack layout? Supposing I've already dump/load-ed with
> 1.9 in FSFS7, and packed it.
Future commits will use the new strategy. So, speed
will slightly go up over time.
> Does this require a format bump (and another dump/load)? Or do you
> plan on providing a way to "repack" (or simply unpack and pack) a
> packed repository? Currently there is no unpack or repack command ...
This is no format change; the beauty of format 7 is
that we can learn and tweak strategies as we go without
a format bump. OTOH, there is not much potential for
further improvement right now. FSFS is limited by its
fixed sharing scheme.
I have no *plans* for unpack/repack but the thought
has occurred to me. It should not be hard to do as
an offline operation, maybe alongside a reshard command.
If you want to write an 'svnadmin unpack', I'd support
you in that effort.
-- Stefan^2.