You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Stefan Fuhrmann <st...@wandisco.com> on 2012/07/06 10:32:08 UTC
Revprop packing implemented
Hi devs,
This week I had one of my "how hard can it be?" moments
and finally implemented revprop packing (did that mainly
offline). It passes all tests and seems to work pretty well.
It's design deviates from the existing revprop packing branch
in that it is more scalable and simpler to implement. Key
differences:
* Have a configurable limit (default 64k) to the pack file size.
Concatenate revprops only up to that limit, but at least one
revision's revprops per file. Have the manifest map revs to
revprop pack files. IOW, let the OS do all the heavy lifting
of storage management.
* Make revprop packing a mandatory part of FSFS packing
in format 6 repos. There is no need to track revprop packing
separately nor to give it a separate format info.
Since the new code will not be used unless you create
a format 6 repo, I'd like to commit everything directly to
/trunk for review instead of creating a new branch or
"overwriting" the existing one.
This is the order in which I want to commit the changes:
* refactor existing code
* update the design file
* add the revprop pack support
* add tests; write more tests
* bump the FSFS format
Any objections?
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Mon, Jul 9, 2012 at 11:40 AM, Philip Martin
<ph...@wandisco.com>wrote:
> Stefan Fuhrmann <st...@wandisco.com> writes:
>
> > On Mon, Jul 9, 2012 at 9:19 AM, Philip Martin <
> philip.martin@wandisco.com>wrote:
> >
> >> Stefan Fuhrmann <st...@wandisco.com> writes:
> >>
> >> > This week I had one of my "how hard can it be?" moments
> >> > and finally implemented revprop packing (did that mainly
> >> > offline). It passes all tests and seems to work pretty well.
> >>
> >> There are buildbot failures, I don't know if they are caused by the
> >> packing code. valgrind gives me:
> >>
> >> The build bot failures are probably not caused by this.
> >
> >
> >> $ valgrind -q .libs/lt-fs-pack-test 1
> >> ==23822== Conditional jump or move depends on uninitialised value(s)
> >> ==23822== at 0x524F066: read_config (fs_fs.c:1118)
> >> ==23822== by 0x5264B15: pack_body (fs_fs.c:9571)
> >> ==23822== by 0x524DF5F: with_some_lock_file (fs_fs.c:614)
> >> ==23822== by 0x524E035: svn_fs_fs__with_write_lock (fs_fs.c:632)
> >> ==23822== by 0x5264F35: svn_fs_fs__pack (fs_fs.c:9636)
> >> ==23822== by 0x524CAA4: fs_pack (fs.c:273)
> >> ==23822== by 0x5039291: svn_fs_pack (fs-loader.c:526)
> >> ==23822== by 0x4020AA: create_packed_filesystem (fs-pack-test.c:200)
> >> ==23822== by 0x4024C5: pack_filesystem (fs-pack-test.c:290)
> >> ==23822== by 0x4E2EBDF: do_test_num (svn_test_main.c:265)
> >> ==23822== by 0x4E2F5E2: main (svn_test_main.c:525)
> >> ==23822==
> >>
> >> r1358977 fixes that.
> > Thanks for the feedback!
>
> This one:
>
> http://ci.apache.org/builders/bb-openbsd/builds/1612
>
> failed:
>
> FAIL: fs-pack-test 5: get/set revprop while packing FSFS filesystem
> FAIL: fs-pack-test 6: get/set large packed revprops in FSFS
>
> Perhaps r1358779 fixes it?
>
Yes, it should.
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Philip Martin <ph...@wandisco.com>.
Stefan Fuhrmann <st...@wandisco.com> writes:
> On Mon, Jul 9, 2012 at 9:19 AM, Philip Martin <ph...@wandisco.com>wrote:
>
>> Stefan Fuhrmann <st...@wandisco.com> writes:
>>
>> > This week I had one of my "how hard can it be?" moments
>> > and finally implemented revprop packing (did that mainly
>> > offline). It passes all tests and seems to work pretty well.
>>
>> There are buildbot failures, I don't know if they are caused by the
>> packing code. valgrind gives me:
>>
>> The build bot failures are probably not caused by this.
>
>
>> $ valgrind -q .libs/lt-fs-pack-test 1
>> ==23822== Conditional jump or move depends on uninitialised value(s)
>> ==23822== at 0x524F066: read_config (fs_fs.c:1118)
>> ==23822== by 0x5264B15: pack_body (fs_fs.c:9571)
>> ==23822== by 0x524DF5F: with_some_lock_file (fs_fs.c:614)
>> ==23822== by 0x524E035: svn_fs_fs__with_write_lock (fs_fs.c:632)
>> ==23822== by 0x5264F35: svn_fs_fs__pack (fs_fs.c:9636)
>> ==23822== by 0x524CAA4: fs_pack (fs.c:273)
>> ==23822== by 0x5039291: svn_fs_pack (fs-loader.c:526)
>> ==23822== by 0x4020AA: create_packed_filesystem (fs-pack-test.c:200)
>> ==23822== by 0x4024C5: pack_filesystem (fs-pack-test.c:290)
>> ==23822== by 0x4E2EBDF: do_test_num (svn_test_main.c:265)
>> ==23822== by 0x4E2F5E2: main (svn_test_main.c:525)
>> ==23822==
>>
>> r1358977 fixes that.
> Thanks for the feedback!
This one:
http://ci.apache.org/builders/bb-openbsd/builds/1612
failed:
FAIL: fs-pack-test 5: get/set revprop while packing FSFS filesystem
FAIL: fs-pack-test 6: get/set large packed revprops in FSFS
Perhaps r1358779 fixes it?
--
Cerified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Mon, Jul 9, 2012 at 9:19 AM, Philip Martin <ph...@wandisco.com>wrote:
> Stefan Fuhrmann <st...@wandisco.com> writes:
>
> > This week I had one of my "how hard can it be?" moments
> > and finally implemented revprop packing (did that mainly
> > offline). It passes all tests and seems to work pretty well.
>
> There are buildbot failures, I don't know if they are caused by the
> packing code. valgrind gives me:
>
> The build bot failures are probably not caused by this.
> $ valgrind -q .libs/lt-fs-pack-test 1
> ==23822== Conditional jump or move depends on uninitialised value(s)
> ==23822== at 0x524F066: read_config (fs_fs.c:1118)
> ==23822== by 0x5264B15: pack_body (fs_fs.c:9571)
> ==23822== by 0x524DF5F: with_some_lock_file (fs_fs.c:614)
> ==23822== by 0x524E035: svn_fs_fs__with_write_lock (fs_fs.c:632)
> ==23822== by 0x5264F35: svn_fs_fs__pack (fs_fs.c:9636)
> ==23822== by 0x524CAA4: fs_pack (fs.c:273)
> ==23822== by 0x5039291: svn_fs_pack (fs-loader.c:526)
> ==23822== by 0x4020AA: create_packed_filesystem (fs-pack-test.c:200)
> ==23822== by 0x4024C5: pack_filesystem (fs-pack-test.c:290)
> ==23822== by 0x4E2EBDF: do_test_num (svn_test_main.c:265)
> ==23822== by 0x4E2F5E2: main (svn_test_main.c:525)
> ==23822==
>
> r1358977 fixes that.
Thanks for the feedback!
-- Stefan^2.
> --
> Cerified & Supported Apache Subversion Downloads:
> http://www.wandisco.com/subversion/download
>
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Philip Martin <ph...@wandisco.com>.
Stefan Fuhrmann <st...@wandisco.com> writes:
> This week I had one of my "how hard can it be?" moments
> and finally implemented revprop packing (did that mainly
> offline). It passes all tests and seems to work pretty well.
There are buildbot failures, I don't know if they are caused by the
packing code. valgrind gives me:
$ valgrind -q .libs/lt-fs-pack-test 1
==23822== Conditional jump or move depends on uninitialised value(s)
==23822== at 0x524F066: read_config (fs_fs.c:1118)
==23822== by 0x5264B15: pack_body (fs_fs.c:9571)
==23822== by 0x524DF5F: with_some_lock_file (fs_fs.c:614)
==23822== by 0x524E035: svn_fs_fs__with_write_lock (fs_fs.c:632)
==23822== by 0x5264F35: svn_fs_fs__pack (fs_fs.c:9636)
==23822== by 0x524CAA4: fs_pack (fs.c:273)
==23822== by 0x5039291: svn_fs_pack (fs-loader.c:526)
==23822== by 0x4020AA: create_packed_filesystem (fs-pack-test.c:200)
==23822== by 0x4024C5: pack_filesystem (fs-pack-test.c:290)
==23822== by 0x4E2EBDF: do_test_num (svn_test_main.c:265)
==23822== by 0x4E2F5E2: main (svn_test_main.c:525)
==23822==
--
Cerified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Fwd: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Sun, Jul 8, 2012 at 11:10 AM, Bert Huijben <be...@qqmail.nl> wrote:
> Hi Stefan,****
>
> ** **
>
> Why do you use a default of 64k for a revprop pack?
>
In the Apache repo, the average revprop list is <250 bytes / revision.
So, 64k is already large enough to reduce the number of files significantly.
OTOH, in many storage backends, 64k are the smallest amount of
data you can reasonably read (e.g. RAID stripes use 64k or 128k
blocks per disk).
> ****
>
> Isn’t something like 1MB a more sensible default? I would guess that for
> the 1000 revisions we usually pack that would fit in a single pack, while
> with 64K you only can get a few tens/hundreds revisions in a pack,
> depending on how large the log messages usually are.****
>
> ** **
>
> Does such a small pack size have specific benefits?
>
The small default block size minimizes the I/O and processing
overhead when accessing a single revprop: It is about twice as
expensive (manifest + pack file) as reading a non-packed revprop.
If you know that you rarely write to packed revprops *and* if you
enable revprop caching, you should enable compression for
packed revprops. Typical compression rate is 1:4 and the
default pack size limit is raised to 256k. You may choose an
even larger value.
The key is that you need revprop caching in that case or checkouts,
list -v and logs will become very slow.
-- Stefan^2.
> *From:* Stefan Fuhrmann [mailto:stefan.fuhrmann@wandisco.com]
> *Sent:* vrijdag 6 juli 2012 10:32
> *To:* Subversion Development
> *Subject:* Revprop packing implemented****
>
> ** **
>
> Hi devs,
>
> This week I had one of my "how hard can it be?" moments
> and finally implemented revprop packing (did that mainly
> offline). It passes all tests and seems to work pretty well.
>
> It's design deviates from the existing revprop packing branch
> in that it is more scalable and simpler to implement. Key
> differences:
>
> * Have a configurable limit (default 64k) to the pack file size.
> Concatenate revprops only up to that limit, but at least one
> revision's revprops per file. Have the manifest map revs to
> revprop pack files. IOW, let the OS do all the heavy lifting
> of storage management.
> * Make revprop packing a mandatory part of FSFS packing
> in format 6 repos. There is no need to track revprop packing
> separately nor to give it a separate format info.
>
> Since the new code will not be used unless you create
> a format 6 repo, I'd like to commit everything directly to
> /trunk for review instead of creating a new branch or
> "overwriting" the existing one.
>
> This is the order in which I want to commit the changes:
>
> * refactor existing code
> * update the design file
> * add the revprop pack support
> * add tests; write more tests
> * bump the FSFS format
>
> Any objections?
>
> -- Stefan^2.
>
> --
> Certified & Supported Apache Subversion Downloads:
> http://www.wandisco.com/subversion/download****
>
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
RE: Revprop packing implemented
Posted by Bert Huijben <be...@qqmail.nl>.
Hi Stefan,
Why do you use a default of 64k for a revprop pack?
Isn't something like 1MB a more sensible default? I would guess that for the
1000 revisions we usually pack that would fit in a single pack, while with
64K you only can get a few tens/hundreds revisions in a pack, depending on
how large the log messages usually are.
Does such a small pack size have specific benefits?
Bert
From: Stefan Fuhrmann [mailto:stefan.fuhrmann@wandisco.com]
Sent: vrijdag 6 juli 2012 10:32
To: Subversion Development
Subject: Revprop packing implemented
Hi devs,
This week I had one of my "how hard can it be?" moments
and finally implemented revprop packing (did that mainly
offline). It passes all tests and seems to work pretty well.
It's design deviates from the existing revprop packing branch
in that it is more scalable and simpler to implement. Key
differences:
* Have a configurable limit (default 64k) to the pack file size.
Concatenate revprops only up to that limit, but at least one
revision's revprops per file. Have the manifest map revs to
revprop pack files. IOW, let the OS do all the heavy lifting
of storage management.
* Make revprop packing a mandatory part of FSFS packing
in format 6 repos. There is no need to track revprop packing
separately nor to give it a separate format info.
Since the new code will not be used unless you create
a format 6 repo, I'd like to commit everything directly to
/trunk for review instead of creating a new branch or
"overwriting" the existing one.
This is the order in which I want to commit the changes:
* refactor existing code
* update the design file
* add the revprop pack support
* add tests; write more tests
* bump the FSFS format
Any objections?
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Fwd: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Fri, Jul 6, 2012 at 6:50 PM, Blair Zajac <bl...@orcaware.com> wrote:
> On 7/6/12 6:27 AM, C. Michael Pilato wrote:
>
>> On 07/06/2012 04:32 AM, Stefan Fuhrmann wrote:
>>
>>> Hi devs,
>>>
>>> This week I had one of my "how hard can it be?" moments
>>> and finally implemented revprop packing (did that mainly
>>> offline). It passes all tests and seems to work pretty well.
>>>
>>
>> Cool!
>>
>> [...]
>>
>> Since the new code will not be used unless you create
>>> a format 6 repo, I'd like to commit everything directly to
>>> /trunk for review instead of creating a new branch or
>>> "overwriting" the existing one.
>>>
>>> This is the order in which I want to commit the changes:
>>>
>>> * refactor existing code
>>> * update the design file
>>> * add the revprop pack support
>>> * add tests; write more tests
>>>
>>
>> Should you transpose these last two? Put the tests in place first, then
>> the
>> code? Or are these tests of the particular low-level feature behavior
>> that
>> mean nothing when the feature code isn't in place?
>>
>> You are right. As it turned out, we needed only one more
test case and a minor extension to another.
Of course, there is always room for more tests but I had
expected that we need more even for the basic functionality.
In that case, writing all those tests while having people
review the code seemed like a better strategy.
> * bump the FSFS format
>>>
>>> Any objections?
>>>
>>
>> I have none, so long as trunk remains "stable" to the degree we've
>> defined it.
>>
>
> +1. This is great. My 50,000,000 million revisions thank you!
>
> svnadmin upgrade should now make your repo incompatible
to anything we released ;)
If you are doing experiments, please have a look at the new
revprop packing related options in fsfs.conf.
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Blair Zajac <bl...@orcaware.com>.
On 7/6/12 6:27 AM, C. Michael Pilato wrote:
> On 07/06/2012 04:32 AM, Stefan Fuhrmann wrote:
>> Hi devs,
>>
>> This week I had one of my "how hard can it be?" moments
>> and finally implemented revprop packing (did that mainly
>> offline). It passes all tests and seems to work pretty well.
>
> Cool!
>
> [...]
>
>> Since the new code will not be used unless you create
>> a format 6 repo, I'd like to commit everything directly to
>> /trunk for review instead of creating a new branch or
>> "overwriting" the existing one.
>>
>> This is the order in which I want to commit the changes:
>>
>> * refactor existing code
>> * update the design file
>> * add the revprop pack support
>> * add tests; write more tests
>
> Should you transpose these last two? Put the tests in place first, then the
> code? Or are these tests of the particular low-level feature behavior that
> mean nothing when the feature code isn't in place?
>
>> * bump the FSFS format
>>
>> Any objections?
>
> I have none, so long as trunk remains "stable" to the degree we've defined it.
+1. This is great. My 50,000,000 million revisions thank you!
Blair
Re: Revprop packing implemented
Posted by "C. Michael Pilato" <cm...@collab.net>.
On 07/06/2012 04:32 AM, Stefan Fuhrmann wrote:
> Hi devs,
>
> This week I had one of my "how hard can it be?" moments
> and finally implemented revprop packing (did that mainly
> offline). It passes all tests and seems to work pretty well.
Cool!
[...]
> Since the new code will not be used unless you create
> a format 6 repo, I'd like to commit everything directly to
> /trunk for review instead of creating a new branch or
> "overwriting" the existing one.
>
> This is the order in which I want to commit the changes:
>
> * refactor existing code
> * update the design file
> * add the revprop pack support
> * add tests; write more tests
Should you transpose these last two? Put the tests in place first, then the
code? Or are these tests of the particular low-level feature behavior that
mean nothing when the feature code isn't in place?
> * bump the FSFS format
>
> Any objections?
I have none, so long as trunk remains "stable" to the degree we've defined it.
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Enterprise Cloud Development
Fwd: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
Effing gmail :(
On Sun, Jul 8, 2012 at 2:28 AM, Hyrum K Wright <hy...@hyrumwright.org>wrote:
> On Fri, Jul 6, 2012 at 3:32 AM, Stefan Fuhrmann
> <st...@wandisco.com> wrote:
> > Hi devs,
> >
> > This week I had one of my "how hard can it be?" moments
> > and finally implemented revprop packing (did that mainly
> > offline). It passes all tests and seems to work pretty well.
> >
> > It's design deviates from the existing revprop packing branch
> > in that it is more scalable and simpler to implement. Key
> > differences:
> >
> > * Have a configurable limit (default 64k) to the pack file size.
> > Concatenate revprops only up to that limit, but at least one
> > revision's revprops per file. Have the manifest map revs to
> > revprop pack files. IOW, let the OS do all the heavy lifting
> > of storage management.
> > * Make revprop packing a mandatory part of FSFS packing
> > in format 6 repos. There is no need to track revprop packing
> > separately nor to give it a separate format info.
>
> Do you have an overview of the entire design, rather than just a delta
> between your implementation and the existing one?
>
> subversion/libsvn_fs_fs/structure explains more of it.
Beyond that, there is no specific design. The core ideas are
* limited pack file size
* separate manifest file & folders as with revision data packing
* single-file-move-into-place semantics when writing revprops
(as today with non-packed revprops)
I.e. we combine proven parts of revision data packing and
non-packed revprop handling. Everything else is just "data
processing".
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Hyrum K Wright <hy...@hyrumwright.org>.
On Fri, Jul 6, 2012 at 3:32 AM, Stefan Fuhrmann
<st...@wandisco.com> wrote:
> Hi devs,
>
> This week I had one of my "how hard can it be?" moments
> and finally implemented revprop packing (did that mainly
> offline). It passes all tests and seems to work pretty well.
>
> It's design deviates from the existing revprop packing branch
> in that it is more scalable and simpler to implement. Key
> differences:
>
> * Have a configurable limit (default 64k) to the pack file size.
> Concatenate revprops only up to that limit, but at least one
> revision's revprops per file. Have the manifest map revs to
> revprop pack files. IOW, let the OS do all the heavy lifting
> of storage management.
> * Make revprop packing a mandatory part of FSFS packing
> in format 6 repos. There is no need to track revprop packing
> separately nor to give it a separate format info.
Do you have an overview of the entire design, rather than just a delta
between your implementation and the existing one?
> ...
-Hyrum
Re: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Tue, Jul 10, 2012 at 3:31 PM, Markus Schaber
<m....@3s-software.com>wrote:
> Hi, Stefan,
>
> Von: Stefan Fuhrmann [mailto:stefan.fuhrmann@wandisco.com]
> >> What's the performance penalty for modifying a packed rev props? From
> what you've said, it sounds like packing works by fitting as many rev props
> into a 64k chunk, so, let's say I've got a million rev repository. And the
> first 1000 revs fit neatly inside the first 64k pack. What happens when I
> propedit r1000 and increase its size so that it no longer neatly fits
> inside the 64k pack?
>
> > That one pack file gets split. I just committed a change that will
> result in two roughly equally sized pack files after the split - if the
> revprop sizes allow for it.
>
> Just to be sure: Did you catch the corner case where a revision X in the
> middle of the chunk gets a revprop that exceeds the 64k pack limit, and the
> code needs to split the Chunk into three (pre-X, X, post-X)?
>
Yes. There is even a test case for that. It tests "huge" revprops at the
begining,
in the middle and at the end of the pack file.
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
AW: Revprop packing implemented
Posted by Markus Schaber <m....@3s-software.com>.
Hi, Stefan,
Von: Stefan Fuhrmann [mailto:stefan.fuhrmann@wandisco.com]
>> What's the performance penalty for modifying a packed rev props? From what you've said, it sounds like packing works by fitting as many rev props into a 64k chunk, so, let's say I've got a million rev repository. And the first 1000 revs fit neatly inside the first 64k pack. What happens when I propedit r1000 and increase its size so that it no longer neatly fits inside the 64k pack?
> That one pack file gets split. I just committed a change that will result in two roughly equally sized pack files after the split - if the revprop sizes allow for it.
Just to be sure: Did you catch the corner case where a revision X in the middle of the chunk gets a revprop that exceeds the 64k pack limit, and the code needs to split the Chunk into three (pre-X, X, post-X)?
Best regards
Markus Schaber
--
___________________________
We software Automation.
3S-Smart Software Solutions GmbH
Markus Schaber | Developer
Memminger Str. 151 | 87439 Kempten | Germany | Tel. +49-831-54031-0 | Fax +49-831-54031-50
Email: m.schaber@3s-software.com | Web: http://www.3s-software.com
CoDeSys internet forum: http://forum.3s-software.com
Download CoDeSys sample projects: http://www.3s-software.com/index.shtml?sample_projects
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
Re: Revprop packing implemented
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Tue, Jul 10, 2012 at 8:09 AM, Trent Nelson <tr...@snakebite.org> wrote:
> Hi Stefan,
>
>
> > This week I had one of my "how hard can it be?" moments
> > and finally implemented revprop packing
>
> Lots of tools, like svnsync, store metadata in r0 rev props. Could this
> rev be excluded from packing?
>
It is excluded already.
What's the performance penalty for modifying a packed rev props? From
> what you've said, it sounds like packing works by fitting as many rev
> props into a 64k chunk, so, let's say I've got a million rev repository.
> And the first 1000 revs fit neatly inside the first 64k pack. What
> happens when I propedit r1000 and increase its size so that it no longer
> neatly fits inside the 64k pack?
>
That one pack file gets split. I just committed a change
that will result in two roughly equally sized pack files
after the split - if the revprop sizes allow for it.
(Just checking that it won't cause a knock-on re-write effect of all the
> other 999,000 packed rev props. Sounds like the manifest map takes care
> of avoiding that, but just wanted to make sure.)
>
> And, heh, this is going to kill Enversion, which currently writes
> extensive root metadata to each revision's rev prop as it analyzes a
> repository.
The I/O will roughly be the same as for 1.7. If you are
writing >30k per revision, you will end up with basically
the same file structure as before (1 pack file per rev)
plus one manifest file per shard.
Will there be a way (even if it's at the API level, not CLI)
> to disable rev prop packing?
>
No - unless you can demonstrate that the performance
impact is significant.
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Revprop packing implemented
Posted by Trent Nelson <tr...@snakebite.org>.
Hi Stefan,
> This week I had one of my "how hard can it be?" moments
> and finally implemented revprop packing
Lots of tools, like svnsync, store metadata in r0 rev props. Could this
rev be excluded from packing?
What's the performance penalty for modifying a packed rev props? From
what you've said, it sounds like packing works by fitting as many rev
props into a 64k chunk, so, let's say I've got a million rev repository.
And the first 1000 revs fit neatly inside the first 64k pack. What
happens when I propedit r1000 and increase its size so that it no longer
neatly fits inside the 64k pack?
(Just checking that it won't cause a knock-on re-write effect of all the
other 999,000 packed rev props. Sounds like the manifest map takes care
of avoiding that, but just wanted to make sure.)
And, heh, this is going to kill Enversion, which currently writes
extensive root metadata to each revision's rev prop as it analyzes a
repository. Will there be a way (even if it's at the API level, not CLI)
to disable rev prop packing?
Trent.