You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Paul Querna <ch...@force-elite.com> on 2009/04/28 01:12:51 UTC
revprop packing for fsfs
Hello,
revprops on fsfs are currently still stored in one file per revsion,
and optionally sharded.
Previously there was a revprop-sqlite branch[1], which apparently
'worked' but was never merged to trunk[2].
Last time this was discussed was in October 2008[3], when hwright had
packed revprops, but later reverted the work as revprops are mutable
in r33724. The thread at the time included questions on the ability
to use SQLite on NFS, and if FSFS already has issues on NFS.
My question for the list: Is the design of using SQLite as the primary
data storage for revprops acceptable?
Currently the repcache uses sqlite, but it is entirely optional. In
the event of a problem with the database, at worst representations are
not shared.
If you believe SQLIte should not be used for revprops, is there an
alternative design that would accomplish the goal of not using one
file per revision for revprops?
Thanks,
Paul
[1] - http://svn.haxx.se/dev/archive-2007-06/0505.shtml
[2] - http://svn.collab.net/viewvc/svn?view=revision&revision=29147
[3] - http://svn.haxx.se/dev/archive-2008-10/0654.shtml
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1955443
Re: [patch] revprop packing for fsfs
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Haven't looked at the patches, but this is to remind people of the option
to test packing features by building with
-DPACK_AFTER_EVERY_COMMIT -DSVN_FS_FS_DEFAULT_MAX_FILES_PER_DIR=%d
or by passing
FSFS_PACKING=yes FSFS_SHARDING=%d
to 'make check'. (I recommend the former.) Normally I set %d to a small
number, e.g., 1-4.
Daniel
Paul Querna wrote on Thu, 30 Apr 2009 at 23:55 -0700:
> On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
> <hy...@mail.utexas.edu> wrote:
> > On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
> >> I'm hacking up a patch using SQLite for storing the revprops right
> >> now, hopefully be able to post it in a few days when I get some more
> >> hacking time in on it.
> >
> > How about the attached? I'm testing now, but it looks to give a ~90% space
> > improvement in revprop storage. Still a few kinks, but if you've got some
> > time to test it, I'd appreciate it.
> >
> > -Hyrum
>
>
> Thanks Hyrum!
>
> Based off your patch, new patch is attached.
>
> Revprops use the normal sharded format, until a pack is run, at which
> point they are combined into a single sqlite database.
>
> Thoughts?
>
> Thanks,
>
> Paul
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2012225
RE: Re: [patch] revprop packing for fsfs
Posted by Andy Bolstridge <an...@bolstridge.plus.com>.
No, that's s much as I expected to hear, I just get worried sometimes when people say "scrap x, replace with y".
eg. http://stackoverflow.com/questions/398802/using-mssql-backend-for-subversioning-repositories
or
http://stackoverflow.com/questions/777912/subversion-over-sql-server
When you've finished laughing (easier backups!) you see why I think incremental changes to existing code is usually preferable :) if sqlite does a lot, and replaces crufty code, then that's great. If no-one has time to implement it though... then small changes to existing is probably a better choice to make.
cheers.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2352810
Re: [patch] revprop packing for fsfs
Posted by "Hyrum K. Wright" <hy...@hyrumwright.org>.
On May 20, 2009, at 2:40 AM, webpost@tigris.org wrote:
> I won't criticise SQLite, but if there was a case for keeping the
> flat file system wouldn't a simple solution be to write modified
> revprops as a single file, and mark the revprop in the packed file
> as obsolete?
>
> Then reading a revprop reads the packed file as normal, but if the
> flag is set, read the revprop file that will be found on disk.
>
> A regular pack operation will then recombine the revprop into the
> packed file.
>
> Just my 2p, but if SQLite gives more functionality or is just faster
> - then I'm well happy with that!
Still haven't had time to review the patch, though I have time to do
some drive-by kibitzing. :)
Disks have improved vastly since Subversion debuted several years
ago. When it first came out, we spent a lot of time optimizing disk
accesses, on both the client and server. (Incidentally, this led to a
lot of mess in the working copy library, precipitating the current
rewrite known as wc-ng.)
The increase in disk performance means that we can sacrifice a little
bit of performance, for space and maintainability considerations.
Subversion should be in the business of doing version control, not
worrying about low-level file storage formats. Using a separate
library to manage the on-disk formats let's us concentrate on solving
version control problems. There isn't anything special about sqlite
per se, it's just the most convenient choice for the time being.
That's probably more than you'd like to hear, it's something I've been
thinking about recently, so it all spilled out. (And yes, Paul, I'm
still planning on reviewing the revprop-sqlite patch. Where did I put
those tuits?)
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2352669
RE: Re: [patch] revprop packing for fsfs
Posted by we...@tigris.org.
I won't criticise SQLite, but if there was a case for keeping the flat file system wouldn't a simple solution be to write modified revprops as a single file, and mark the revprop in the packed file as obsolete?
Then reading a revprop reads the packed file as normal, but if the flag is set, read the revprop file that will be found on disk.
A regular pack operation will then recombine the revprop into the packed file.
Just my 2p, but if SQLite gives more functionality or is just faster - then I'm well happy with that!
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2328521
Re: [patch] revprop packing for fsfs
Posted by "C. Michael Pilato" <cm...@collab.net>.
I've got your mail/patch marked for processing. Just digging for Round
Tuits at the moment.
Paul Querna wrote:
> bump, anything I should do to move this patch along?
>
> It seems to work for me, and passed the test cases when i ran em on my mac....
>
> Thanks,
>
> Paul
>
> On Thu, Apr 30, 2009 at 11:55 PM, Paul Querna <ch...@force-elite.com> wrote:
>> On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
>> <hy...@mail.utexas.edu> wrote:
>>> On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
>>>> I'm hacking up a patch using SQLite for storing the revprops right
>>>> now, hopefully be able to post it in a few days when I get some more
>>>> hacking time in on it.
>>> How about the attached? I'm testing now, but it looks to give a ~90% space
>>> improvement in revprop storage. Still a few kinks, but if you've got some
>>> time to test it, I'd appreciate it.
>>>
>>> -Hyrum
>>
>> Thanks Hyrum!
>>
>> Based off your patch, new patch is attached.
>>
>> Revprops use the normal sharded format, until a pack is run, at which
>> point they are combined into a single sqlite database.
>>
>> Thoughts?
>>
>> Thanks,
>>
>> Paul
>>
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2097588
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2097651
Re: [patch] revprop packing for fsfs
Posted by Paul Querna <ch...@force-elite.com>.
bump, anything I should do to move this patch along?
It seems to work for me, and passed the test cases when i ran em on my mac....
Thanks,
Paul
On Thu, Apr 30, 2009 at 11:55 PM, Paul Querna <ch...@force-elite.com> wrote:
> On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
> <hy...@mail.utexas.edu> wrote:
>> On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
>>> I'm hacking up a patch using SQLite for storing the revprops right
>>> now, hopefully be able to post it in a few days when I get some more
>>> hacking time in on it.
>>
>> How about the attached? I'm testing now, but it looks to give a ~90% space
>> improvement in revprop storage. Still a few kinks, but if you've got some
>> time to test it, I'd appreciate it.
>>
>> -Hyrum
>
>
> Thanks Hyrum!
>
> Based off your patch, new patch is attached.
>
> Revprops use the normal sharded format, until a pack is run, at which
> point they are combined into a single sqlite database.
>
> Thoughts?
>
> Thanks,
>
> Paul
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2097588
[patch] revprop packing for fsfs
Posted by Paul Querna <ch...@force-elite.com>.
On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
<hy...@mail.utexas.edu> wrote:
> On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
>> I'm hacking up a patch using SQLite for storing the revprops right
>> now, hopefully be able to post it in a few days when I get some more
>> hacking time in on it.
>
> How about the attached? I'm testing now, but it looks to give a ~90% space
> improvement in revprop storage. Still a few kinks, but if you've got some
> time to test it, I'd appreciate it.
>
> -Hyrum
Thanks Hyrum!
Based off your patch, new patch is attached.
Revprops use the normal sharded format, until a pack is run, at which
point they are combined into a single sqlite database.
Thoughts?
Thanks,
Paul
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2009616
Re: revprop packing for fsfs
Posted by "Hyrum K. Wright" <hy...@hyrumwright.org>.
On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
> On Tue, Apr 28, 2009 at 11:58 PM, Daniel Shahaf <d.s@daniel.shahaf.name
> > wrote:
>> Peter Samuelson wrote on Tue, 28 Apr 2009 at 16:50 -0500:
>>>
>>> [Paul Querna]
>>>> Last time this was discussed was in October 2008[3], when hwright
>>>> had
>>>> packed revprops, but later reverted the work as revprops are
>>>> mutable
>>>> in r33724.
>>>
>>> Is there a reason to try too hard to avoid allocation holes in the
>>> revprops file? Seems to me you could just align each revision to,
>>> say,
>>> 16 or 32 bytes, and if a propedit operation wants to make it too
>>> big to
>>> fit, you move it to the end.
>>
>>> Maybe increment a "hole counter" and if that gets to be too large
>>> a fraction of shard size, take a lock and repack the file.
>>>
>>
>> How can you do the "repacking" step atomically?
>
> And this gets to the root of the problem. Either we spend time making
> a half-complete database equivalent for storing props, or just keep it
> simple and use a library for it. (SQLite, whatever).
>
>>> This would only be problematic if some automated process were to
>>> edit a
>>> lot of revprops from comparatively old revisions. (Which I honestly
>>> don't think is a case worth optimizing for.)
>>>
>>>> My question for the list: Is the design of using SQLite as the
>>>> primary
>>>> data storage for revprops acceptable?
>>>
>>> A flat file seems preferable, unless there are serious problems with
>>> that which I haven't seen.
>>>
>
> Block re-use, and inline editing seem deceptively simple. If you don't
> need partial file editing or block reuse (ie, you could use an append
> only schema), then yeah, a flat file seems much more viable, but those
> things are just a deep hole you could avoid by using a pre-built
> library like SQLite (or BerkeleyDB for that matter).
>
> I'm hacking up a patch using SQLite for storing the revprops right
> now, hopefully be able to post it in a few days when I get some more
> hacking time in on it.
How about the attached? I'm testing now, but it looks to give a ~90%
space improvement in revprop storage. Still a few kinks, but if
you've got some time to test it, I'd appreciate it.
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1987190
Re: revprop packing for fsfs
Posted by Paul Querna <ch...@force-elite.com>.
On Tue, Apr 28, 2009 at 11:58 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Peter Samuelson wrote on Tue, 28 Apr 2009 at 16:50 -0500:
>>
>> [Paul Querna]
>> > Last time this was discussed was in October 2008[3], when hwright had
>> > packed revprops, but later reverted the work as revprops are mutable
>> > in r33724.
>>
>> Is there a reason to try too hard to avoid allocation holes in the
>> revprops file? Seems to me you could just align each revision to, say,
>> 16 or 32 bytes, and if a propedit operation wants to make it too big to
>> fit, you move it to the end.
>
>> Maybe increment a "hole counter" and if that gets to be too large
>> a fraction of shard size, take a lock and repack the file.
>>
>
> How can you do the "repacking" step atomically?
And this gets to the root of the problem. Either we spend time making
a half-complete database equivalent for storing props, or just keep it
simple and use a library for it. (SQLite, whatever).
>> This would only be problematic if some automated process were to edit a
>> lot of revprops from comparatively old revisions. (Which I honestly
>> don't think is a case worth optimizing for.)
>>
>> > My question for the list: Is the design of using SQLite as the primary
>> > data storage for revprops acceptable?
>>
>> A flat file seems preferable, unless there are serious problems with
>> that which I haven't seen.
>>
Block re-use, and inline editing seem deceptively simple. If you don't
need partial file editing or block reuse (ie, you could use an append
only schema), then yeah, a flat file seems much more viable, but those
things are just a deep hole you could avoid by using a pre-built
library like SQLite (or BerkeleyDB for that matter).
I'm hacking up a patch using SQLite for storing the revprops right
now, hopefully be able to post it in a few days when I get some more
hacking time in on it.
Thanks,
Paul
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1986358
Re: revprop packing for fsfs
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Peter Samuelson wrote on Tue, 28 Apr 2009 at 16:50 -0500:
>
> [Paul Querna]
> > Last time this was discussed was in October 2008[3], when hwright had
> > packed revprops, but later reverted the work as revprops are mutable
> > in r33724.
>
> Is there a reason to try too hard to avoid allocation holes in the
> revprops file? Seems to me you could just align each revision to, say,
> 16 or 32 bytes, and if a propedit operation wants to make it too big to
> fit, you move it to the end.
> Maybe increment a "hole counter" and if that gets to be too large
> a fraction of shard size, take a lock and repack the file.
>
How can you do the "repacking" step atomically?
Daniel
> This would only be problematic if some automated process were to edit a
> lot of revprops from comparatively old revisions. (Which I honestly
> don't think is a case worth optimizing for.)
>
> > My question for the list: Is the design of using SQLite as the primary
> > data storage for revprops acceptable?
>
> A flat file seems preferable, unless there are serious problems with
> that which I haven't seen.
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1976067
Re: revprop packing for fsfs
Posted by Peter Samuelson <pe...@p12n.org>.
[Paul Querna]
> Last time this was discussed was in October 2008[3], when hwright had
> packed revprops, but later reverted the work as revprops are mutable
> in r33724.
Is there a reason to try too hard to avoid allocation holes in the
revprops file? Seems to me you could just align each revision to, say,
16 or 32 bytes, and if a propedit operation wants to make it too big to
fit, you move it to the end. Maybe increment a "hole counter" and if
that gets to be too large a fraction of shard size, take a lock and
repack the file.
This would only be problematic if some automated process were to edit a
lot of revprops from comparatively old revisions. (Which I honestly
don't think is a case worth optimizing for.)
> My question for the list: Is the design of using SQLite as the primary
> data storage for revprops acceptable?
A flat file seems preferable, unless there are serious problems with
that which I haven't seen.
--
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/