You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Paul Querna <ch...@force-elite.com> on 2009/05/01 06:55:31 UTC
[patch] revprop packing for fsfs
On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
<hy...@mail.utexas.edu> wrote:
> On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
>> I'm hacking up a patch using SQLite for storing the revprops right
>> now, hopefully be able to post it in a few days when I get some more
>> hacking time in on it.
>
> How about the attached? I'm testing now, but it looks to give a ~90% space
> improvement in revprop storage. Still a few kinks, but if you've got some
> time to test it, I'd appreciate it.
>
> -Hyrum
Thanks Hyrum!
Based off your patch, new patch is attached.
Revprops use the normal sharded format, until a pack is run, at which
point they are combined into a single sqlite database.
Thoughts?
Thanks,
Paul
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2009616
Re: [patch] revprop packing for fsfs
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Haven't looked at the patches, but this is to remind people of the option
to test packing features by building with
-DPACK_AFTER_EVERY_COMMIT -DSVN_FS_FS_DEFAULT_MAX_FILES_PER_DIR=%d
or by passing
FSFS_PACKING=yes FSFS_SHARDING=%d
to 'make check'. (I recommend the former.) Normally I set %d to a small
number, e.g., 1-4.
Daniel
Paul Querna wrote on Thu, 30 Apr 2009 at 23:55 -0700:
> On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
> <hy...@mail.utexas.edu> wrote:
> > On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
> >> I'm hacking up a patch using SQLite for storing the revprops right
> >> now, hopefully be able to post it in a few days when I get some more
> >> hacking time in on it.
> >
> > How about the attached? I'm testing now, but it looks to give a ~90% space
> > improvement in revprop storage. Still a few kinks, but if you've got some
> > time to test it, I'd appreciate it.
> >
> > -Hyrum
>
>
> Thanks Hyrum!
>
> Based off your patch, new patch is attached.
>
> Revprops use the normal sharded format, until a pack is run, at which
> point they are combined into a single sqlite database.
>
> Thoughts?
>
> Thanks,
>
> Paul
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2012225
RE: Re: [patch] revprop packing for fsfs
Posted by Andy Bolstridge <an...@bolstridge.plus.com>.
No, that's s much as I expected to hear, I just get worried sometimes when people say "scrap x, replace with y".
eg. http://stackoverflow.com/questions/398802/using-mssql-backend-for-subversioning-repositories
or
http://stackoverflow.com/questions/777912/subversion-over-sql-server
When you've finished laughing (easier backups!) you see why I think incremental changes to existing code is usually preferable :) if sqlite does a lot, and replaces crufty code, then that's great. If no-one has time to implement it though... then small changes to existing is probably a better choice to make.
cheers.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2352810
Re: [patch] revprop packing for fsfs
Posted by "Hyrum K. Wright" <hy...@hyrumwright.org>.
On May 20, 2009, at 2:40 AM, webpost@tigris.org wrote:
> I won't criticise SQLite, but if there was a case for keeping the
> flat file system wouldn't a simple solution be to write modified
> revprops as a single file, and mark the revprop in the packed file
> as obsolete?
>
> Then reading a revprop reads the packed file as normal, but if the
> flag is set, read the revprop file that will be found on disk.
>
> A regular pack operation will then recombine the revprop into the
> packed file.
>
> Just my 2p, but if SQLite gives more functionality or is just faster
> - then I'm well happy with that!
Still haven't had time to review the patch, though I have time to do
some drive-by kibitzing. :)
Disks have improved vastly since Subversion debuted several years
ago. When it first came out, we spent a lot of time optimizing disk
accesses, on both the client and server. (Incidentally, this led to a
lot of mess in the working copy library, precipitating the current
rewrite known as wc-ng.)
The increase in disk performance means that we can sacrifice a little
bit of performance, for space and maintainability considerations.
Subversion should be in the business of doing version control, not
worrying about low-level file storage formats. Using a separate
library to manage the on-disk formats let's us concentrate on solving
version control problems. There isn't anything special about sqlite
per se, it's just the most convenient choice for the time being.
That's probably more than you'd like to hear, it's something I've been
thinking about recently, so it all spilled out. (And yes, Paul, I'm
still planning on reviewing the revprop-sqlite patch. Where did I put
those tuits?)
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2352669
RE: Re: [patch] revprop packing for fsfs
Posted by we...@tigris.org.
I won't criticise SQLite, but if there was a case for keeping the flat file system wouldn't a simple solution be to write modified revprops as a single file, and mark the revprop in the packed file as obsolete?
Then reading a revprop reads the packed file as normal, but if the flag is set, read the revprop file that will be found on disk.
A regular pack operation will then recombine the revprop into the packed file.
Just my 2p, but if SQLite gives more functionality or is just faster - then I'm well happy with that!
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2328521
Re: [patch] revprop packing for fsfs
Posted by "C. Michael Pilato" <cm...@collab.net>.
I've got your mail/patch marked for processing. Just digging for Round
Tuits at the moment.
Paul Querna wrote:
> bump, anything I should do to move this patch along?
>
> It seems to work for me, and passed the test cases when i ran em on my mac....
>
> Thanks,
>
> Paul
>
> On Thu, Apr 30, 2009 at 11:55 PM, Paul Querna <ch...@force-elite.com> wrote:
>> On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
>> <hy...@mail.utexas.edu> wrote:
>>> On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
>>>> I'm hacking up a patch using SQLite for storing the revprops right
>>>> now, hopefully be able to post it in a few days when I get some more
>>>> hacking time in on it.
>>> How about the attached? I'm testing now, but it looks to give a ~90% space
>>> improvement in revprop storage. Still a few kinks, but if you've got some
>>> time to test it, I'd appreciate it.
>>>
>>> -Hyrum
>>
>> Thanks Hyrum!
>>
>> Based off your patch, new patch is attached.
>>
>> Revprops use the normal sharded format, until a pack is run, at which
>> point they are combined into a single sqlite database.
>>
>> Thoughts?
>>
>> Thanks,
>>
>> Paul
>>
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2097588
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2097651
Re: [patch] revprop packing for fsfs
Posted by Paul Querna <ch...@force-elite.com>.
bump, anything I should do to move this patch along?
It seems to work for me, and passed the test cases when i ran em on my mac....
Thanks,
Paul
On Thu, Apr 30, 2009 at 11:55 PM, Paul Querna <ch...@force-elite.com> wrote:
> On Wed, Apr 29, 2009 at 4:33 PM, Hyrum K. Wright
> <hy...@mail.utexas.edu> wrote:
>> On Apr 29, 2009, at 5:19 PM, Paul Querna wrote:
>>> I'm hacking up a patch using SQLite for storing the revprops right
>>> now, hopefully be able to post it in a few days when I get some more
>>> hacking time in on it.
>>
>> How about the attached? I'm testing now, but it looks to give a ~90% space
>> improvement in revprop storage. Still a few kinks, but if you've got some
>> time to test it, I'd appreciate it.
>>
>> -Hyrum
>
>
> Thanks Hyrum!
>
> Based off your patch, new patch is attached.
>
> Revprops use the normal sharded format, until a pack is run, at which
> point they are combined into a single sqlite database.
>
> Thoughts?
>
> Thanks,
>
> Paul
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2097588