You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by John Beranek <jo...@redux.org.uk> on 2010/07/28 09:01:52 UTC
Repository shrinkage on conversion 1.5 -> 1.6?
I maintain a 76000 revision Subversion 1.5 repository that add up to a
101GiB FSFS db.
I'm looking to upgrade to 1.6, so did a dump and load cycle. The
resulting dump file was 173GiB (I use --deltas). What surprised me is
that the restored repository came out as 71GB - 30GiB smaller!
Is this unusual? Does it suggest some data has gone missing, or what?
Nothing I saw in the release notes suggest repository efficiencies of
this order - I have _not_ packed the repository.
Cheers,
John.
--
John Beranek To generalise is to be an idiot.
http://redux.org.uk/ -- William Blake
RE: Repository shrinkage on conversion 1.5 -> 1.6?
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Yves Martin [mailto:ymartin59@free.fr]
> Sent: zaterdag 31 juli 2010 0:07
> To: John Beranek
> Cc: users@subversion.apache.org
> Subject: Re: Repository shrinkage on conversion 1.5 -> 1.6?
>
> On Wed, 2010-07-28 at 11:44 +0100, John Beranek wrote:
>
> > I believe we do have a lot of traditional ttb branching, plus
> > additionally some copying from "common" directories to project
> directories.
> >
> > Thanks for your quick answer, it reassures me somewhat that data hasn't
> > gone missing. ;)
>
> Hello John,
>
> If interested, you can use my procedure and fsfs-reshard script to
> upgrade from 1.5 to 1.6 without dump/load (really faster)
> and improve performance/storage with a specific shard size tuning
> (default is 1000 revisions in a pack):
> http://ymartin59.free.fr/wordpress/index.php/projets/tune-subversion-
> fsfs-repository/
Sharding reduces the number of files and thereby reduces the blocksize overhead. I think his huge storage requirement difference is due to the 'representation sharing', where identical files are only stored once.
See http://subversion.apache.org/docs/release-notes/1.6.html#rep-sharing
You can't get this size difference by just resharding. (But resharding and packing can certainly give some performance boost, depending on your server configuration)
Bert
Re: Repository shrinkage on conversion 1.5 -> 1.6?
Posted by Yves Martin <ym...@free.fr>.
On Wed, 2010-07-28 at 11:44 +0100, John Beranek wrote:
> I believe we do have a lot of traditional ttb branching, plus
> additionally some copying from "common" directories to project directories.
>
> Thanks for your quick answer, it reassures me somewhat that data hasn't
> gone missing. ;)
Hello John,
If interested, you can use my procedure and fsfs-reshard script to
upgrade from 1.5 to 1.6 without dump/load (really faster)
and improve performance/storage with a specific shard size tuning
(default is 1000 revisions in a pack):
http://ymartin59.free.fr/wordpress/index.php/projets/tune-subversion-fsfs-repository/
Regards
Yves
Re: Repository shrinkage on conversion 1.5 -> 1.6?
Posted by John Beranek <jo...@redux.org.uk>.
On 28/07/10 10:50, B Smith-Mannschott wrote:
> On Wed, Jul 28, 2010 at 11:01, John Beranek <jo...@redux.org.uk> wrote:
>>
>> I maintain a 76000 revision Subversion 1.5 repository that add up to a
>> 101GiB FSFS db.
>>
>> I'm looking to upgrade to 1.6, so did a dump and load cycle. The
>> resulting dump file was 173GiB (I use --deltas). What surprised me is
>> that the restored repository came out as 71GB - 30GiB smaller!
>>
>> Is this unusual? Does it suggest some data has gone missing, or what?
>> Nothing I saw in the release notes suggest repository efficiencies of
>> this order - I have _not_ packed the repository.
>>
>
> Hi John,
>
> It looks like representation sharing works particularly well for your
> repository.
>
> http://subversion.apache.org/docs/release-notes/1.6.html#rep-sharing
Ah, now that's something I must've overlooked!
> The 30% reduction you're seeing is a pretty nice space savings. Do you
> do a lot of branching? Do you have the identical files comitted
> separately to different parts of the repository?
I believe we do have a lot of traditional ttb branching, plus
additionally some copying from "common" directories to project directories.
Thanks for your quick answer, it reassures me somewhat that data hasn't
gone missing. ;)
Cheers,
John.
--
John Beranek To generalise is to be an idiot.
http://redux.org.uk/ -- William Blake
Re: Repository shrinkage on conversion 1.5 -> 1.6?
Posted by B Smith-Mannschott <bs...@gmail.com>.
On Wed, Jul 28, 2010 at 11:01, John Beranek <jo...@redux.org.uk> wrote:
>
> I maintain a 76000 revision Subversion 1.5 repository that add up to a
> 101GiB FSFS db.
>
> I'm looking to upgrade to 1.6, so did a dump and load cycle. The
> resulting dump file was 173GiB (I use --deltas). What surprised me is
> that the restored repository came out as 71GB - 30GiB smaller!
>
> Is this unusual? Does it suggest some data has gone missing, or what?
> Nothing I saw in the release notes suggest repository efficiencies of
> this order - I have _not_ packed the repository.
>
Hi John,
It looks like representation sharing works particularly well for your
repository.
http://subversion.apache.org/docs/release-notes/1.6.html#rep-sharing
The 30% reduction you're seeing is a pretty nice space savings. Do you
do a lot of branching? Do you have the identical files comitted
separately to different parts of the repository?
// Ben