You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mix <mi...@acm.org> on 2002/07/31 01:15:37 UTC
shrinking repository...
Hey. Its been a while since I updated my repositories by performing a
"svnadmin dump" and "svnadmin load". Just compiled version 2809 and did
the dump/load thing and one of my repositories went from 44 MB disk
usage down to 8.5 MB. Checked it out and the repository seems fine, no
data loss, its just a lot smaller.
The repository in question had a lot of tags (directory copies). Is this
expected?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Branko Čibej <br...@xbc.nu>.
cmpilato@collab.net wrote:
>Karl Fogel <kf...@newton.ch.collab.net> writes:
>
>
>
>>Mix <mi...@acm.org> writes:
>>
>>
>>>Several of the files are > 100K. The new repos were made with revision
>>>2809 so yeh, they are kind of recent. Can't remember which revision
>>>made the old repos.
>>>
>>>My other repos don't have large files with numerous commits and they
>>>are all roughly the same size.
>>>
>>>
>>That's it, then. Brane activated the new deltification code in
>>revision 2803. :-)
>>
>>
>
>Looks like we're getting savings even in the < 100K category. I have
>a copy of the Subversion repos at 2656. 59 Megs large with
>unnecessary log files removed. I dumped and reloaded it. 43 Megs
>now. `strings' table is quite a bit smaller.
>
That can't be because of the combiner. Could it be that the strings
table was fragmented, and reloading made it more compact?
> But also, with the new
>1-Meg granularity on log files, I've saved 1.5 Megs. Woohoo!
>
And with the new hot-backup.py script, you'll save on backup size, too,
because we don't keep the shared mem environments around. That's another
Meg right there. :-)
(But don't switch to the new hot_backup.py before upgrading the server!)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Branko Čibej <br...@xbc.nu>.
Daniel Berlin wrote:
>
> On Wednesday, July 31, 2002, at 10:32 AM, cmpilato@collab.net wrote:
>
>> Karl Fogel <kf...@newton.ch.collab.net> writes:
>>
>>> Mix <mi...@acm.org> writes:
>>>
>>>> Several of the files are > 100K. The new repos were made with revision
>>>> 2809 so yeh, they are kind of recent. Can't remember which revision
>>>> made the old repos.
>>>>
>>>> My other repos don't have large files with numerous commits and they
>>>> are all roughly the same size.
>>>
>>>
>>> That's it, then. Brane activated the new deltification code in
>>> revision 2803. :-)
>>
>>
>> Looks like we're getting savings even in the < 100K category. I have
>> a copy of the Subversion repos at 2656. 59 Megs large with
>> unnecessary log files removed. I dumped and reloaded it. 43 Megs
>> now. `strings' table is quite a bit smaller. But also, with the new
>> 1-Meg granularity on log files, I've saved 1.5 Megs. Woohoo!
>
>
> Must be that we don't need to allocate as many overflow pages or
> something because we can stuff stuff in the pages used by the
> now-deltified >100k files.
Hey, I just remembered another thing I changed: In a lot of places,
especially in reps-strings.c, we used to have buffers on the stack with
hard-coded, arbitrarily chosen sizes. I changed all those to use
(dynamically-allocated) SVN_STREAM_CHUNK_SIZE. So, indeed, Dan may be right!
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Daniel Berlin <db...@dberlin.org>.
On Wednesday, July 31, 2002, at 10:32 AM, cmpilato@collab.net wrote:
> Karl Fogel <kf...@newton.ch.collab.net> writes:
>
>> Mix <mi...@acm.org> writes:
>>> Several of the files are > 100K. The new repos were made with
>>> revision
>>> 2809 so yeh, they are kind of recent. Can't remember which revision
>>> made the old repos.
>>>
>>> My other repos don't have large files with numerous commits and they
>>> are all roughly the same size.
>>
>> That's it, then. Brane activated the new deltification code in
>> revision 2803. :-)
>
> Looks like we're getting savings even in the < 100K category. I have
> a copy of the Subversion repos at 2656. 59 Megs large with
> unnecessary log files removed. I dumped and reloaded it. 43 Megs
> now. `strings' table is quite a bit smaller. But also, with the new
> 1-Meg granularity on log files, I've saved 1.5 Megs. Woohoo!
Must be that we don't need to allocate as many overflow pages or
something because we can stuff stuff in the pages used by the
now-deltified >100k files.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by cm...@collab.net.
Karl Fogel <kf...@newton.ch.collab.net> writes:
> Mix <mi...@acm.org> writes:
> > Several of the files are > 100K. The new repos were made with revision
> > 2809 so yeh, they are kind of recent. Can't remember which revision
> > made the old repos.
> >
> > My other repos don't have large files with numerous commits and they
> > are all roughly the same size.
>
> That's it, then. Brane activated the new deltification code in
> revision 2803. :-)
Looks like we're getting savings even in the < 100K category. I have
a copy of the Subversion repos at 2656. 59 Megs large with
unnecessary log files removed. I dumped and reloaded it. 43 Megs
now. `strings' table is quite a bit smaller. But also, with the new
1-Meg granularity on log files, I've saved 1.5 Megs. Woohoo!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Branko Čibej <br...@xbc.nu>.
cmpilato@collab.net wrote:
>Karl Fogel <kf...@newton.ch.collab.net> writes:
>
>
>
>>Mix <mi...@acm.org> writes:
>>
>>
>>>Several of the files are > 100K. The new repos were made with revision
>>>2809 so yeh, they are kind of recent. Can't remember which revision
>>>made the old repos.
>>>
>>>My other repos don't have large files with numerous commits and they
>>>are all roughly the same size.
>>>
>>>
>>That's it, then. Brane activated the new deltification code in
>>revision 2803. :-)
>>
>>
>
>Sez cmpilato to Brane: *smooch* :-)
>
>
/me rules, ok? :-)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by cm...@collab.net.
Karl Fogel <kf...@newton.ch.collab.net> writes:
> Mix <mi...@acm.org> writes:
> > Several of the files are > 100K. The new repos were made with revision
> > 2809 so yeh, they are kind of recent. Can't remember which revision
> > made the old repos.
> >
> > My other repos don't have large files with numerous commits and they
> > are all roughly the same size.
>
> That's it, then. Brane activated the new deltification code in
> revision 2803. :-)
Sez cmpilato to Brane: *smooch* :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Mix <mi...@acm.org> writes:
> Several of the files are > 100K. The new repos were made with revision
> 2809 so yeh, they are kind of recent. Can't remember which revision
> made the old repos.
>
> My other repos don't have large files with numerous commits and they
> are all roughly the same size.
That's it, then. Brane activated the new deltification code in
revision 2803. :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Mix <mi...@acm.org>.
Karl Fogel wrote:
> Do you have big files ( > 100K or so)? If you made the new repository
> *really* recently, then one difference is that we're deltifying large
> files now. As of Brane's commit, like, today :-).
Several of the files are > 100K. The new repos were made with revision
2809 so yeh, they are kind of recent. Can't remember which revision made
the old repos.
My other repos don't have large files with numerous commits and they are
all roughly the same size.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Mix <mi...@acm.org> writes:
> -rw------- 1 mixtim mixtim 42778624 Jul 30 22:27 crafty/db/strings
> -rw------- 1 mixtim mixtim 6029312 Jul 30 22:32 crafty/db/strings
>
> I can't see anything wrong with the new repository, its just a lot smaller.
>
> Should I be worried?
Do you have big files ( > 100K or so)? If you made the new repository
*really* recently, then one difference is that we're deltifying large
files now. As of Brane's commit, like, today :-).
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Mix <mi...@acm.org>.
cmpilato@collab.net wrote:
> Well, I don't know about the rest of the world, but I'm frightened. :-)
>
>>The repository in question had a lot of tags (directory copies). Is this
>>expected?
>
> Have you verified the integrity of those branches and tags?
Yeh, just verified the trunk and every tag -- they are identical.
The big difference between the two repositories is the strings file.
Here is the old one followed by the new one:
-rw------- 1 mixtim mixtim 42778624 Jul 30 22:27 crafty/db/strings
-rw------- 1 mixtim mixtim 6029312 Jul 30 22:32 crafty/db/strings
I can't see anything wrong with the new repository, its just a lot smaller.
Should I be worried?
In case anyone is interested, this repo contains every version of the
crafty chess engine from version 15.0 to 18.15 with a tag for all 73
versions.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by cm...@collab.net.
Mix <mi...@acm.org> writes:
> Hey. Its been a while since I updated my repositories by performing a
> "svnadmin dump" and "svnadmin load". Just compiled version 2809 and did
> the dump/load thing and one of my repositories went from 44 MB disk
> usage down to 8.5 MB. Checked it out and the repository seems fine, no
> data loss, its just a lot smaller.
Well, I don't know about the rest of the world, but I'm frightened. :-)
> The repository in question had a lot of tags (directory copies). Is this
> expected?
Have you verified the integrity of those branches and tags?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Mix <mi...@acm.org>.
Karl Fogel wrote:
> I think it's just the Berkeley log.0000* files, not sure though.
No, only one log file was present in each repo and I had already
self-limited their size to 1MB so that isn't a factor. I'll check into
this a bit more and post my findings.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: shrinking repository...
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Mix <mi...@acm.org> writes:
> Hey. Its been a while since I updated my repositories by performing a
> "svnadmin dump" and "svnadmin load". Just compiled version 2809 and
> did the dump/load thing and one of my repositories went from 44 MB
> disk usage down to 8.5 MB. Checked it out and the repository seems
> fine, no data loss, its just a lot smaller.
>
> The repository in question had a lot of tags (directory copies). Is
> this expected?
I think it's just the Berkeley log.0000* files, not sure though.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org