You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Dieter Oberkofler <d....@qualiant.at> on 2008/03/28 11:57:00 UTC
Best practices on how to reorganize repository
Our SVN repository is growing very quickly and we are starting to have
problems with the backup of the fully repository and would therefore like to
reduce it in size by reorganizing it. What are the best practices (in order
of importance) to:
1) Physically remove branches and tags that are no longer needed?
2) Physically remove removed files (that should never have been in the
repository in the first place)?
3) Physically remove old history revisions?
Thank you,
Dieter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Best practices on how to reorganize repository
Posted by "Fouts Christopher (QNA RTP PT PREV)" <Ch...@qimonda.com>.
Are you including executables in your repository, hence,
it's quick growth?
>-----Original Message-----
>From: Henrik Sundberg [mailto:storangen@gmail.com]
>Sent: Friday, March 28, 2008 8:50 AM
>To: users@subversion.tigris.org
>Subject: Re: Best practices on how to reorganize repository
>
>2008/3/28, Dieter Oberkofler <d....@qualiant.at>:
>> Our SVN repository is growing very quickly and we are
>starting to have
>> problems with the backup of the fully repository and would therefore
>> like to reduce it in size by reorganizing it. What are the best
>> practices (in order of importance) to:
>> 1) Physically remove branches and tags that are no longer needed?
>> 2) Physically remove removed files (that should never have been in
>> the repository in the first place)?
>> 3) Physically remove old history revisions?
>> Thank you,
>> Dieter
>
>Not answering your question..., but can you tell us why the
>repository is growing so fast?
>/$
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>For additional commands, e-mail: users-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: RE: Best practices on how to reorganize repository
Posted by "Gleason, Todd" <tg...@impac.com>.
Sounds like a lot. Were the tags/branches created in Svn by using svn
copy, or were they imported separately?
An alternative import strategy might be to make a read-only repository
containing the history (for reference), and the only import all the new
stuff you need into the repository intended for active use.
-----Original Message-----
From: Dieter Oberkofler [mailto:d.1234567890@qualiant.at]
Sent: Friday, March 28, 2008 12:09 PM
To: users@subversion.tigris.org
Subject: RE: Best practices on how to reorganize repository
1) It is quite a large project by itself (the trunk has about 50000
files)
2) We have converted almost 15 years of history into SVN (about 30000
revisions)
3) We also have quite some binary data in the repository (documentation,
tools, images, etc)
4) A very large number of tags and branches have been used
5) A large number of quite large vendors branches are part of the
repository
-----Original Message-----
From: Henrik Sundberg [mailto:storangen@gmail.com]
Sent: Friday, March 28, 2008 13:50
To: users@subversion.tigris.org
Subject: Re: Best practices on how to reorganize repository
2008/3/28, Dieter Oberkofler <d....@qualiant.at>:
> Our SVN repository is growing very quickly and we are starting to have
> problems with the backup of the fully repository and would therefore
> like to reduce it in size by reorganizing it. What are the best
> practices (in order of importance) to:
> 1) Physically remove branches and tags that are no longer needed?
> 2) Physically remove removed files (that should never have been in
> the repository in the first place)?
> 3) Physically remove old history revisions?
> Thank you,
> Dieter
Not answering your question..., but can you tell us why the repository
is
growing so fast?
/$
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by Talden <ta...@gmail.com>.
On Sat, Mar 29, 2008 at 7:08 AM, Dieter Oberkofler
<d....@qualiant.at> wrote:
> 1) It is quite a large project by itself (the trunk has about 50000 files)
> 2) We have converted almost 15 years of history into SVN (about 30000
> revisions)
> 3) We also have quite some binary data in the repository (documentation,
> tools, images, etc)
> 4) A very large number of tags and branches have been used
> 5) A large number of quite large vendors branches are part of the repository
Be aware that the supported mechanism of "dump -> filter -> load" does
not does the heavy lifting of tracing where a filtered path has been
copied to. If path X has been copied into 5 branches and 2 tags and
you only filter X, two tags and two branches then the 3 remaining
branches which did have cheap copies are now 3 full-text revisions --
that is, you can obliterate and end up with a larger repository than
you started with.
There are other, more obscure ways in which a repository could become
larger - the reload with obliterated content could select different
revisions to use the 'skip' part of the skip-delta model and could
result in small diff revisions becoming full-text revisions.
Surgery like this on a repository is a serious under-taking and may
not yield the space savings that would seem apparent at first glance -
given you can produce quite misleading history in revisions with
obliterated content you may want to reconsider. That said, it
certainly has its place.
Remember to reload into a new repo with the same ID) - that way, if
the revision numbers aren't changed and no users have any checkouts
with paths that were obliterated, they may be able to continue to use
their working-copy. Most often though users will need to checkout
again - this dump-filter-reload mechanism is a destructive and
untracked operation.
--
Talden
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by Erik Huelsmann <eh...@gmail.com>.
On Sun, Mar 30, 2008 at 2:20 PM, John Peacock
<jo...@havurah-software.org> wrote:
> Karl Heinz Marbaise wrote:
> >> 5) A large number of quite large vendors branches are
> >> part of the repository
> > This is not the reason, cause a tag/branch needs about 1 KiBi per each...
>
> Only if the vendor branches were done as a copy within Subversion. If each
> vendor drop was imported independently, then there would be a highly inefficient
> storage of duplicate files.
Yup. Although there's work underway to actually make that kind of
storage more efficient, assuming that most files in the vendor drop
haven't changed: the rep-sharing branch already implements sharing of
content for unrelated, but content-wise-equal files. For BDB there are
some design problems with this issue which need tackling, but we think
we'll be sharing content for unrelated files in one of the post-1.5
releases.
Bye,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by John Peacock <jo...@havurah-software.org>.
Karl Heinz Marbaise wrote:
>> 5) A large number of quite large vendors branches are
>> part of the repository
> This is not the reason, cause a tag/branch needs about 1 KiBi per each...
Only if the vendor branches were done as a copy within Subversion. If each
vendor drop was imported independently, then there would be a highly inefficient
storage of duplicate files.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi there,
can you give the size of the repository?
10 GiBi? 20 GiBi?
> 4) A very large number of tags and branches have been used
> 5) A large number of quite large vendors branches are
> part of the repository
This is not the reason, cause a tag/branch needs about 1 KiBi per each...
I'm working with repositories which have 16000 Revisions...
Kind regards
Karl Heinz Marbaise
--
SoftwareEntwicklung Beratung Schulung Tel.: +49 (0) 2405 / 415 893
Dipl.Ing.(FH) Karl Heinz Marbaise ICQ#: 135949029
Hauptstrasse 177 USt.IdNr: DE191347579
52146 Würselen http://www.soebes.de
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Best practices on how to reorganize repository
Posted by Dieter Oberkofler <d....@qualiant.at>.
1) It is quite a large project by itself (the trunk has about 50000 files)
2) We have converted almost 15 years of history into SVN (about 30000
revisions)
3) We also have quite some binary data in the repository (documentation,
tools, images, etc)
4) A very large number of tags and branches have been used
5) A large number of quite large vendors branches are part of the repository
-----Original Message-----
From: Henrik Sundberg [mailto:storangen@gmail.com]
Sent: Friday, March 28, 2008 13:50
To: users@subversion.tigris.org
Subject: Re: Best practices on how to reorganize repository
2008/3/28, Dieter Oberkofler <d....@qualiant.at>:
> Our SVN repository is growing very quickly and we are starting to have
> problems with the backup of the fully repository and would therefore
> like to reduce it in size by reorganizing it. What are the best
> practices (in order of importance) to:
> 1) Physically remove branches and tags that are no longer needed?
> 2) Physically remove removed files (that should never have been in
> the repository in the first place)?
> 3) Physically remove old history revisions?
> Thank you,
> Dieter
Not answering your question..., but can you tell us why the repository is
growing so fast?
/$
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by Henrik Sundberg <st...@gmail.com>.
2008/3/28, Dieter Oberkofler <d....@qualiant.at>:
> Our SVN repository is growing very quickly and we are starting to have
> problems with the backup of the fully repository and would therefore like to
> reduce it in size by reorganizing it. What are the best practices (in order
> of importance) to:
> 1) Physically remove branches and tags that are no longer needed?
> 2) Physically remove removed files (that should never have been in the
> repository in the first place)?
> 3) Physically remove old history revisions?
> Thank you,
> Dieter
Not answering your question..., but can you tell us why the repository
is growing so fast?
/$
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi,
> Our SVN repository is growing very quickly and we are starting to have
> problems with the backup of the fully repository
What is the real problem doing a backup ? The backup media?
Kind regards
Karl Heinz Marbaise
--
SoftwareEntwicklung Beratung Schulung Tel.: +49 (0) 2405 / 415 893
Dipl.Ing.(FH) Karl Heinz Marbaise ICQ#: 135949029
Hauptstrasse 177 USt.IdNr: DE191347579
52146 Würselen http://www.soebes.de
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Best practices on how to reorganize repository
Posted by John Peacock <jo...@havurah-software.org>.
Dieter Oberkofler wrote:
> 1) Physically remove branches and tags that are no longer needed?
> 2) Physically remove removed files (that should never have been in the
> repository in the first place)?
> 3) Physically remove old history revisions?
None of these options are currently possible with the existing Subversion code
(search the archives for 'svn obliterate' for the many discussions on this
topic). The only current solution is a dump/load cycle using svndumpfilter (or
its ilk) to strip out the files you don't want present. If you have been using
this repository for some time (i.e. pre-1.4), this would also net superior
compression of the remaining repository files (assuming you are using FSFS and
not BDB).
Sorry
John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org