You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by An...@versit.de on 2008/10/22 08:45:41 UTC
Problem using subversion on a large repository
Hi,
my problem is using subversion in a large development project
Just the numbers:
1 ~ 500 Software Packages with up to ~100 files per
package
2. a lot of testig data
3. total size of a checkout is ~5G
4. HW SLES 10 with 32G memory + 2 quadcore
5. subversion default package in SLES 10 ->
subversion-1.3.1-1.10
the size of the repository itself is
7124552 G
The problem to delete old (unused) source-trees using the
svanadmin dump/load
feature but i'm not able to even dump this repository because
i have a lot of tags created with:
svn copy
and these tags have no space in the rep but seems
to use a lot of space in the dumps
a dump of a sigle revision piped into gzip 9 took
1 day and created a 80 G dumpfile
the propblem is what I want to admin the repository for
-> cleanup unused revisons / trees
i need the dump/filter/load syntax because no other command is
doing this job
I don't even have enougth disk space to do this and recreating a
new repository with the
80G gzip 9 single revision dump creates a repository with close
this size
(remember my current rep size is ~7G)
-> How i can cleanup my rep ?
Freundliche Grüße
Andreas Otto
ISV13 - Systemverantwortung Leben
Telefon 0431/603-2388
Sophienblatt 33
24114 Kiel
VersIT Versicherungs-Informatik GmbH
Gottlieb-Daimler-Str. 2, 68165 Mannheim
Registergericht: Mannheim HRB 6287
Vorsitzender der Geschäftsführung: Claus-Peter Gutt
Re: Problem using subversion on a large repository
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Oct 22, 2008 at 4:45 AM, <An...@versit.de> wrote:
> Hi,
>
> my problem is using subversion in a large development project
>
> Just the numbers:
>
> 1 ~ 500 Software Packages with up to ~100 files per package
> 2. a lot of testig data
> 3. total size of a checkout is ~5G
> 4. HW SLES 10 with 32G memory + 2 quadcore
> 5. subversion default package in SLES 10 ->
> subversion-1.3.1-1.10
>
> the size of the repository itself is
>
> 7124552 G
>
> The problem to delete old (unused) source-trees using the
>
> svanadmin dump/load
>
> feature but i'm not able to even dump this repository because
> i have a lot of tags created with:
>
> svn copy
>
> and these tags have no space in the rep but seems
> to use a lot of space in the dumps
>
> a dump of a sigle revision piped into gzip 9 took
> 1 day and created a 80 G dumpfile
How are you doing your dumps? In the "normal" case of doing a full
repository dump:
svnadmin dump repos > repos.dump
For copies. all you will have in the dumpfile is some minimal copyfrom
information that describes what SVN needs to recreate the copy. The
dumpfile does not contain an expanded version of the copied files.
If you are dumping specific revisions, are you using the --incremental
option? If you do not, then I believe you get expanded dump
information as the dump is designed to stand on its own. Adding the
--incremental option tells dump that you only intend to apply this
back to the same repository.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Problem using subversion on a large repository
Posted by Felix Gilcher <fe...@bitextender.com>.
On Oct 22, 2008, at 12:21 PM, Purple Streak wrote:
> 2008/10/22 Felix Gilcher <fe...@bitextender.com>:
>>> Anyways, how did you dump that single revision? The exact command
>>> would be
>>> of interest. As far as I know, the dumpfile uses a diff format as
>>> well, so
>
> I think by default the dump actually does full versions each time.
> Take a look at
>
> http://svnbook.red-bean.com/en/1.5/svn.ref.svnadmin.c.dump.html
Yeah well, we're sort of both right: It will only dump changed files,
but those not as a delta.
>
> in particular the --deltas option. I don't know if this is clever
> enough to reduces the copies to almost nothing but I would hope it
> would. It also depends on how the svndumpfilter works in this
> instance but I would try it and see how you go :)
svndumpfilter will not work with --delta: "There are, however,
disadvantages to using this option—deltified dump files are more CPU-
intensive to create, cannot be operated on by svndumpfilter"
>
> Also if the dump files still get too big then check out the
> --incremental option as well to split up the dump into separate files.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
cheers
felix
--
Felix Gilcher
Bitextender GmbH
Paul-Heyse-Str. 6
D-80336 München
T: +49 89 57 08 15 16
F: +49 89 57 08 15 17
M: +49 172 840 88 28
felix.gilcher@bitextender.com
http://bitextender.com/
Amtsgericht München, HRB 174280
Geschäftsführer: David Zülke, Florian Clever
Re: Problem using subversion on a large repository
Posted by Purple Streak <mr...@googlemail.com>.
2008/10/22 Felix Gilcher <fe...@bitextender.com>:
>> Anyways, how did you dump that single revision? The exact command would be
>> of interest. As far as I know, the dumpfile uses a diff format as well, so
I think by default the dump actually does full versions each time.
Take a look at
http://svnbook.red-bean.com/en/1.5/svn.ref.svnadmin.c.dump.html
in particular the --deltas option. I don't know if this is clever
enough to reduces the copies to almost nothing but I would hope it
would. It also depends on how the svndumpfilter works in this
instance but I would try it and see how you go :)
Also if the dump files still get too big then check out the
--incremental option as well to split up the dump into separate files.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Problem using subversion on a large repository
Posted by Felix Gilcher <fe...@bitextender.com>.
Darn, forgot to CC the list :)
On Oct 22, 2008, at 11:35 AM, Felix Gilcher wrote:
> Well, I hope you're missing a decimal point on this one:
>
>> 7124552 G
>
> ;)
>
> Anyways, how did you dump that single revision? The exact command
> would be of interest. As far as I know, the dumpfile uses a diff
> format as well, so the full dump should only be slightly larger than
> the repository itself, not like 10 times the size. However, if you
> dump only the last revision without specifying --incremental, it
> might spit out the full data for the head revision, which would be
> each copy for itself, no ancestry information taken into account.
>
> Anyways, what are you trying to accomplish? You can just use svn
> delete to remove unused branches/tags/parts of the tree. It will not
> regain you any space, but as most of your stuff is a copy, your loss
> is not really big. Copies only take a couple of bytes, they're
> cheap. It's usually not required to clean a repository.
>
> If you really want to go down the road of filtering out revisions,
> you can alway pipe the output of svnadmin dump straight to
> dumpfilter and then to a file, this would at least remove the parts
> you don't need. As svnadmin dump does not alter the original
> repository, the risk of damage is pretty much nonexistent - just
> keep the old repo around until you've created and checked the new one.
>
> cheers
>
> felix
>
> On Oct 22, 2008, at 10:45 AM, Andreas.Otto@versit.de wrote:
>
>>
>> Hi,
>>
>> my problem is using subversion in a large development project
>>
>> Just the numbers:
>>
>> 1 ~ 500 Software Packages with up to ~100 files per
>> package
>> 2. a lot of testig data
>> 3. total size of a checkout is ~5G
>> 4. HW SLES 10 with 32G memory + 2 quadcore
>> 5. subversion default package in SLES 10 ->
>> subversion-1.3.1-1.10
>>
>> the size of the repository itself is
>>
>> 7124552 G
>>
>> The problem to delete old (unused) source-trees using the
>>
>> svanadmin dump/load
>>
>> feature but i'm not able to even dump this repository because
>> i have a lot of tags created with:
>>
>> svn copy
>>
>> and these tags have no space in the rep but seems
>> to use a lot of space in the dumps
>>
>> a dump of a sigle revision piped into gzip 9 took
>> 1 day and created a 80 G dumpfile
>>
>> the propblem is what I want to admin the repository for
>>
>> -> cleanup unused revisons / trees
>>
>> i need the dump/filter/load syntax because no other command
>> is doing this job
>> I don't even have enougth disk space to do this and
>> recreating a new repository with the
>> 80G gzip 9 single revision dump creates a repository with
>> close this size
>> (remember my current rep size is ~7G)
>>
>> -> How i can cleanup my rep ?
>>
>>
>>
>>
>>
>> Freundliche Grüße
>>
>> Andreas Otto
>> ISV13 - Systemverantwortung Leben
>>
>> Telefon 0431/603-2388
>> Sophienblatt 33
>> 24114 Kiel
>>
>> VersIT Versicherungs-Informatik GmbH
>> Gottlieb-Daimler-Str. 2, 68165 Mannheim
>> Registergericht: Mannheim HRB 6287
>> Vorsitzender der Geschäftsführung: Claus-Peter Gutt
>>
>>
>
--
Felix Gilcher
Bitextender GmbH
Paul-Heyse-Str. 6
D-80336 München
T: +49 89 57 08 15 16
F: +49 89 57 08 15 17
M: +49 172 840 88 28
felix.gilcher@bitextender.com
http://bitextender.com/
Amtsgericht München, HRB 174280
Geschäftsführer: David Zülke, Florian Clever