You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by arun prasath <ge...@gmail.com> on 2015/11/25 10:25:02 UTC
SVN 1.6.17 dump is growing larger than repository size (approx. more
than 10 times)
Hello Team,
I am creating Subversion 1.6.17 dump for a repository hosted in Linux
server. SVNSERVE is serving the repository. We are migrating to SVN 1.9.2.
*While creating the dump for repository of size 1.8 GB (revisions 3000+),
the dump command completes revision 119 and hangs and keep updating the
dumpfile which grows to 7-8 GBs. dump command is not moving to next
revision but kept updating the dump file. So, i just closed the putty to
stop creating the dump.*
However, I ran the svnadmin verify which completes revision 119 and take
some time to verify revision 120 and complete the verification successfully
for all the repository revisions.
Since, there is no error output I have no logs to attach. Please suggest
the work around for this situation. How can create the dump and migrate to
target server. I am planning to use rsync from the server. I will post how
it goes.
I am expecting the workaround to this situation and let me know if this is
a known issue in SVN 1.6.17. This is my active repository and created the
dump without stopping the svnserve program in the production server.
Thanks,
Arun
RE: SVN 1.6.17 dump is growing larger than repository size (approx. more than 10 times)
Posted by Bert Huijben <be...@qqmail.nl>.
By default svnadmin dump doesn’t produce deltas… so a small change on a file will have it include the entire file in the dumpfile.
If you pass ‘--deltas' to svnadmin dump (see ‘svnadmin help dump’) your file will be smaller, at the cost of some extra processing during the dump creation.
Bert
From: Barry Gershenfeld [mailto:gbarry42@gmail.com]
Sent: woensdag 25 november 2015 21:57
To: Subversion <us...@subversion.apache.org>
Subject: Re: SVN 1.6.17 dump is growing larger than repository size (approx. more than 10 times)
Unless svn's changed, you can look in your repositories directory on the server, e.g., repos/myproj/db/revs/0/ and you can see each revision stored there. Under the db directory is revs and also revprops, and under each of those is one or more subdirectory (mine just had one called "0"). If you see one that is outrageously large, then dump is just trying to do its job. If they are all reasonable size, then maybe there's a circular reference or some other bug.
Also, svnadmin dump just goes to standard-out, so you can send it to the screen, or through a grep filter or some other ingenious scheme to try to get a handle on what's happening.
Barry
On Wed, Nov 25, 2015 at 4:41 AM, Nico Kadel-Garcia <nkadel@gmail.com <ma...@gmail.com> > wrote:
On Wed, Nov 25, 2015 at 4:25 AM, arun prasath <get2arun@gmail.com <ma...@gmail.com> > wrote:
> Hello Team,
>
> I am creating Subversion 1.6.17 dump for a repository hosted in Linux
> server. SVNSERVE is serving the repository. We are migrating to SVN 1.9.2.
>
> While creating the dump for repository of size 1.8 GB (revisions 3000+), the
> dump command completes revision 119 and hangs and keep updating the dumpfile
> which grows to 7-8 GBs. dump command is not moving to next revision but kept
> updating the dump file. So, i just closed the putty to stop creating the
> dump.
Which Linux? And are you using the verndor provided version, or a
locally compiled one? Subversion 1.6 was last up to 1.6.23: is there
any reason you can't upgrade that? Ideally with a hotcopy or "rsync"
based copy of the repository to somehwere else, for testing the copy?
> However, I ran the svnadmin verify which completes revision 119 and take
> some time to verify revision 120 and complete the verification successfully
> for all the repository revisions.
Hmm. You might consider skipping the svnadmin dump command, and simply
setting up an svnsync mirror on the Subversion 1.9 server. Even using
"rsync" to bring the old repository over should let you run a dump and
reload on the server with Subversion 1.9, as long as there's not some
other weird corruption with revision 119 or 120.
> Since, there is no error output I have no logs to attach. Please suggest the
> work around for this situation. How can create the dump and migrate to
> target server. I am planning to use rsync from the server. I will post how
> it goes.
>
> I am expecting the workaround to this situation and let me know if this is a
> known issue in SVN 1.6.17. This is my active repository and created the dump
> without stopping the svnserve program in the production server.
It's not one I've seen, but I don't let my repositories get that big.
And I'd look at revisions 119 and 120 to see if there were erroneous
commits of huge binary files, in which case I'd want to exclude them
from the backup.
Re: SVN 1.6.17 dump is growing larger than repository size (approx.
more than 10 times)
Posted by Barry Gershenfeld <gb...@gmail.com>.
Unless svn's changed, you can look in your repositories directory on the
server, e.g., repos/myproj/db/revs/0/ and you can see each revision stored
there. Under the db directory is revs and also revprops, and under each of
those is one or more subdirectory (mine just had one called "0"). If you
see one that is outrageously large, then dump is just trying to do its
job. If they are all reasonable size, then maybe there's a circular
reference or some other bug.
Also, svnadmin dump just goes to standard-out, so you can send it to the
screen, or through a grep filter or some other ingenious scheme to try to
get a handle on what's happening.
Barry
On Wed, Nov 25, 2015 at 4:41 AM, Nico Kadel-Garcia <nk...@gmail.com> wrote:
> On Wed, Nov 25, 2015 at 4:25 AM, arun prasath <ge...@gmail.com> wrote:
> > Hello Team,
> >
> > I am creating Subversion 1.6.17 dump for a repository hosted in Linux
> > server. SVNSERVE is serving the repository. We are migrating to SVN
> 1.9.2.
> >
> > While creating the dump for repository of size 1.8 GB (revisions 3000+),
> the
> > dump command completes revision 119 and hangs and keep updating the
> dumpfile
> > which grows to 7-8 GBs. dump command is not moving to next revision but
> kept
> > updating the dump file. So, i just closed the putty to stop creating the
> > dump.
>
> Which Linux? And are you using the verndor provided version, or a
> locally compiled one? Subversion 1.6 was last up to 1.6.23: is there
> any reason you can't upgrade that? Ideally with a hotcopy or "rsync"
> based copy of the repository to somehwere else, for testing the copy?
>
> > However, I ran the svnadmin verify which completes revision 119 and take
> > some time to verify revision 120 and complete the verification
> successfully
> > for all the repository revisions.
>
> Hmm. You might consider skipping the svnadmin dump command, and simply
> setting up an svnsync mirror on the Subversion 1.9 server. Even using
> "rsync" to bring the old repository over should let you run a dump and
> reload on the server with Subversion 1.9, as long as there's not some
> other weird corruption with revision 119 or 120.
>
> > Since, there is no error output I have no logs to attach. Please suggest
> the
> > work around for this situation. How can create the dump and migrate to
> > target server. I am planning to use rsync from the server. I will post
> how
> > it goes.
> >
> > I am expecting the workaround to this situation and let me know if this
> is a
> > known issue in SVN 1.6.17. This is my active repository and created the
> dump
> > without stopping the svnserve program in the production server.
>
> It's not one I've seen, but I don't let my repositories get that big.
> And I'd look at revisions 119 and 120 to see if there were erroneous
> commits of huge binary files, in which case I'd want to exclude them
> from the backup.
>
Re: SVN 1.6.17 dump is growing larger than repository size (approx.
more than 10 times)
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Wed, Nov 25, 2015 at 4:25 AM, arun prasath <ge...@gmail.com> wrote:
> Hello Team,
>
> I am creating Subversion 1.6.17 dump for a repository hosted in Linux
> server. SVNSERVE is serving the repository. We are migrating to SVN 1.9.2.
>
> While creating the dump for repository of size 1.8 GB (revisions 3000+), the
> dump command completes revision 119 and hangs and keep updating the dumpfile
> which grows to 7-8 GBs. dump command is not moving to next revision but kept
> updating the dump file. So, i just closed the putty to stop creating the
> dump.
Which Linux? And are you using the verndor provided version, or a
locally compiled one? Subversion 1.6 was last up to 1.6.23: is there
any reason you can't upgrade that? Ideally with a hotcopy or "rsync"
based copy of the repository to somehwere else, for testing the copy?
> However, I ran the svnadmin verify which completes revision 119 and take
> some time to verify revision 120 and complete the verification successfully
> for all the repository revisions.
Hmm. You might consider skipping the svnadmin dump command, and simply
setting up an svnsync mirror on the Subversion 1.9 server. Even using
"rsync" to bring the old repository over should let you run a dump and
reload on the server with Subversion 1.9, as long as there's not some
other weird corruption with revision 119 or 120.
> Since, there is no error output I have no logs to attach. Please suggest the
> work around for this situation. How can create the dump and migrate to
> target server. I am planning to use rsync from the server. I will post how
> it goes.
>
> I am expecting the workaround to this situation and let me know if this is a
> known issue in SVN 1.6.17. This is my active repository and created the dump
> without stopping the svnserve program in the production server.
It's not one I've seen, but I don't let my repositories get that big.
And I'd look at revisions 119 and 120 to see if there were erroneous
commits of huge binary files, in which case I'd want to exclude them
from the backup.
Re: SVN 1.6.17 dump is growing larger than repository size (approx.
more than 10 times)
Posted by arun prasath <ge...@gmail.com>.
I will use this and give a try.
Thank you very much for help information.
regards,
Arun
On Sat, Dec 12, 2015 at 9:12 PM, Andreas Stieger <an...@gmx.de>
wrote:
> On 12/12/15 13:54, arun prasath wrote:
>
>> For -M options - is this is the cache for processing for dump files. do I
>> need to increase the value in MB like -M 1024 to speed up.
>>
>
> Yes this affects processing dump files. This option is documented. It
> increases the size of the memory cache used to reduce redundant operations.
> Give it plenty if you have it.
>
> Andreas
>
Re: SVN 1.6.17 dump is growing larger than repository size (approx.
more than 10 times)
Posted by Andreas Stieger <an...@gmx.de>.
On 12/12/15 13:54, arun prasath wrote:
> For -M options - is this is the cache for processing for dump files.
> do I need to increase the value in MB like -M 1024 to speed up.
Yes this affects processing dump files. This option is documented. It
increases the size of the memory cache used to reduce redundant
operations. Give it plenty if you have it.
Andreas
Re: SVN 1.6.17 dump is growing larger than repository size (approx.
more than 10 times)
Posted by arun prasath <ge...@gmail.com>.
Hi Andreas,
I have used svnsync and it took a while for rev 120 but it went
successfully. There is no space constraint but it was taking the longer
time when compared to other repositories. So for now, i have migrated most
repositories and will see how i am going to migrate the rest in coming
months. rsync is not useful for me since I am moving from one to another
server without altering the existing setup.
For -M options - is this is the cache for processing for dump files. do I
need to increase the value in MB like -M 1024 to speed up.
Sorry for late reply.
regards,
Arun
On Wed, Nov 25, 2015 at 4:31 PM, Andreas Stieger <An...@gmx.de>
wrote:
> Hello,
>
> Yes the dump can be significantly larger than the repository size, due to
> delta encoding used in the on-disk representation. If you examine revision
> 120 I am sure you will fine a particularly complex change that would cause
> this, and your statement about 120 being larger than normal is in line with
> that.
>
> This is not a problem, it is simply expected behaviour. If the size of the
> dumpstream is a problem for you, you can use "--deltas" to reduce size of
> the dump stream.
>
> If things are slow for you, specify a memory cache size (e.g. 1-2 GB if
> you have it) to speed up operations.
>
> -M [--memory-cache-size] ARG : size of the extra in-memory cache in MB
> used to
> minimize redundant operations. Default: 16.
> [used for FSFS repositories only]
>
> You can use svnsync for a more transparent migration. I am not sure how
> rsync should be relevant or useful here when doing dumps anyway.
>
> Andreas
>
> *Gesendet:* Mittwoch, 25. November 2015 um 10:25 Uhr
> *Von:* "arun prasath" <ge...@gmail.com>
> *An:* users@subversion.apache.org
> *Betreff:* SVN 1.6.17 dump is growing larger than repository size
> (approx. more than 10 times)
> Hello Team,
>
> I am creating Subversion 1.6.17 dump for a repository hosted in Linux
> server. SVNSERVE is serving the repository. We are migrating to SVN 1.9.2.
>
> *While creating the dump for repository of size 1.8 GB (revisions 3000+),
> the dump command completes revision 119 and hangs and keep updating the
> dumpfile which grows to 7-8 GBs. dump command is not moving to next
> revision but kept updating the dump file. So, i just closed the putty to
> stop creating the dump.*
>
> However, I ran the svnadmin verify which completes revision 119 and take
> some time to verify revision 120 and complete the verification successfully
> for all the repository revisions.
>
> Since, there is no error output I have no logs to attach. Please suggest
> the work around for this situation. How can create the dump and migrate to
> target server. I am planning to use rsync from the server. I will post how
> it goes.
>
> I am expecting the workaround to this situation and let me know if this is
> a known issue in SVN 1.6.17. This is my active repository and created the
> dump without stopping the svnserve program in the production server.
>
>
> Thanks,
> Arun
>