You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Stümpfig,
Thomas <th...@siemens.com> on 2016/12/07 17:54:45 UTC
svnsync on large files
Hi everybody,
i get an error with svnsync for large files. svnsync: E175002: REPORT request on
The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
The sync is with svn 1.9.5 (Visual SVN Server) over both https
I tried to rdump, svnadmin load incremental (works) and adjust revprop info in rev 0 to the new imported files. But the I got a path already exists for following syncs.
Might not be related to my manipulations but could be the case.
Any Help is appreciated
Mit freundlichen Grüßen
Thomas Stümpfig
Siemens Indusry Software GmbH
Product Lifecycle Management
Global Sales & Services
Franz-Geuer Straße 10
50823 Köln
Franz-Geuer-Straße 10
50823 Köln Tel.: +49 (2153 )9107117
Fax.: +49 (221) 20802699
Mobil: +49 (175) 2205 712
thomas.stuempfig@siemens.com<ma...@siemens.com>
www.siemens.com/plm<http://www.siemens.com/plm>
-----------------
Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564
Re: svnsync on large files
Posted by Pavel Lyalyakin <pa...@visualsvn.com>.
Hello Daniel,
On Thu, Dec 8, 2016 at 1:08 AM, Daniel Shahaf <da...@apache.org> wrote:
>
> Pavel Lyalyakin wrote on Wed, Dec 07, 2016 at 20:59:24 +0300:
> > On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas
> > <th...@siemens.com> wrote:
> > >
> > > The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
> > >
> > > The sync is with svn 1.9.5 (Visual SVN Server) over both https
> >
> >
> > You should use file:/// access to the local repository instead of
> > HTTP(S).
>
> Why? There are perfectly good reasons to use http://localhost/, for
> example, to have a one-stop-shop killswitch for disabling "everything"
> from accessing the repository.
"E175012: Connection timed out" error might occur when you use
`svnsync` and source and target repositories are both accessed by
HTTP(S). My guess was that Thomas was getting this error.
When both URLs are HTTP(S), `svnsync` works unreliable. It creates two
connections to the servers and the local server will drop the
connection on timeout in case there are delays on the side of remote
server. The local server has to wait untill the remote server fully
receives the transaction and commits the data and may drop the
connection if it takes a bit longer. In most cases, such problem could
occur when syncing larger revisions but it also depends on the bandwidth.
Using file:/// access to local repository helps avoid this problem.
> I'm guessing the dump/incremental-load/update-r0-revprops surgery went
> wrong. I would (a) double-check that the revision that had been
> manually synced did in fact sync correctly, (b) review the r0 revprops.
> (Thomas: you can just post the 'proplist -v' here if you aren't sure
> whether it's correct now.)
>
> Also, as Pavel says, we need the full error message.
>
> Cheers,
>
> Daniel
>
> > Here is an example:
> > [[[
> > svnsync sync "https://svn.example.com/svn/MyRepo"
> > "file:///C:/Repositories/MyRepo/"
> > ]]]
> >
> > Does the error occur when you use file:/// access to local repository?
> >
> > > I tried to rdump, svnadmin load incremental (works) and adjust
> > > revprop info in rev 0 to the new imported files. But the I got
> > > a path already exists for following syncs. Might not be related to
> > > my manipulations but could be the case.
--
With best regards,
Pavel Lyalyakin
VisualSVN Team
Re: svnsync on large files
Posted by Daniel Shahaf <da...@apache.org>.
St�mpfig, Thomas wrote on Thu, Dec 08, 2016 at 07:36:10 +0000:
> Hi Daniel,
> you are a guru. I don't know why but it worked!
>
> I guess the magic is in the difference of network transmission of svnsync vs svnrdump / svnadmin load
That's pretty unlikely: svnsync and svnrdump use the very same network
interface. (svn_ra_replay*())
> Are there suggestions from your side to avoid this error in the future?
Check in the master's error log the reason for the E175002. I imagine
some limit was run into, in which case, have httpd.conf exempt svnsync
from that limit.
> Thank you very much for your help!!!
>
You're welcome.
Daniel
RE: svnsync on large files
Posted by Stümpfig,
Thomas <th...@siemens.com>.
Hi Daniel,
you are a guru. I don't know why but it worked!
I guess the magic is in the difference of network transmission of svnsync vs svnrdump / svnadmin load
Are there suggestions from your side to avoid this error in the future?
Thank you very much for your help!!!
Regards
Thomas
-----Original Message-----
From: Daniel Shahaf [mailto:danielsh@apache.org]
Sent: Donnerstag, 8. Dezember 2016 07:20
To: Stümpfig, Thomas (DF PL S&SE DE PSM EAI) <th...@siemens.com>
Cc: Pavel Lyalyakin <pa...@visualsvn.com>; users@subversion.apache.org
Subject: Re: svnsync on large files
Stümpfig, Thomas wrote on Wed, Dec 07, 2016 at 23:32:38 +0000:
> This is the situation of the starting point. (I restored)
The steps should be this:
f() {
token="$USER@$(hostname):${RANDOM}:${RANDOM}"
svn propset --revprop -r0 -- svn:sync-lock $token
svn propget --revprop -r0 --strict svn:sync-lock | fgrep -q -- $token || return 1
svn propset --revprop -r0 svn:sync-currently-copying 54618
svnrdump dump -r54618 --incremental https://mysourceturl/repository >tmpfile
svnadmin load /svnmirror/projektablage <tmpfile
svn propset --revprop -r0 svn:sync-last-merged-rev 54618
svn propdel --revprop -r0 svn:sync-currently-copying
svn propdel --revprop -r0 svn:sync-lock }
Note:
1. Using the same metadata revprops and order as svnsync.
2. The fgrep is to make up for the svn:sync-lock not being taken atomically. (The cmdline client passes NULL for the ORIGINAL_PROPVAL argument of svn_client_revprop_set2().)
3. Using a tmpfile since the dumpfile format isn't truncation-safe.
4. $RANDOM isn't in POSIX; if not available, substitute the first N bytes from /dev/urandom.
5. Make sure to use the correct source URL on the dump command.
6. The hooks may need tweaking.
Cheers,
Daniel
-----------------
Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564
svnsync: fewer round-trips for small revisions (was: Re: svnsync on
large files)
Posted by Daniel Shahaf <da...@apache.org>.
Daniel Shahaf wrote on Thu, Dec 08, 2016 at 06:20:20 +0000:
> The steps should be this:
>
> f() {
> token="$USER@$(hostname):${RANDOM}:${RANDOM}"
> svn propset --revprop -r0 -- svn:sync-lock $token
> svn propget --revprop -r0 --strict svn:sync-lock | fgrep -q -- $token || return 1
> svn propset --revprop -r0 svn:sync-currently-copying 54618
> svnrdump dump -r54618 --incremental https://mysourceturl/repository >tmpfile
> svnadmin load /svnmirror/projektablage <tmpfile
> svn propset --revprop -r0 svn:sync-last-merged-rev 54618
> svn propdel --revprop -r0 svn:sync-currently-copying
> svn propdel --revprop -r0 svn:sync-lock
> }
>
> Note:
>
> 1. Using the same metadata revprops and order as svnsync.
Looking at this again: svn:sync-lock and svn:sync-currently-copying
serve the same purpose; if we could combine them, we'd save two
round-trips per revision, which would be a win for small revisions.
Cheers,
Daniel
Re: svnsync on large files
Posted by Daniel Shahaf <da...@apache.org>.
Stmpfig, Thomas wrote on Wed, Dec 07, 2016 at 23:32:38 +0000:
> This is the situation of the starting point. (I restored)
The steps should be this:
f() {
token="$USER@$(hostname):${RANDOM}:${RANDOM}"
svn propset --revprop -r0 -- svn:sync-lock $token
svn propget --revprop -r0 --strict svn:sync-lock | fgrep -q -- $token || return 1
svn propset --revprop -r0 svn:sync-currently-copying 54618
svnrdump dump -r54618 --incremental https://mysourceturl/repository >tmpfile
svnadmin load /svnmirror/projektablage <tmpfile
svn propset --revprop -r0 svn:sync-last-merged-rev 54618
svn propdel --revprop -r0 svn:sync-currently-copying
svn propdel --revprop -r0 svn:sync-lock
}
Note:
1. Using the same metadata revprops and order as svnsync.
2. The fgrep is to make up for the svn:sync-lock not being taken
atomically. (The cmdline client passes NULL for the ORIGINAL_PROPVAL
argument of svn_client_revprop_set2().)
3. Using a tmpfile since the dumpfile format isn't truncation-safe.
4. $RANDOM isn't in POSIX; if not available, substitute the first
N bytes from /dev/urandom.
5. Make sure to use the correct source URL on the dump command.
6. The hooks may need tweaking.
Cheers,
Daniel
RE: svnsync on large files
Posted by Stümpfig,
Thomas <th...@siemens.com>.
Hi guys
Thank you for your support.
Here is the full error message.
E:\svnbackup>svnsync sync --source-username=myusername --source-password=xxxxx --sync-username= myusername --sync-password=xxxxx https://mydesturl/repository https://mysourceturl/repository
Transmitting file data .....svnsync: E175002: REPORT request on '/svn/projektablage
age/!svn/rev/54618' failed
(I modified the urls for infosec puposes ... )
E:\svnbackup>svn proplist --revprop -r 0 file:///E:/svnmirror/projektablage
Unversioned properties on revision 0:
svn:date
svn:sync-currently-copying
svn:sync-from-url
svn:sync-from-uuid
svn:sync-last-merged-rev
E:\svnbackup>svn propget --revprop -r 0 svn:sync-currently-copying file:///E:/svnmirror/projektablage
54618
E:\svnbackup>svn propget --revprop -r 0 svn:sync-last-merged-rev file:///E:/svnmirror/projektablage
54617
E:\svnbackup>svn info -r HEAD file:///E:/svnmirror/projektablage
Path: projektablage
URL: file:///E:/svnmirror/projektablage
Relative URL: ^/
Repository Root: file:///E:/svnmirror/projektablage
Repository UUID: e75c21af-ffb9-1a4d-8d4e-43c49f6e0dd9
Revision: 54617
Node Kind: directory
Last Changed Author: xxxxxxx
Last Changed Rev: 54617
Last Changed Date: 2016-12-05 11:38:18 +0100 (Mon, 05 Dec 2016)
This is the situation of the starting point. (I restored)
Regards
Thomas
-----Original Message-----
From: Daniel Shahaf [mailto:danielsh@apache.org]
Sent: Mittwoch, 7. Dezember 2016 23:09
To: Pavel Lyalyakin <pa...@visualsvn.com>
Cc: Stümpfig, Thomas (DF PL S&SE DE PSM EAI) <th...@siemens.com>; users@subversion.apache.org
Subject: Re: svnsync on large files
Pavel Lyalyakin wrote on Wed, Dec 07, 2016 at 20:59:24 +0300:
> On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas
> <th...@siemens.com> wrote:
> >
> > The rev to be synced is 2,291,973,385 Bytes large (Size in
> > txn-protorevs directory. Of target mirror)
> >
> > The sync is with svn 1.9.5 (Visual SVN Server) over both https
>
>
> You should use file:/// access to the local repository instead of
> HTTP(S).
Why? There are perfectly good reasons to use http://localhost/, for example, to have a one-stop-shop killswitch for disabling "everything"
from accessing the repository.
I'm guessing the dump/incremental-load/update-r0-revprops surgery went wrong. I would (a) double-check that the revision that had been manually synced did in fact sync correctly, (b) review the r0 revprops.
(Thomas: you can just post the 'proplist -v' here if you aren't sure whether it's correct now.)
Also, as Pavel says, we need the full error message.
Cheers,
Daniel
> Here is an example:
> [[[
> svnsync sync "https://svn.example.com/svn/MyRepo"
> "file:///C:/Repositories/MyRepo/"
> ]]]
>
> Does the error occur when you use file:/// access to local repository?
>
> > I tried to rdump, svnadmin load incremental (works) and adjust
> > revprop info in rev 0 to the new imported files. But the I got a
> > path already exists for following syncs. Might not be related to my
> > manipulations but could be the case.
-----------------
Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564
Re: svnsync on large files
Posted by Daniel Shahaf <da...@apache.org>.
Pavel Lyalyakin wrote on Wed, Dec 07, 2016 at 20:59:24 +0300:
> On Wed, Dec 7, 2016 at 8:54 PM, St�mpfig, Thomas
> <th...@siemens.com> wrote:
> >
> > The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
> >
> > The sync is with svn 1.9.5 (Visual SVN Server) over both https
>
>
> You should use file:/// access to the local repository instead of
> HTTP(S).
Why? There are perfectly good reasons to use http://localhost/, for
example, to have a one-stop-shop killswitch for disabling "everything"
from accessing the repository.
I'm guessing the dump/incremental-load/update-r0-revprops surgery went
wrong. I would (a) double-check that the revision that had been
manually synced did in fact sync correctly, (b) review the r0 revprops.
(Thomas: you can just post the 'proplist -v' here if you aren't sure
whether it's correct now.)
Also, as Pavel says, we need the full error message.
Cheers,
Daniel
> Here is an example:
> [[[
> svnsync sync "https://svn.example.com/svn/MyRepo"
> "file:///C:/Repositories/MyRepo/"
> ]]]
>
> Does the error occur when you use file:/// access to local repository?
>
> > I tried to rdump, svnadmin load incremental (works) and adjust
> > revprop info in rev 0 to the new imported files. But the I got
> > a path already exists for following syncs. Might not be related to
> > my manipulations but could be the case.
RE: svnsync on large files
Posted by Stümpfig,
Thomas <th...@siemens.com>.
Would you suggest the following command
svnsync initialize --allow-non-empty http://svn.example.com/svn-mirror \
NEW-SOURCE-URL
?
-----Original Message-----
From: Pavel Lyalyakin [mailto:pavel.lyalyakin@visualsvn.com]
Sent: Mittwoch, 7. Dezember 2016 18:59
To: Stümpfig, Thomas (DF PL S&SE DE PSM EAI) <th...@siemens.com>; users@subversion.apache.org
Subject: Re: svnsync on large files
Hello Thomas,
On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas <th...@siemens.com> wrote:
>
> Hi everybody,
>
> i get an error with svnsync for large files. svnsync: E175002: REPORT
> request on
It seems to be only the first part of the error message. What's the full wording of the error?
>
> The rev to be synced is 2,291,973,385 Bytes large (Size in
> txn-protorevs directory. Of target mirror)
>
> The sync is with svn 1.9.5 (Visual SVN Server) over both https
You should use file:/// access to the local repository instead of HTTP(S). Here is an example:
[[[
svnsync sync "https://svn.example.com/svn/MyRepo"
"file:///C:/Repositories/MyRepo/"
]]]
Does the error occur when you use file:/// access to local repository?
> I tried to rdump, svnadmin load incremental (works) and adjust revprop info in rev 0 to the new imported files. But the I got a path already exists for following syncs.
> Might not be related to my manipulations but could be the case.
>
>
>
> Any Help is appreciated
>
>
>
> Mit freundlichen Grüßen
>
> Thomas Stümpfig
>
>
>
> Siemens Indusry Software GmbH
>
> Product Lifecycle Management
>
> Global Sales & Services
>
>
>
> Franz-Geuer Straße 10
>
> 50823 Köln
>
> Franz-Geuer-Straße 10
> 50823 Köln Tel.: +49 (2153 )9107117
>
> Fax.: +49 (221) 20802699
>
> Mobil: +49 (175) 2205 712
>
> thomas.stuempfig@siemens.com
>
> www.siemens.com/plm
>
>
>
> -----------------
> Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823
> Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban
> August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht:
> Amtsgericht Köln, HRB 84564
--
With best regards,
Pavel Lyalyakin
VisualSVN Team
-----------------
Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564
Re: svnsync on large files
Posted by Pavel Lyalyakin <pa...@visualsvn.com>.
Hello Thomas,
On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas
<th...@siemens.com> wrote:
>
> Hi everybody,
>
> i get an error with svnsync for large files. svnsync: E175002: REPORT request on
It seems to be only the first part of the error message. What's the
full wording of the error?
>
> The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
>
> The sync is with svn 1.9.5 (Visual SVN Server) over both https
You should use file:/// access to the local repository instead of
HTTP(S). Here is an example:
[[[
svnsync sync "https://svn.example.com/svn/MyRepo"
"file:///C:/Repositories/MyRepo/"
]]]
Does the error occur when you use file:/// access to local repository?
> I tried to rdump, svnadmin load incremental (works) and adjust revprop info in rev 0 to the new imported files. But the I got a path already exists for following syncs.
> Might not be related to my manipulations but could be the case.
>
>
>
> Any Help is appreciated
>
>
>
> Mit freundlichen Grüßen
>
> Thomas Stümpfig
>
>
>
> Siemens Indusry Software GmbH
>
> Product Lifecycle Management
>
> Global Sales & Services
>
>
>
> Franz-Geuer Straße 10
>
> 50823 Köln
>
> Franz-Geuer-Straße 10
> 50823 Köln Tel.: +49 (2153 )9107117
>
> Fax.: +49 (221) 20802699
>
> Mobil: +49 (175) 2205 712
>
> thomas.stuempfig@siemens.com
>
> www.siemens.com/plm
>
>
>
> -----------------
> Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564
--
With best regards,
Pavel Lyalyakin
VisualSVN Team