You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Stümpfig,
Thomas <th...@siemens.com> on 2015/03/04 15:26:49 UTC
Update Error over VPN
Hi everybody,
we have a strange issue here with svn 1.8.11
upon svn update with we get a REPORT require on '/svn/..../!svn/me' failed
but only on a specific WAN connection.
With a lan connection everything is fine. Updates to the same working folder are ok.
The WAN connection is a Juniper VPN connection (ipsec, nat traversal compatible) over a carrier grade nat internet connection.
Any idea what is going wrong here?
Regards
Thomas
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
RE: Update Error over VPN
Posted by Stümpfig,
Thomas <th...@siemens.com>.
Hi all,
i'll try to get a Wireshark extract that covers the error. The problem I have with it, is that Wireshark on the server will capture a huge amount of data. We have ~20-100 parallel updates on the server. I observed peaks with 250 http sessions. Capturing the interface for a long time will be difficult. (I am not very good with Wireshark capture filters). How can I produce a serf debug output? The platform is Windows Server 2008R2 with current Microsoft patches. I am not aware of the exact configuration of the network. Siemens is a big company, and the hardware over the WAN is also unknown to me. :-(. I am just the guy who cares about the svn server.
Regards
Thomas
-----Original Message-----
From: Bert Huijben [mailto:bert@qqmail.nl]
Sent: Dienstag, 10. März 2015 14:33
To: 'Alfred von Campe'
Cc: 'Branko Čibej'; Stümpfig, Thomas; users@subversion.apache.org; 'Nico Kadel-Garcia'
Subject: RE: Update Error over VPN
> -----Original Message-----
> From: Nico Kadel-Garcia [mailto:nkadel@gmail.com]
> Sent: dinsdag 10 maart 2015 13:59
> To: Alfred von Campe
> Cc: Branko Čibej; Stümpfig, Thomas; users@subversion.apache.org
> Subject: Re: Update Error over VPN
>
> On Tue, Mar 10, 2015 at 7:16 AM, Alfred von Campe <alfred@von-
> campe.com> wrote:
> > I was going to chime in earlier, but something in Brane’s response
> > finally put me over the edge. We are having a very similar error
> > when updating
> over
> > VPN (the problem does not occur when in the office):
> >
> > Summary of conflicts:
> > Text conflicts: 1
> > svn: E175002: REPORT request on '/hepd/Shelby/!svn/me’ failed
This error can be triggered (without further details) if our serf HTTP library decides to cancel the request... But for that it would need a good reason.
It would be very interesting to know what caused this... A network trace (or serf debug output) would be very valuable. What platform do you see these problems on?
In another somehow related case that I'm investigating (strict firewall over a VPN link) it really helped to switch to the Subversion 1.9 pre-beta version... It would be very interesting to know if this has the same effect for you.
Bert
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
RE: Update Error over VPN
Posted by Stümpfig,
Thomas <th...@siemens.com>.
I forgot to mention, indeed, the full path in the error message indicates that large avi files were involved in the update operation.
-----Original Message-----
From: Alfred von Campe [mailto:alfred@von-campe.com]
Sent: Dienstag, 10. März 2015 15:59
To: Bert Huijben
Cc: Branko Čibej; Stümpfig, Thomas; users@subversion.apache.org; Nico Kadel-Garcia
Subject: Re: Update Error over VPN
Bert:
> This error can be triggered (without further details) if our serf HTTP library decides to cancel the request... But for that it would need a good reason.
>
> It would be very interesting to know what caused this... A network trace (or serf debug output) would be very valuable. What platform do you see these problems on?
The client and server are both running Linux: CentOS 6.6 with WANdisco V1.8.11 binaries on the client and Red Hat 5.8 with compiled from source 1.7.5 binaries on the server. How to I create a network trace and/or serf debug output?
> In another somehow related case that I'm investigating (strict firewall over a VPN link) it really helped to switch to the Subversion 1.9 pre-beta version... It would be very interesting to know if this has the same effect for you.
I can ask the user to give this a try. Do you have binaries for CentOS 6.6 available?
Alfred
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
Re: Update Error over VPN
Posted by Alfred von Campe <al...@von-campe.com>.
Bert:
> This error can be triggered (without further details) if our serf HTTP library decides to cancel the request... But for that it would need a good reason.
>
> It would be very interesting to know what caused this... A network trace (or serf debug output) would be very valuable. What platform do you see these problems on?
The client and server are both running Linux: CentOS 6.6 with WANdisco V1.8.11 binaries on the client and Red Hat 5.8 with compiled from source 1.7.5 binaries on the server. How to I create a network trace and/or serf debug output?
> In another somehow related case that I'm investigating (strict firewall over a VPN link) it really helped to switch to the Subversion 1.9 pre-beta version... It would be very interesting to know if this has the same effect for you.
I can ask the user to give this a try. Do you have binaries for CentOS 6.6 available?
Alfred
RE: Update Error over VPN
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Nico Kadel-Garcia [mailto:nkadel@gmail.com]
> Sent: dinsdag 10 maart 2015 13:59
> To: Alfred von Campe
> Cc: Branko Čibej; Stümpfig, Thomas; users@subversion.apache.org
> Subject: Re: Update Error over VPN
>
> On Tue, Mar 10, 2015 at 7:16 AM, Alfred von Campe <alfred@von-
> campe.com> wrote:
> > I was going to chime in earlier, but something in Brane’s response finally
> > put me over the edge. We are having a very similar error when updating
> over
> > VPN (the problem does not occur when in the office):
> >
> > Summary of conflicts:
> > Text conflicts: 1
> > svn: E175002: REPORT request on '/hepd/Shelby/!svn/me’ failed
This error can be triggered (without further details) if our serf HTTP library decides to cancel the request... But for that it would need a good reason.
It would be very interesting to know what caused this... A network trace (or serf debug output) would be very valuable. What platform do you see these problems on?
In another somehow related case that I'm investigating (strict firewall over a VPN link) it really helped to switch to the Subversion 1.9 pre-beta version... It would be very interesting to know if this has the same effect for you.
Bert
Re: Update Error over VPN
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Tue, Mar 10, 2015 at 7:16 AM, Alfred von Campe <al...@von-campe.com> wrote:
> I was going to chime in earlier, but something in Brane’s response finally
> put me over the edge. We are having a very similar error when updating over
> VPN (the problem does not occur when in the office):
>
> Summary of conflicts:
> Text conflicts: 1
> svn: E175002: REPORT request on '/hepd/Shelby/!svn/me’ failed
>
> The issue occurs while trying to do a merge, and it happens at the point in
> time when a few large binaries (which take noticeably longer when doing an
> “svn up” because of their size) are processed. The reason I am replying is
> that we do have a load balancer in front of our pair of Subversion servers.
> I hope this helps to shed some light on this issue.
>
> Alfred
So, tell the load balancer to talk to only one node for a while and
see what happens? And I *assume* you have "sticky sessions" set up in
your load balancer?
Re: Update Error over VPN
Posted by Alfred von Campe <al...@von-campe.com>.
I was going to chime in earlier, but something in Brane’s response finally put me over the edge. We are having a very similar error when updating over VPN (the problem does not occur when in the office):
Summary of conflicts:
Text conflicts: 1
svn: E175002: REPORT request on '/hepd/Shelby/!svn/me’ failed
The issue occurs while trying to do a merge, and it happens at the point in time when a few large binaries (which take noticeably longer when doing an “svn up” because of their size) are processed. The reason I am replying is that we do have a load balancer in front of our pair of Subversion servers. I hope this helps to shed some light on this issue.
Alfred
> On Mar 10, 2015, at 2:09, Branko Čibej <br...@wandisco.com> wrote:
>
> On 10.03.2015 01:09, Stümpfig, Thomas wrote:
>>
>> Hi Bert, All
>> “a REPORT require on ‘/svn/…./!svn/me’ failed “ this is the only message my colleague is getting. Same message with command line tools ,
>
> I'm sorry, but there's no string like that in the Subversion 1.8.x source code. So this can't be the exact error message.
>
>> . Server side (VisualSVN) error cannot exactly be linked to the users action –perhaps this could be an enhancement to mod_dav_svn – send a transaction id in the http header and log it. The server Error very probably is: "Error writing base64 data: APR does not understand this error code [500, #620018] (We have ~300 commits per day and many more reading operations.
>> The error occurs only in WAN circumstances, and only on large updates (>2GB). In LAN environment everything is ok. The user in question has got a 100Mb/s DOCIS3.0 line IPV6 based IPV4 carrier grade nat accessing the companies network via juniper client.
>
> The error code (620018) is the canonical APR error code for "connection aborted", but the question is, what is terminating the connection (it's also strange that "APR does not understand that error code," but that's likely a different issue).
>
> So, do you have a (hardware or software) proxy/load balancer or similar on the entry point from the WAN? If yes, does it have logs you could look at?
>
>> Meanwhile I read about apache timeout,… and set this to 1800 -> 30min… I hope this fixes the issues …. I’ll keep this list informed
>
> FWIW, the kind of update you're talking about shouldn't take more than 10 minutes in the worst case over a 100Mb/s link.
>
> -- Brane
>
Re: Update Error over VPN
Posted by Branko Čibej <br...@wandisco.com>.
On 10.03.2015 01:09, Stümpfig, Thomas wrote:
>
>
>
> Hi Bert, All
>
> “a REPORT require on ‘/svn/…./!svn/me’ failed“ this is the only
> message my colleague is getting. Same message with command line tools ,
>
I'm sorry, but there's no string like that in the Subversion 1.8.x
source code. So this can't be the exact error message.
> . Server side (VisualSVN) error cannot exactly be linked to the users
> action –perhaps this could be an enhancement to mod_dav_svn – send a
> transaction id in the http header and log it. The server Error very
> probably is: "Error writing base64 data: APR does not understand this
> error code [500, #620018] (We have ~300 commits per day and many more
> reading operations.
> The error occurs only in WAN circumstances, and only on large updates
> (>2GB). In LAN environment everything is ok. The user in question has
> got a 100Mb/s DOCIS3.0 line IPV6 based IPV4 carrier grade nat
> accessing the companies network via juniper client.
>
The error code (620018) is the canonical APR error code for "connection
aborted", but the question is, what is terminating the connection (it's
also strange that "APR does not understand that error code," but that's
likely a different issue).
So, do you have a (hardware or software) proxy/load balancer or similar
on the entry point from the WAN? If yes, does it have logs you could
look at?
> Meanwhile I read about apache timeout,… and set this to 1800 ->
> 30min… I hope this fixes the issues …. I’ll keep this list informed
>
FWIW, the kind of update you're talking about shouldn't take more than
10 minutes in the worst case over a 100Mb/s link.
-- Brane
RE: Update Error over VPN
Posted by Stümpfig,
Thomas <th...@siemens.com>.
Hi Bert, All
"a REPORT require on '/svn/..../!svn/me' failed " this is the only message my colleague is getting. Same message with command line tools ,
. Server side (VisualSVN) error cannot exactly be linked to the users action -perhaps this could be an enhancement to mod_dav_svn - send a transaction id in the http header and log it. The server Error very probably is: "Error writing base64 data: APR does not understand this error code [500, #620018] (We have ~300 commits per day and many more reading operations.
The error occurs only in WAN circumstances, and only on large updates (>2GB). In LAN environment everything is ok. The user in question has got a 100Mb/s DOCIS3.0 line IPV6 based IPV4 carrier grade nat accessing the companies network via juniper client.
Meanwhile I read about apache timeout,... and set this to 1800 -> 30min... I hope this fixes the issues .... I'll keep this list informed
regards
Thomas
From: Bert Huijben [mailto:bert@qqmail.nl]
Sent: Sonntag, 8. März 2015 13:20
To: Stümpfig, Thomas; users@subversion.apache.org
Subject: RE: Update Error over VPN
Hi Thomas,
It is kind of hard to answer questions on what might have failed if you only post a specific part of a failure.
Usually the error would contain a lot more details, such as how it failed; possibly network related errors. You just note that it failed... and then try to ask us why.
In general the Subversion client should have told you why, and then we might be able to help you resolve why it failed... Now the only thing we can do is note that it failed... Which is something you already knew.
The E0001234 codes provide valuable information and in many cases allow you to google for the result, but in many cases there are even more helpful hints in the error messages.
Bert
From: Stümpfig, Thomas [mailto:thomas.stuempfig@siemens.com]
Sent: zondag 8 maart 2015 10:12
To: Stümpfig, Thomas; users@subversion.apache.org<ma...@subversion.apache.org>
Subject: RE: Update Error over VPN
Nobody has any suggestions? What are the influences on networkside other than a correct http(s)/tcp/ip transmission.
Where can I get info about timouts etc..
From: Stümpfig, Thomas [mailto:thomas.stuempfig@siemens.com]
Sent: Mittwoch, 4. März 2015 15:27
To: users@subversion.apache.org<ma...@subversion.apache.org>
Subject: Update Error over VPN
Hi everybody,
we have a strange issue here with svn 1.8.11
upon svn update with we get a REPORT require on '/svn/..../!svn/me' failed
but only on a specific WAN connection.
With a lan connection everything is fine. Updates to the same working folder are ok.
The WAN connection is a Juniper VPN connection (ipsec, nat traversal compatible) over a carrier grade nat internet connection.
Any idea what is going wrong here?
Regards
Thomas
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
RE: Update Error over VPN
Posted by Bert Huijben <be...@qqmail.nl>.
Hi Thomas,
It is kind of hard to answer questions on what might have failed if you only
post a specific part of a failure.
Usually the error would contain a lot more details, such as how it failed;
possibly network related errors. You just note that it failed
and then try
to ask us why.
In general the Subversion client should have told you why, and then we might
be able to help you resolve why it failed
Now the only thing we can do is
note that it failed
Which is something you already knew.
The E0001234 codes provide valuable information and in many cases allow you
to google for the result, but in many cases there are even more helpful
hints in the error messages.
Bert
From: Stümpfig, Thomas [mailto:thomas.stuempfig@siemens.com]
Sent: zondag 8 maart 2015 10:12
To: Stümpfig, Thomas; users@subversion.apache.org
Subject: RE: Update Error over VPN
Nobody has any suggestions? What are the influences on networkside other
than a correct http(s)/tcp/ip transmission.
Where can I get info about timouts etc..
From: Stümpfig, Thomas [mailto:thomas.stuempfig@siemens.com]
Sent: Mittwoch, 4. März 2015 15:27
To: users@subversion.apache.org <ma...@subversion.apache.org>
Subject: Update Error over VPN
Hi everybody,
we have a strange issue here with svn 1.8.11
upon svn update with we get a REPORT require on /svn/
./!svn/me failed
but only on a specific WAN connection.
With a lan connection everything is fine. Updates to the same working folder
are ok.
The WAN connection is a Juniper VPN connection (ipsec, nat traversal
compatible) over a carrier grade nat internet connection.
Any idea what is going wrong here?
Regards
Thomas
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10,
50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht:
Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry
Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln;
Registergericht: Amtsgericht Köln, HRB 70858
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10,
50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht:
Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry
Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln;
Registergericht: Amtsgericht Köln, HRB 70858
RE: Update Error over VPN
Posted by Stümpfig,
Thomas <th...@siemens.com>.
Nobody has any suggestions? What are the influences on networkside other than a correct http(s)/tcp/ip transmission.
Where can I get info about timouts etc..
From: Stümpfig, Thomas [mailto:thomas.stuempfig@siemens.com]
Sent: Mittwoch, 4. März 2015 15:27
To: users@subversion.apache.org
Subject: Update Error over VPN
Hi everybody,
we have a strange issue here with svn 1.8.11
upon svn update with we get a REPORT require on '/svn/..../!svn/me' failed
but only on a specific WAN connection.
With a lan connection everything is fine. Updates to the same working folder are ok.
The WAN connection is a Juniper VPN connection (ipsec, nat traversal compatible) over a carrier grade nat internet connection.
Any idea what is going wrong here?
Regards
Thomas
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858
-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858