You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Butler, Stephen" <St...@hamburgsud.com> on 2014/07/01 19:45:59 UTC
Subversion 1.8 freezes during long updates
Hi folks,
I'm trying to help users in our corporate network. They currently use Subversion 1.7 + neon (via TortoiseSVN) because of errors in checkout/update using Subversion 1.8. During a long checkout or update, TortoiseSVN 1.8 freezes after downloading a few hundred MBs of data. The svn command-line client does the same.
To get more information, I compiled Subversion (within the TortoiseSVN branches/1.8.x tree) with serf tracing turned up to eleven (SSL_MSG_VERBOSE, etc). Then the errors no longer occur! Apparently the extra output changes the timing somehow. But performance is of course very bad. Less tracing recreates the errors, but without revealing a smoking gun.
I attached the Visual Studio debugger to a frozen cmdline client. It seemed to be looping through impl_pollset_poll (in apr/poll/unix/select.c) without receiving anything.
On the Subversion server, the last message in the Apache access log is a successful file download (matching the last line of client output). The Apache error log had nothing new. The network has lots of firewalls and transparent proxies, as was the case in a problem that Branko recently solved:
http://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/%3C52D6D944.8060308%40wandisco.com%3E
Should I try to corner the network admins, or is there a client/server troubleshooting step I've overlooked?
Thanks for any insights,
Steve Butler
_____________________________________________
Hamburg Südamerikanische Dampfschifffarts-Gesellschaft KG
Registered Office: Hamburg, Germany
Commercial Register: Amtsgericht Hamburg HRA 59448
Re: Subversion 1.8 freezes during long updates
Posted by Branko Čibej <br...@wandisco.com>.
On 02.07.2014 15:08, Ivan Zhakov wrote:
> On 2 July 2014 17:05, Butler, Stephen <St...@hamburgsud.com> wrote:
>>> -----Original Message-----
>>> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
>>> Sent: Wednesday, July 02, 2014 14:31
>>> To: Butler, Stephen
>>> Cc: users@subversion.apache.org
>>> Subject: Re: Subversion 1.8 freezes during long updates
>>>
>>> On 1 July 2014 21:45, Butler, Stephen <St...@hamburgsud.com> wrote:
>>>> Hi folks,
>>>>
>>>> I'm trying to help users in our corporate network. They currently
>>>> use Subversion 1.7 + neon (via TortoiseSVN) because of errors in
>>>> checkout/update using Subversion 1.8. During a long checkout or
>>>> update, TortoiseSVN 1.8 freezes after downloading a few hundred MBs
>>>> of data. The svn command-line client does the same.
>>>>
>>>> To get more information, I compiled Subversion (within the
>>>> TortoiseSVN branches/1.8.x tree) with serf tracing turned up to eleven
>>>> (SSL_MSG_VERBOSE, etc). Then the errors no longer occur! Apparently
>>>> the extra output changes the timing somehow. But performance is of
>>>> course very bad. Less tracing recreates the errors, but without
>>>> revealing a smoking gun.
>>>>
>>>> I attached the Visual Studio debugger to a frozen cmdline client. It
>>>> seemed to be looping through impl_pollset_poll (in apr/poll/unix/select.c)
>>>> without receiving anything.
>>>>
>>>> On the Subversion server, the last message in the Apache access log is a
>>>> successful file download (matching the last line of client output). The
>>>> Apache error log had nothing new. The network has lots of firewalls and
>>>> transparent proxies, as was the case in a problem that Branko recently
>>>> solved:
>>>>
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/
>>>> %3C52D6D944.8060308%40wandisco.com%3E
>>>>
>>>> Should I try to corner the network admins, or is there a client/server
>>>> troubleshooting step I've overlooked?
>>>>
>>>> Thanks for any insights,
>>>>
>>> Hi Steve,
>>>
>>> 1. Does the issue reproduce if you force bulk updates using
>>> '--config-option=servers:global:http-bulk-updates=yes' command line option
>>> or 'SVNAllowBulkUpdates prefer' on the server side?
>> Setting either option seems to solve the problem completely. I should have
>> read the release notes more carefully!
>>
>>> 2. Do you have antivirus installed on client computers?
>> FWIW, yes, but in this case it was "mostly harmless".
>>
> It still will be interesting to investigate the issue. Could please
> specify what antivirus/firewall used on client computer? Does it have
> any active/filtering protection?
We had a case recently, discussed on this list (and verified off-list),
where a Cisco ASA with stateful packet inspection turned on would
silently close connections to the server in exactly this sort of case;
turns out to be a long-standing bug in the IOS firmware that Cisco won't
acknowledge.
I'm not saying this is the case here, just pointing out that we do have
evidence of firewall software or appliances being fundamentally broken.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: Subversion 1.8 freezes during long updates
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 2 July 2014 17:05, Butler, Stephen <St...@hamburgsud.com> wrote:
>> -----Original Message-----
>> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
>> Sent: Wednesday, July 02, 2014 14:31
>> To: Butler, Stephen
>> Cc: users@subversion.apache.org
>> Subject: Re: Subversion 1.8 freezes during long updates
>>
>> On 1 July 2014 21:45, Butler, Stephen <St...@hamburgsud.com> wrote:
>> > Hi folks,
>> >
>> > I'm trying to help users in our corporate network. They currently
>> > use Subversion 1.7 + neon (via TortoiseSVN) because of errors in
>> > checkout/update using Subversion 1.8. During a long checkout or
>> > update, TortoiseSVN 1.8 freezes after downloading a few hundred MBs
>> > of data. The svn command-line client does the same.
>> >
>> > To get more information, I compiled Subversion (within the
>> > TortoiseSVN branches/1.8.x tree) with serf tracing turned up to eleven
>> > (SSL_MSG_VERBOSE, etc). Then the errors no longer occur! Apparently
>> > the extra output changes the timing somehow. But performance is of
>> > course very bad. Less tracing recreates the errors, but without
>> > revealing a smoking gun.
>> >
>> > I attached the Visual Studio debugger to a frozen cmdline client. It
>> > seemed to be looping through impl_pollset_poll (in apr/poll/unix/select.c)
>> > without receiving anything.
>> >
>> > On the Subversion server, the last message in the Apache access log is a
>> > successful file download (matching the last line of client output). The
>> > Apache error log had nothing new. The network has lots of firewalls and
>> > transparent proxies, as was the case in a problem that Branko recently
>> > solved:
>> >
>> >
>> > http://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/
>> > %3C52D6D944.8060308%40wandisco.com%3E
>> >
>> > Should I try to corner the network admins, or is there a client/server
>> > troubleshooting step I've overlooked?
>> >
>> > Thanks for any insights,
>> >
>> Hi Steve,
>>
>> 1. Does the issue reproduce if you force bulk updates using
>> '--config-option=servers:global:http-bulk-updates=yes' command line option
>> or 'SVNAllowBulkUpdates prefer' on the server side?
>
> Setting either option seems to solve the problem completely. I should have
> read the release notes more carefully!
>
>>
>> 2. Do you have antivirus installed on client computers?
>
> FWIW, yes, but in this case it was "mostly harmless".
>
It still will be interesting to investigate the issue. Could please
specify what antivirus/firewall used on client computer? Does it have
any active/filtering protection?
--
Ivan Zhakov
RE: Subversion 1.8 freezes during long updates
Posted by "Butler, Stephen" <St...@hamburgsud.com>.
> -----Original Message-----
> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
> Sent: Wednesday, July 02, 2014 14:31
> To: Butler, Stephen
> Cc: users@subversion.apache.org
> Subject: Re: Subversion 1.8 freezes during long updates
>
> On 1 July 2014 21:45, Butler, Stephen <St...@hamburgsud.com> wrote:
> > Hi folks,
> >
> > I'm trying to help users in our corporate network. They currently
> > use Subversion 1.7 + neon (via TortoiseSVN) because of errors in
> > checkout/update using Subversion 1.8. During a long checkout or
> > update, TortoiseSVN 1.8 freezes after downloading a few hundred MBs
> > of data. The svn command-line client does the same.
> >
> > To get more information, I compiled Subversion (within the
> > TortoiseSVN branches/1.8.x tree) with serf tracing turned up to eleven
> > (SSL_MSG_VERBOSE, etc). Then the errors no longer occur! Apparently
> > the extra output changes the timing somehow. But performance is of
> > course very bad. Less tracing recreates the errors, but without
> > revealing a smoking gun.
> >
> > I attached the Visual Studio debugger to a frozen cmdline client. It
> > seemed to be looping through impl_pollset_poll (in apr/poll/unix/select.c)
> > without receiving anything.
> >
> > On the Subversion server, the last message in the Apache access log is a
> > successful file download (matching the last line of client output). The
> > Apache error log had nothing new. The network has lots of firewalls and
> > transparent proxies, as was the case in a problem that Branko recently
> > solved:
> >
> >
> > http://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/
> > %3C52D6D944.8060308%40wandisco.com%3E
> >
> > Should I try to corner the network admins, or is there a client/server
> > troubleshooting step I've overlooked?
> >
> > Thanks for any insights,
> >
> Hi Steve,
>
> 1. Does the issue reproduce if you force bulk updates using
> '--config-option=servers:global:http-bulk-updates=yes' command line option
> or 'SVNAllowBulkUpdates prefer' on the server side?
Setting either option seems to solve the problem completely. I should have
read the release notes more carefully!
>
> 2. Do you have antivirus installed on client computers?
FWIW, yes, but in this case it was "mostly harmless".
>
> --
> Ivan Zhakov
Thank you,
Steve Butler
_____________________________________________
Hamburg Südamerikanische Dampfschifffarts-Gesellschaft KG
Registered Office: Hamburg, Germany
Commercial Register: Amtsgericht Hamburg HRA 59448
Re: Subversion 1.8 freezes during long updates
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 1 July 2014 21:45, Butler, Stephen <St...@hamburgsud.com> wrote:
> Hi folks,
>
> I'm trying to help users in our corporate network. They currently use Subversion 1.7 + neon (via TortoiseSVN) because of errors in checkout/update using Subversion 1.8. During a long checkout or update, TortoiseSVN 1.8 freezes after downloading a few hundred MBs of data. The svn command-line client does the same.
>
> To get more information, I compiled Subversion (within the TortoiseSVN branches/1.8.x tree) with serf tracing turned up to eleven (SSL_MSG_VERBOSE, etc). Then the errors no longer occur! Apparently the extra output changes the timing somehow. But performance is of course very bad. Less tracing recreates the errors, but without revealing a smoking gun.
>
> I attached the Visual Studio debugger to a frozen cmdline client. It seemed to be looping through impl_pollset_poll (in apr/poll/unix/select.c) without receiving anything.
>
> On the Subversion server, the last message in the Apache access log is a successful file download (matching the last line of client output). The Apache error log had nothing new. The network has lots of firewalls and transparent proxies, as was the case in a problem that Branko recently solved:
>
> http://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/%3C52D6D944.8060308%40wandisco.com%3E
>
> Should I try to corner the network admins, or is there a client/server troubleshooting step I've overlooked?
>
> Thanks for any insights,
>
Hi Steve,
1. Does the issue reproduce if you force bulk updates using
'--config-option=servers:global:http-bulk-updates=yes' command line
option or 'SVNAllowBulkUpdates prefer' on the server side?
2. Do you have antivirus installed on client computers?
--
Ivan Zhakov
Re: Subversion 1.8 freezes during long updates
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Wed, Jul 2, 2014 at 7:49 AM, Butler, Stephen
<St...@hamburgsud.com> wrote:
>> -----Original Message-----
>> From: Nico Kadel-Garcia [mailto:nkadel@gmail.com]
>> Sent: Wednesday, July 02, 2014 1:11
>> To: Butler, Stephen
>> Cc: users@subversion.apache.org
>> Subject: Re: Subversion 1.8 freezes during long updates
>>
>> On Tue, Jul 1, 2014 at 1:45 PM, Butler, Stephen <St...@hamburgsud.com> wrote:
>> > Hi folks,
>> >
>> > I'm trying to help users in our corporate network. They currently use
>> > Subversion 1.7 + neon (via TortoiseSVN) because of errors in
>> > checkout/update using Subversion 1.8. During a long checkout or update,
>> > TortoiseSVN 1.8 freezes after downloading a few hundred MBs of data.
>> > The svn command-line client does the same.
>>
>> Are your users on CIFS shares, or other network shares? CIFS has turned out
>> to be pretty chatty, and I've certainly seen issues with significantly
>> earlier versions of Subversion when directories willed with hundreds or
>> thousands of files made it perform like a dog or even overwhelmed it. Do
>> your users benefit from making their working copies on a local disk, then
>> simply replicating it to a CIFS share?
>
> We don't use shared drives for Subversion. The working copies are all
> on the local disk (in Windows 7).
Well, rats. It was a thought....
RE: Subversion 1.8 freezes during long updates
Posted by "Butler, Stephen" <St...@hamburgsud.com>.
> -----Original Message-----
> From: Nico Kadel-Garcia [mailto:nkadel@gmail.com]
> Sent: Wednesday, July 02, 2014 1:11
> To: Butler, Stephen
> Cc: users@subversion.apache.org
> Subject: Re: Subversion 1.8 freezes during long updates
>
> On Tue, Jul 1, 2014 at 1:45 PM, Butler, Stephen <St...@hamburgsud.com> wrote:
> > Hi folks,
> >
> > I'm trying to help users in our corporate network. They currently use
> > Subversion 1.7 + neon (via TortoiseSVN) because of errors in
> > checkout/update using Subversion 1.8. During a long checkout or update,
> > TortoiseSVN 1.8 freezes after downloading a few hundred MBs of data.
> > The svn command-line client does the same.
>
> Are your users on CIFS shares, or other network shares? CIFS has turned out
> to be pretty chatty, and I've certainly seen issues with significantly
> earlier versions of Subversion when directories willed with hundreds or
> thousands of files made it perform like a dog or even overwhelmed it. Do
> your users benefit from making their working copies on a local disk, then
> simply replicating it to a CIFS share?
We don't use shared drives for Subversion. The working copies are all
on the local disk (in Windows 7).
>
> > To get more information, I compiled Subversion (within the TortoiseSVN
> > branches/1.8.x tree) with serf tracing turned up to eleven (SSL_MSG_VERBOSE,
> > etc). Then the errors no longer occur! Apparently the extra output
> > changes the timing somehow. But performance is of course very bad. Less
> > tracing recreates the errors, but without revealing a smoking gun.
>
> See above. Did you download *to the same location*?
Yes, to directories on the local C: drive.
Thanks,
Steve Butler
_____________________________________________
Hamburg Südamerikanische Dampfschifffarts-Gesellschaft KG
Registered Office: Hamburg, Germany
Commercial Register: Amtsgericht Hamburg HRA 59448
Re: Subversion 1.8 freezes during long updates
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Tue, Jul 1, 2014 at 1:45 PM, Butler, Stephen
<St...@hamburgsud.com> wrote:
> Hi folks,
>
> I'm trying to help users in our corporate network. They currently use Subversion 1.7 + neon (via TortoiseSVN) because of errors in checkout/update using Subversion 1.8. During a long checkout or update, TortoiseSVN 1.8 freezes after downloading a few hundred MBs of data. The svn command-line client does the same.
Are your users on CIFS shares, or other network shares? CIFS has
turned out to be pretty chatty, and I've certainly seen issues with
significantly earlier versions of Subversion when directories willed
with hundreds or thousands of files made it perform like a dog or even
overwhelmed it. Do your users benefit from making their working copies
on a local disk, then simply replicating it to a CIFS share?
> To get more information, I compiled Subversion (within the TortoiseSVN branches/1.8.x tree) with serf tracing turned up to eleven (SSL_MSG_VERBOSE, etc). Then the errors no longer occur! Apparently the extra output changes the timing somehow. But performance is of course very bad. Less tracing recreates the errors, but without revealing a smoking gun.
See above. Did you download *to the same location*?