svn client 1.8.1: Tries to log-in forever, hammering the server, eating memory and cpu

[Elias Gerber <eg...@viscomvisual.com>]

Hello everyone

We have the following situation that is working fine with svn clients <= 
v1.7 in a windows-environment: Our SVN-Server is a VisualSVN-Server, 
with integration of the Windows Authentication: Clients that are within 
the windows-domain authenticate automatically using their 
windows-credentials. Clients that are not part of the domain fail to 
log-in using the windows-credentials (as svn.exe sends the local 
user-information, like 'machine\username'), then svn.exe asks the user 
to enter a username and password. Clients then enter their 
domain-username ('domain\user') and the password, so they can authenticate.

Since svn 1.8.x we experience the following:
Clients that are part of the domain can log in without any problems. 
Clients that are not part of the domain try to log-in forever using the 
local account. The server rejects those credentials - it only knows 
about domain-users, not about local users. The client does not stop to 
log in then, but tries forever: At the server we see thousands of 
rejected log-in requests, at the client we see that it uses one core of 
the cpu 100% and eats more and more memory and never (? - I killed it 
when it reached more than one Gigabyte of memory) stops.

Why do I think it is a problem of the svn client:

With svn clients <= 1.7 this scenario works against VisualSVN-Server 
2.5.x (using subversion 1.7.11) and against VisualSVNS-Server 2.6.2 and 
2.6.3 (using subversion 1.8.1): If log-in using the windows-credentials 
fails, the svn-client asks for a username and password.

With svn clients >= 1.8 the problem occurs against VisualSVN-Server 
2.5.x (subversion 1.7.11) and VisualSVN-Server 2.6.2/3 (subversion 
1.8.1), so I would like to blame the client ;)

My try to test the svn-client was a simple checkout, in the form:

svn.exe --no-auth-cache checkout https://repository/svn/repo c:\tmp

For those who know VisualSVN Server: In the server we've selected 'Use 
Windows authentication' and then checked both:
- 'Basis authentication' (Users are requested to enter their username 
and password in order to authenticate to the server) and
- 'Integrated Windows Authentication' (Windows credentials are 
automatically used to authenticate to the server).

If we remove 'Integrated Windows Authentication' every svn client, 
regardless if it is in the domain or not, asks for a username/password 
and things work fine with svn.exe v1.8.1
Passing --username and --password to svn.exe does not help: If I test using

svn.exe --no-auth-cache --username user --password pass checkout 
https://repository/svn/repo c:\tmp

I see at the server that svn.exe does not send the passed user/pass, but 
still tries to log in with the local windows credentials (regardless if 
the machine is part of the domain or not - the server receives as 
username 'machine\user' or 'domain\user').

Is it possible that something like this is going on:
The client queries the server for supported log-in methods.
The server returns "basic auth" and "windows auth"
The 1.7 client first tries windows auth, fails, and then tries basic auth.
The 1.8 client tries windows auth, and if it fails it just tries again, 
and again, and again.. maybe its expecting that windows auth should work 
anyway: The user is already logged in, so the credentials must be valid. 
But they are not if the client is not part of the domain.

Looking at the stack in process explorer reveals that svn.exe seems to 
be looping around somewhere in libapr_tsvn.dll

Thanks for any help, if you need more information I'll be glad to share.

Elias

Re: svn client 1.8.1: Tries to log-in forever, hammering the server, eating memory and cpu

[Elias Gerber <eg...@viscomvisual.com>]

On 08.08.2013 13:04, Nico Kadel-Garcia wrote:
> On Thu, Aug 8, 2013 at 5:32 AM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>> On Thu, Aug 8, 2013 at 12:36 PM, Stefan Sperling <st...@elego.de> wrote:
>>> On Thu, Aug 08, 2013 at 10:18:42AM +0200, Elias Gerber wrote:
>>>> Since svn 1.8.x we experience the following:
>>>> Clients that are part of the domain can log in without any problems.
>>>> Clients that are not part of the domain try to log-in forever using
>>>> the local account. The server rejects those credentials - it only
>>>> knows about domain-users, not about local users. The client does not
>>>> stop to log in then, but tries forever: At the server we see
>>>> thousands of rejected log-in requests, at the client we see that it
>>>> uses one core of the cpu 100% and eats more and more memory and
>>>> never (? - I killed it when it reached more than one Gigabyte of
>>>> memory) stops.
>>> My guess would be this issue in serf:
>>> http://code.google.com/p/serf/issues/detail?id=77
>>>
>>> The 1.7 client uses neon instead of serf and optionally supports serf,
>>> but not many people used serf with 1.7 clients.
>>> Support for neon has been removed as of 1.8. The serf developers are
>>> working hard on issues that pop up now that people are actually using it.
>>>
>> This is definitely serf issue 77. Issue should be fixed in r2112. And
>> most likely will be backported to next serf 1.3.x release.
> Can you and your colleagues switch to svn+ssh protocol? They don't get
> the trivial single-sign-on for HTTPS access, but it would prevent this
> issue in the meantime.

Thanks for the hint, but to be honest: I just did a quick google search 
about 'Tortoise svn+ssh' and it seems we would need quite some steps to 
set this up. We just stick to 1.7 with those clients that are not part 
of the domain - those computers dont have internet access anyway, so 
they wont get noticed about an update for TortoiseSVN and dont install 
1.8.x "by accident". Another solution would be to uncheck in 
VisualSVNServer 'Integrated Windows Authentication' and only check 
'Basic authentication' - like this svn.exe 1.8.1 is working fine: It 
prompts for a user/pass as expected.
Also the machines that are used most are all part of the domain, so we 
dont have problems there.

Elias

PS: No need to CC me, I'm on the list.

Re: svn client 1.8.1: Tries to log-in forever, hammering the server, eating memory and cpu

[Nico Kadel-Garcia <nk...@gmail.com>]

On Thu, Aug 8, 2013 at 5:32 AM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On Thu, Aug 8, 2013 at 12:36 PM, Stefan Sperling <st...@elego.de> wrote:
>> On Thu, Aug 08, 2013 at 10:18:42AM +0200, Elias Gerber wrote:
>>> Since svn 1.8.x we experience the following:
>>> Clients that are part of the domain can log in without any problems.
>>> Clients that are not part of the domain try to log-in forever using
>>> the local account. The server rejects those credentials - it only
>>> knows about domain-users, not about local users. The client does not
>>> stop to log in then, but tries forever: At the server we see
>>> thousands of rejected log-in requests, at the client we see that it
>>> uses one core of the cpu 100% and eats more and more memory and
>>> never (? - I killed it when it reached more than one Gigabyte of
>>> memory) stops.
>>
>> My guess would be this issue in serf:
>> http://code.google.com/p/serf/issues/detail?id=77
>>
>> The 1.7 client uses neon instead of serf and optionally supports serf,
>> but not many people used serf with 1.7 clients.
>> Support for neon has been removed as of 1.8. The serf developers are
>> working hard on issues that pop up now that people are actually using it.
>>
> This is definitely serf issue 77. Issue should be fixed in r2112. And
> most likely will be backported to next serf 1.3.x release.

Can you and your colleagues switch to svn+ssh protocol? They don't get
the trivial single-sign-on for HTTPS access, but it would prevent this
issue in the meantime.

Re: svn client 1.8.1: Tries to log-in forever, hammering the server, eating memory and cpu

[Ivan Zhakov <iv...@visualsvn.com>]

On Thu, Aug 8, 2013 at 12:36 PM, Stefan Sperling <st...@elego.de> wrote:
> On Thu, Aug 08, 2013 at 10:18:42AM +0200, Elias Gerber wrote:
>> Since svn 1.8.x we experience the following:
>> Clients that are part of the domain can log in without any problems.
>> Clients that are not part of the domain try to log-in forever using
>> the local account. The server rejects those credentials - it only
>> knows about domain-users, not about local users. The client does not
>> stop to log in then, but tries forever: At the server we see
>> thousands of rejected log-in requests, at the client we see that it
>> uses one core of the cpu 100% and eats more and more memory and
>> never (? - I killed it when it reached more than one Gigabyte of
>> memory) stops.
>
> My guess would be this issue in serf:
> http://code.google.com/p/serf/issues/detail?id=77
>
> The 1.7 client uses neon instead of serf and optionally supports serf,
> but not many people used serf with 1.7 clients.
> Support for neon has been removed as of 1.8. The serf developers are
> working hard on issues that pop up now that people are actually using it.
>
This is definitely serf issue 77. Issue should be fixed in r2112. And
most likely will be backported to next serf 1.3.x release.

-- 
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com

Re: svn client 1.8.1: Tries to log-in forever, hammering the server, eating memory and cpu

[Elias Gerber <eg...@viscomvisual.com>]

On 08.08.2013 10:36, Stefan Sperling wrote:
> On Thu, Aug 08, 2013 at 10:18:42AM +0200, Elias Gerber wrote:
>> Since svn 1.8.x we experience the following:
>> Clients that are part of the domain can log in without any problems.
>> Clients that are not part of the domain try to log-in forever using
>> the local account. The server rejects those credentials - it only
>> knows about domain-users, not about local users. The client does not
>> stop to log in then, but tries forever: At the server we see
>> thousands of rejected log-in requests, at the client we see that it
>> uses one core of the cpu 100% and eats more and more memory and
>> never (? - I killed it when it reached more than one Gigabyte of
>> memory) stops.
> My guess would be this issue in serf:
> http://code.google.com/p/serf/issues/detail?id=77
>
> The 1.7 client uses neon instead of serf and optionally supports serf,
> but not many people used serf with 1.7 clients.
> Support for neon has been removed as of 1.8. The serf developers are
> working hard on issues that pop up now that people are actually using it.
>
> Please use 1.7 clients for now if this is a show stopper for you,
> and upgrade to a fixed version of serf once it is released.
> You don't need to upgrade Subversion itself for this fix.

Thank you. That matches exactly our problem.
We will watch out for updates (we actually use TortoiseSVN, I just used 
svn.exe to test things to ensure its not a TortoiseSVN problem).

Elias

Re: svn client 1.8.1: Tries to log-in forever, hammering the server, eating memory and cpu

[Stefan Sperling <st...@elego.de>]

On Thu, Aug 08, 2013 at 10:18:42AM +0200, Elias Gerber wrote:
> Since svn 1.8.x we experience the following:
> Clients that are part of the domain can log in without any problems.
> Clients that are not part of the domain try to log-in forever using
> the local account. The server rejects those credentials - it only
> knows about domain-users, not about local users. The client does not
> stop to log in then, but tries forever: At the server we see
> thousands of rejected log-in requests, at the client we see that it
> uses one core of the cpu 100% and eats more and more memory and
> never (? - I killed it when it reached more than one Gigabyte of
> memory) stops.

My guess would be this issue in serf:
http://code.google.com/p/serf/issues/detail?id=77

The 1.7 client uses neon instead of serf and optionally supports serf,
but not many people used serf with 1.7 clients.
Support for neon has been removed as of 1.8. The serf developers are
working hard on issues that pop up now that people are actually using it.

Please use 1.7 clients for now if this is a show stopper for you,
and upgrade to a fixed version of serf once it is released.
You don't need to upgrade Subversion itself for this fix.