You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Nicolai Scheer <ni...@gmail.com> on 2014/07/08 15:57:44 UTC
SVN 1.8 Kerberos Client problems
Hi,
I did an update to svn 1.8.9 on our server yesterday and am now
running into problems whilst updating the corresponding windows
client.
Server details:
CentOS 6.5 x64
SVN 1.8.9, compiled on my own
Apache 2.2.15
Authentication using mod_auth_kerb to windows domain
Client:
Windows Server 2008
TSVN 1.7.13
Everything ran fine so far. I then upgraded to TSVN 1.8.7 which ships
with svn 1.8.9 binaries.
Now, on every repository involving action I get kerberos errors in my
apache log:
[Tue Jul 08 15:08:10 2014] [error] [client 192.168.1.152]
gss_accept_sec_context() failed: No credentials were supplied, or the
credentials were unavailable or inaccessible (, Unknown error)
Strange thing is, TSVN works fine from a user perspective (no error
displayed whatsoever...). I then tried to use the svn command line
binaries, same result, evrythings works, but apache error log is DOSed
with these error entries.
I tried to either "prefer" or deny BulkUpdates on the server, nothing changes.
Since subversion 1.8 comes with serf, it might have do to with the way
serf handles kerberos.
>From the linux cmd (on the svn server itself), I can use kerberos
(kinit + svn command afterwards) just fine, without any errors popping
up in the apache log.
Our browsers (IE, Chrome, FF) worked fine all the time using kerberos,
before and after upgrading svn on the server.
TSVN 1.7.13 works as well, so do the svn binaries shipped this version.
I'm quite stuck, because I do not know where to start. To my mind
kerberos is set up correctly because all clients work, except svn 1.8
I thought I could solve the problem by falling back to BulkUpdate, but
it does not seem to help.
Any suggestions are appreciated!
Greetings
Nico
PS:
svn apache config :
SVNInMemoryCacheSize 131072
SVNCacheTextDeltas On
SVNCacheFullTexts On
SVNCompressionLevel 6
SVNCacheRevProps On
<Location /repos>
SSLRequireSSL
DAV svn
SVNListParentPath On
SVNParentPath /var/local/svn/repos
# Kerberos authentification
AuthType Kerberos
AuthName "Realm"
KrbServiceName HTTP
KrbLocalUserMapping on
KrbAuthRealms REALM.INTERN
KrbMethodNegotiate on
KrbMethodK5Passwd on
Krb5KeyTab /etc/httpd/kerberos/keytab
KrbVerifyKDC on
require valid-user
AuthzSVNAccessFile /var/local/svn/auth/svn_authz
SVNPathAuthz short_circuit
</Location>
Re: SVN 1.8 Kerberos Client problems
Posted by Nicolai Scheer <ni...@gmail.com>.
Hi!
Firstly, thanks for your quick answer!
On 8 July 2014 21:24, Lieven Govaerts <lg...@mobsol.be> wrote:
> Hi Nico,
>
[...]
>>
>> Everything ran fine so far. I then upgraded to TSVN 1.8.7 which ships
>> with svn 1.8.9 binaries.
>> Now, on every repository involving action I get kerberos errors in my
>> apache log:
>>
>> [Tue Jul 08 15:08:10 2014] [error] [client 192.168.1.152]
>> gss_accept_sec_context() failed: No credentials were supplied, or the
>> credentials were unavailable or inaccessible (, Unknown error)
>>
>> Strange thing is, TSVN works fine from a user perspective (no error
>> displayed whatsoever...). I then tried to use the svn command line
>> binaries, same result, evrythings works, but apache error log is DOSed
>> with these error entries.
>
> That's because you have this directive in your apache config:
> KrbMethodK5Passwd on
>
> With this configuration, mod_auth_kerb will offer both Negotiate and
> Basic authentication. Your logs show that Negotiate authentication
> fails, but in this case svn will automatically fall back to Basic
> authentication, which succeeds.
>
>> I tried to either "prefer" or deny BulkUpdates on the server, nothing changes.
>
> This option has nothing to do with authentication, it specifies how -
> after authentication succeeded - svn+serf will communicate with
> mod_dav_svn.
Apparently, yes :)
I tried this option, because to me it seemed as if the svn client
reauthenticates all the time during a svn operation (e.g. update).
So I thought this might help - i.e. many requests vs. one big. I
excpected to happen the error only once when using BulkUpdates,
because it should only authenticate once. Unfortunately it seems I'm
wrong :|
>> From the linux cmd (on the svn server itself), I can use kerberos
>> (kinit + svn command afterwards) just fine, without any errors popping
>> up in the apache log.
>>
>
> Which GSSAPI implementation are you using on the server? MIT? Heimdal?
> Which versions?
[root@svn log]# rpm -qa | grep krb
krb5-libs-1.10.3-15.el6_5.1.x86_64
python-krbV-1.0.90-3.el6.x86_64
krb5-devel-1.10.3-15.el6_5.1.x86_64
krb5-workstation-1.10.3-15.el6_5.1.x86_64
pam_krb5-2.3.11-9.el6.x86_64
Info about krb5-workstation:
Installed Packages
Name : krb5-workstation
Arch : x86_64
Version : 1.10.3
Release : 15.el6_5.1
Size : 1.0 M
Repo : installed
>From repo : updates
Summary : Kerberos 5 programs for use on workstations
URL : http://web.mit.edu/kerberos/www/
License : MIT
Description : Kerberos is a network authentication system. The krb5-workstation
: package contains the basic Kerberos programs (kinit,
klist, kdestroy,
: kpasswd). If your network uses Kerberos, this package should be
: installed on every workstation.
> Debugging this stuff will not be easy. Most likely you'll find some
> more information in the Kerberos implementation logging, which is
> configured in your krb5.conf file. On my system the log file is
> /var/log/krb5.log. Try to increase the log level to get more detailed
> error information.
>
> The best alternative is to find out what's going over the wire.
> Disable your SSL configuration and set KrbMethodK5Passwd to off (on a
> test server) and use Wireshark to trace all traffic between your
> Windows client and svn server. Wireshark on Windows can decode the
> SPNego tokens in the Authorization headers (at least for NTLM but I
> suppose for Kerberos also). This allows you to see if all info is
> correct (domain, username etc). If you want you can send me the
> wireshark trace privately and I'll have a look.
I tried to do some testing, and got weird results.
First of all, I don't get the kerberos error mentioned above anymore.
Instead TSVN and the command line client just hang on every operation.
Well, it seems they hang, but they finish eventually 30 minutes later,
even on the simplest actions. This was on a server that is not part of
our domain.
(maybe the server not being part of our domain was the problem
yesterday, because it would try negotiate without success then)
Next, I switched to a Windows 2008 sevrer which is part of our domain.
I cleaned any user properties (e.g. subversion folder and saved auth
stuff) before every run.
I tested with svn command line clients version 1.7.10 and 1.8.9
(shipped with the corresponding TSVN version)
Test command:
Simple checkout of a project
svn client version 1.7.10
1. KrbMethodNegotiate on: asks for password, works
2. KrbMethodNegotiate off: asks for password, works
svn client version 1.8.9
3. KrbMethodNegotiate on: does not ask for password, does not always work
4. KrbMethodNegotiate off: asks for password, works
Test case 1 and 2 seem reasonable, IIRC neon just can't do negotiate
but the password fallback works.
Test case 4 seems to work well, i.e. the password fallback is enforced
and everything works, even with serf in place.
Test case 3 produces very strange results.
First I get random svn: E200014: Checksum mismatch errors and the
checkout just stops. Sometimes I get an segmentation fault and
checkout stops as well.
Sometimes checkout runs fine.
And I found a path, that when checked out, makes the svn client hang
indefinitely...
At the same time I get the following error in my apache error log:
[Wed Jul 09 14:18:28 2014] [error] [client 172.16.37.37] Provider
encountered an error while streaming a REPORT response. [500, #0]
[Wed Jul 09 14:18:28 2014] [error] [client 172.16.37.37] A failure
occurred while driving the update report editor [500, #103]
This happens on the file 'ext-all-debug-w-comments.js' (from the ExtJS
framework, > 2 MB).
This hang situation can be reproduced with the linux client as well.
This hang only occurs with negotiate on + ssl... If I switch either
of, it works...
I never encountered the kerberos error from yesterday during my tests again...
Checksum mismatches and segmentation faults are too hard to chew for
our develpoment machine. Unfortunately I don't have enough time to dig
into these issues very deeply and I wouldn't know where to start
either. Since our current version of TSVN does not support kerberos
negotiate, I think I'll switch it off.
We're using trac for ticket managment as well, and as said before,
negotiation works fine there (i.e. webbrowser as clients).
I'll happily provide any additional information, since the errors
mentioned above seem quite serious. If the kerberos implementation on
the server is flawed, that's one thing. But the svn client should not
produce segfaults and checksum mismatches in such a condition.
Well, maybe it's not the client itself, but the serf library...
Greetings
Nico
Re: SVN 1.8 Kerberos Client problems
Posted by Lieven Govaerts <lg...@mobsol.be>.
Hi Nico,
On Tue, Jul 8, 2014 at 3:57 PM, Nicolai Scheer <ni...@gmail.com> wrote:
> Hi,
>
> I did an update to svn 1.8.9 on our server yesterday and am now
> running into problems whilst updating the corresponding windows
> client.
>
> Server details:
>
> CentOS 6.5 x64
> SVN 1.8.9, compiled on my own
> Apache 2.2.15
> Authentication using mod_auth_kerb to windows domain
>
> Client:
> Windows Server 2008
> TSVN 1.7.13
>
> Everything ran fine so far. I then upgraded to TSVN 1.8.7 which ships
> with svn 1.8.9 binaries.
> Now, on every repository involving action I get kerberos errors in my
> apache log:
>
> [Tue Jul 08 15:08:10 2014] [error] [client 192.168.1.152]
> gss_accept_sec_context() failed: No credentials were supplied, or the
> credentials were unavailable or inaccessible (, Unknown error)
>
> Strange thing is, TSVN works fine from a user perspective (no error
> displayed whatsoever...). I then tried to use the svn command line
> binaries, same result, evrythings works, but apache error log is DOSed
> with these error entries.
That's because you have this directive in your apache config:
KrbMethodK5Passwd on
With this configuration, mod_auth_kerb will offer both Negotiate and
Basic authentication. Your logs show that Negotiate authentication
fails, but in this case svn will automatically fall back to Basic
authentication, which succeeds.
> I tried to either "prefer" or deny BulkUpdates on the server, nothing changes.
This option has nothing to do with authentication, it specifies how -
after authentication succeeded - svn+serf will communicate with
mod_dav_svn.
> Since subversion 1.8 comes with serf, it might have do to with the way
> serf handles kerberos.
Not unlikely.
> From the linux cmd (on the svn server itself), I can use kerberos
> (kinit + svn command afterwards) just fine, without any errors popping
> up in the apache log.
>
Which GSSAPI implementation are you using on the server? MIT? Heimdal?
Which versions?
> Our browsers (IE, Chrome, FF) worked fine all the time using kerberos,
> before and after upgrading svn on the server.
>
> TSVN 1.7.13 works as well, so do the svn binaries shipped this version.
>
> I'm quite stuck, because I do not know where to start. To my mind
> kerberos is set up correctly because all clients work, except svn 1.8
>
Serf on Windows uses Microsoft's SSPI API to handle Negotiate
authentication. We know this works well with Windows based servers
(e.g. VisualSVN) that also use SSPI on the server side.
Serf on Linux/Mac uses GSSAPI and one of its implementations. This
works well with other *nix servers.
It's likely that the combination client on Windows with
server+mod_auth_kerb on Linux hasn't seen as much testing as the cross
platform combinations, and that certain scenario's are not supported
well.
Debugging this stuff will not be easy. Most likely you'll find some
more information in the Kerberos implementation logging, which is
configured in your krb5.conf file. On my system the log file is
/var/log/krb5.log. Try to increase the log level to get more detailed
error information.
The best alternative is to find out what's going over the wire.
Disable your SSL configuration and set KrbMethodK5Passwd to off (on a
test server) and use Wireshark to trace all traffic between your
Windows client and svn server. Wireshark on Windows can decode the
SPNego tokens in the Authorization headers (at least for NTLM but I
suppose for Kerberos also). This allows you to see if all info is
correct (domain, username etc). If you want you can send me the
wireshark trace privately and I'll have a look.
> I thought I could solve the problem by falling back to BulkUpdate, but
> it does not seem to help.
>
> Any suggestions are appreciated!
>
> Greetings
>
> Nico
>
hth,
Lieven
> PS:
>
> svn apache config :
>
> SVNInMemoryCacheSize 131072
> SVNCacheTextDeltas On
> SVNCacheFullTexts On
> SVNCompressionLevel 6
> SVNCacheRevProps On
> <Location /repos>
> SSLRequireSSL
> DAV svn
>
> SVNListParentPath On
> SVNParentPath /var/local/svn/repos
>
> # Kerberos authentification
> AuthType Kerberos
> AuthName "Realm"
> KrbServiceName HTTP
> KrbLocalUserMapping on
> KrbAuthRealms REALM.INTERN
> KrbMethodNegotiate on
> KrbMethodK5Passwd on
> Krb5KeyTab /etc/httpd/kerberos/keytab
> KrbVerifyKDC on
>
> require valid-user
>
> AuthzSVNAccessFile /var/local/svn/auth/svn_authz
> SVNPathAuthz short_circuit
> </Location>