You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Irfan Hamid <ih...@salesforce.com> on 2016/02/19 00:28:22 UTC
Kerberos enabled client connection failure GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
Hi,
I have a single ZooKeeper server test setup with Kerberos where it seems
the ZK server is able to obtain the TGT from Kerberos but when my client
tries to connect it gets the exception shown below. However, *it is able to
connect and create znodes despite the authentication failure.* I have a
Kerberos service principal of the form zookeeper/fqdn.to.dev.box@REALM.COM
and a ticket that I have setup on the ZK server with the server jaas.conf
looking like this prototype:
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/path/to/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper/fqdn.to.dev.box@REALM.COM";
};
On the client side I have a principal of the form zkcli@REALM.COM and an
associated ticket to which is pointing my jaas.conf like this:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/path/to/zkcli.keytab"
storeKey=true
useTicketCache=false
principal="zkcli@REALM.COM";
};
I start the client
with -Djava.security.auth.login.config=${solr.home}/build/jaas.conf. But
when I start the client app, zookeeper.out spews the following exception:
2016-02-18 15:18:24,906 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /10.22.34.129:40343
Found ticket for zookeeper/
ihamid-wsl1.internal.salesforce.com@ENG.SALESFORCE.COM to go to krbtgt/
ENG.SALESFORCE.COM@ENG.SALESFORCE.COM expiring on Fri Feb 19 01:18:04 PST
2016
2016-02-18 15:18:24,916 [myid:] - ERROR [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperSaslServer$1@122] - Zookeeper Server failed
to create a SaslServer to interact with a client during session initiation:
javax.security.sasl.SaslException: Failure to initialize security context
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos credentails)]
javax.security.sasl.SaslException: Failure to initialize security context
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos credentails)]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:125)
at
com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
at javax.security.sasl.Sasl.createSaslServer(Sasl.java:524)
at
org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
at
org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
at
org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
at org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos credentails)
at
sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:87)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
at
sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62)
at
sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
at
com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:108)
... 12 more
2016-02-18 15:18:24,920 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish
new session at /10.22.34.129:40343
2016-02-18 15:18:24,925 [myid:] - INFO [SyncThread:0:FileTxnLog@199] -
Creating new log file: log.1c
2016-02-18 15:18:24,930 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617]
- Established session 0x152f6ad15830000 with negotiated timeout 4000 for
client /10.22.34.129:40343
2016-02-18 15:18:24,935 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
processing sessionid:0x152f6ad15830000 type:create cxid:0x1 zxid:0x1d
txntype:-1 reqpath:n/a Error Path:/searchserver Error:KeeperErrorCode =
NodeExists for /searchserver
2016-02-18 15:18:24,944 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
processing sessionid:0x152f6ad15830000 type:create cxid:0x2 zxid:0x1e
txntype:-1 reqpath:n/a Error Path:/searchserver/devpod
Error:KeeperErrorCode = NodeExists for /searchserver/devpod
2016-02-18 15:18:24,945 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
processing sessionid:0x152f6ad15830000 type:create cxid:0x3 zxid:0x1f
txntype:-1 reqpath:n/a Error Path:/searchserver/devpod/statesv1
Error:KeeperErrorCode = NodeExists for /searchserver/devpod/statesv1
TIA,
Irfan.
Re: Kerberos enabled client connection failure GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
Posted by Irfan Hamid <ih...@salesforce.com>.
<facepalm>
The jvm args were being set in a different target from what I was running.
Once I fixed that, I am getting the following error on the client side:
X`20160218161203.843``43`0`0``````WARNING`syncStarter-184352057-SendThread(localhost:2181)`Session
0x0 for server null, unexpected error, closing socket connection and
attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.<init>(Login.java:44)
at
org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:198)
at
org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:104)
at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:943)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
In the sources this is at:
public class Login {
* Logger LOG = Logger.getLogger(Login.class);*
public CallbackHandler callbackHandler;
I'm trying to add log4j jar to my classpath and see if that fixes this
issue.
</facepalm>
Thanks,
Irfan.
On Thu, Feb 18, 2016 at 3:28 PM, Irfan Hamid <ih...@salesforce.com> wrote:
> Hi,
>
> I have a single ZooKeeper server test setup with Kerberos where it seems
> the ZK server is able to obtain the TGT from Kerberos but when my client
> tries to connect it gets the exception shown below. However, *it is able
> to connect and create znodes despite the authentication failure.* I have
> a Kerberos service principal of the form zookeeper/
> fqdn.to.dev.box@REALM.COM and a ticket that I have setup on the ZK server
> with the server jaas.conf looking like this prototype:
>
> Server {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> keyTab="/path/to/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> principal="zookeeper/fqdn.to.dev.box@REALM.COM";
> };
>
>
>
> On the client side I have a principal of the form zkcli@REALM.COM and an
> associated ticket to which is pointing my jaas.conf like this:
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> keyTab="/path/to/zkcli.keytab"
> storeKey=true
> useTicketCache=false
> principal="zkcli@REALM.COM";
> };
>
> I start the client
> with -Djava.security.auth.login.config=${solr.home}/build/jaas.conf. But
> when I start the client app, zookeeper.out spews the following exception:
>
> 2016-02-18 15:18:24,906 [myid:] - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /10.22.34.129:40343
> Found ticket for zookeeper/
> ihamid-wsl1.internal.salesforce.com@ENG.SALESFORCE.COM to go to krbtgt/
> ENG.SALESFORCE.COM@ENG.SALESFORCE.COM expiring on Fri Feb 19 01:18:04 PST
> 2016
> 2016-02-18 15:18:24,916 [myid:] - ERROR [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperSaslServer$1@122] - Zookeeper Server failed
> to create a SaslServer to interact with a client during session initiation:
> javax.security.sasl.SaslException: Failure to initialize security context
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos credentails)]
> javax.security.sasl.SaslException: Failure to initialize security context
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos credentails)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:125)
> at
> com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at javax.security.sasl.Sasl.createSaslServer(Sasl.java:524)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
> at org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos credentails)
> at
> sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:87)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
> at
> sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
> at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
> at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62)
> at
> sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:108)
> ... 12 more
> 2016-02-18 15:18:24,920 [myid:] - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /10.22.34.129:40343
> 2016-02-18 15:18:24,925 [myid:] - INFO [SyncThread:0:FileTxnLog@199] -
> Creating new log file: log.1c
> 2016-02-18 15:18:24,930 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617]
> - Established session 0x152f6ad15830000 with negotiated timeout 4000 for
> client /10.22.34.129:40343
> 2016-02-18 15:18:24,935 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> when processing sessionid:0x152f6ad15830000 type:create cxid:0x1 zxid:0x1d
> txntype:-1 reqpath:n/a Error Path:/searchserver Error:KeeperErrorCode =
> NodeExists for /searchserver
> 2016-02-18 15:18:24,944 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> when processing sessionid:0x152f6ad15830000 type:create cxid:0x2 zxid:0x1e
> txntype:-1 reqpath:n/a Error Path:/searchserver/devpod
> Error:KeeperErrorCode = NodeExists for /searchserver/devpod
> 2016-02-18 15:18:24,945 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> when processing sessionid:0x152f6ad15830000 type:create cxid:0x3 zxid:0x1f
> txntype:-1 reqpath:n/a Error Path:/searchserver/devpod/statesv1
> Error:KeeperErrorCode = NodeExists for /searchserver/devpod/statesv1
>
>
> TIA,
> Irfan.
>