You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Ahshan (Jira)" <ji...@apache.org> on 2019/10/14 06:11:00 UTC
[jira] [Updated] (ZOOKEEPER-3576) Zookeeper Fails with AUTH_FAILED
state with SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahshan updated ZOOKEEPER-3576:
------------------------------
Description:
Although i'm able to authenticate successfully with the kerberoes account *"zookeeper/kafka-d1.eng.company.com@COMPANY.COM" , i still happen to encounter* AUTH_FAILED during client Authentication
Following is the verification made from my end :
# Checked DNS ( Both Forward and Backward)
nslookup kafka-d1.eng.company.com
Server: 172.16.2.3
Address: 172.16.2.3#53
Name: kafka-d1.eng.company.com
Address: 10.14.61.17
Reverse DNS
nslookup 10.14.61.17
Server: 172.16.2.3
Address: 172.16.2.3#53
17.61.14.10.in-addr.arpa name = kafka-d1.eng.company.com.
2. Kerberoes Authentication
kinit -kt /etc/keytabs/zookeeper.keytab -V zookeeper/kafka-d1.eng.company.com
Using default cache: /tmp/krb5cc_0
Using principal: zookeeper/kafka-d1.eng.company.com@COMPANY.COM
Using keytab: /etc/keytabs/zookeeper.keytab
Authenticated to Kerberos v5
Below is the krb5 configuration File:
cat /etc/krb5.conf
[libdefaults]
default_realm = COMPANY.COM
dns_lookup_kdc = true
dns_lookup_realm = true
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts
default_tkt_enctypes = aes256-cts
permitted_enctypes = aes256-cts
udp_preference_limit = 1
kdc_timeout = 3000
ignore_acceptor_hostname = true
[realms]
COMPANY.COM =
{ kdc = srv-ussc-dc01e.company.com admin_server = srv-exxx.company.com kdc = srv-exxxe.company.com }
[domain_realm]
kafka-d1.eng.company.com = COMPANY.COM
*Error Message :[^zoo.cfg][^zookeeper_server.log]*
{noformat}
./zkCli.sh -server kafka-d1.eng.company.com:2181
Connecting to kafka-d1.eng.company.com:2181
2019-10-14 02:08:16,625 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-10-14 02:08:16,628 [myid:] - INFO [main:Environment@100] - Client environment:host.name=kafka-d1.eng.company.com
2019-10-14 02:08:16,628 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_201/jre
2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/share/zookeeper/bin/../build/classes:/usr/share/zookeeper/bin/../build/lib/*.jar:/usr/share/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/share/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/share/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/share/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/share/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/share/zookeeper/bin/../zookeeper-3.4.10.jar:/usr/share/zookeeper/bin/../src/java/lib/*.jar:/usr/share/zookeeper/bin/../conf:
2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-327.el7.x86_64
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/usr/share/zookeeper-3.4.10/bin
2019-10-14 02:08:16,632 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-d1.eng.company.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
Welcome to ZooKeeper!
JLine support is enabled
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/kafka-d1.eng.company.com@COMPANY.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false
[zk: kafka-d1.eng.company.com:2181(CONNECTING) 0] principal is zookeeper/kafka-d1.eng.company.com@COMPANY.COM
Will use keytab
Commit Succeeded 2019-10-14 02:08:16,971 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):Login@295] - Client successfully logged in.
2019-10-14 02:08:16,973 [myid:] - INFO [Thread-1:Login$1@128] - TGT refresh thread started.
2019-10-14 02:08:16,975 [myid:] - INFO [Thread-1:Login@303] - TGT valid starting at: Mon Oct 14 02:08:16 EDT 2019
2019-10-14 02:08:16,976 [myid:] - INFO [Thread-1:Login@304] - TGT expires: Mon Oct 14 12:08:16 EDT 2019
2019-10-14 02:08:16,976 [myid:] - INFO [Thread-1:Login$1@183] - TGT refresh sleeping until: Mon Oct 14 10:08:57 EDT 2019
2019-10-14 02:08:16,977 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):SecurityUtils$1@124] - Client will use GSSAPI as SASL mechanism.
2019-10-14 02:08:16,988 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server kafka-d1.eng.company.com/10.14.61.17:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-10-14 02:08:16,994 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@876] - Socket connection established to kafka-d1.eng.company.com/10.14.61.17:2181, initiating session
2019-10-14 02:08:17,002 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server kafka-d1.eng.company.com/10.14.61.17:2181, sessionid = 0x16dc8cbdb3b0002, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null
2019-10-14 02:08:17,024 [myid:] - ERROR [main-SendThread(kafka-d1.eng.company.com:2181):ZooKeeperSaslClient@247] - SASL authentication failed using login context 'Client'.WATCHER::WatchedEvent state:AuthFailed type:None path:null{noformat}
was:
Although i'm able to authenticate successfully with the kerberoes account *"zookeeper/kafka-d1.eng.company.com@COMPANY.COM" , i still happen to encounter* AUTH_FAILED during client Authentication
Following is the verification made from my end :
# Checked DNS ( Both Forward and Backward)
nslookup kafka-d1.eng.company.com
Server: 172.16.2.3
Address: 172.16.2.3#53
Name: kafka-d1.eng.company.com
Address: 10.14.61.17
Reverse DNS
nslookup 10.14.61.17
Server: 172.16.2.3
Address: 172.16.2.3#53
17.61.14.10.in-addr.arpa name = kafka-d1.eng.company.com.
2. Kerberoes Authentication
kinit -kt /etc/keytabs/zookeeper.keytab -V zookeeper/kafka-d1.eng.company.com
Using default cache: /tmp/krb5cc_0
Using principal: zookeeper/kafka-d1.eng.company.com@COMPANY.COM
Using keytab: /etc/keytabs/zookeeper.keytab
Authenticated to Kerberos v5
Below is the krb5 configuration File:
cat /etc/krb5.conf
[libdefaults]
default_realm = COMPANY.COM
dns_lookup_kdc = true
dns_lookup_realm = true
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts
default_tkt_enctypes = aes256-cts
permitted_enctypes = aes256-cts
udp_preference_limit = 1
kdc_timeout = 3000
ignore_acceptor_hostname = true
[realms]
COMPANY.COM = {
kdc = srv-ussc-dc01e.company.com
admin_server = srv-exxx.company.com
kdc = srv-exxxe.company.com
}
[domain_realm]
kafka-d1.eng.company.com = COMPANY.COM
*Error Message :[^zoo.cfg][^zookeeper_server.log]*
{noformat}
WatchedEvent state:SyncConnected type:None path:null
2019-10-14 01:46:47,858 [myid:] - ERROR [main-SendThread(localhost:2181):ZooKeeperSaslClient@308] - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-10-14 01:46:47,859 [myid:] - ERROR [main-SendThread(localhost:2181):ClientCnxn$SendThread@1072] - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.{noformat}
> Zookeeper Fails with AUTH_FAILED state with SASL
> -------------------------------------------------
>
> Key: ZOOKEEPER-3576
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3576
> Project: ZooKeeper
> Issue Type: Bug
> Components: kerberos, security
> Affects Versions: 3.4.10
> Reporter: Ahshan
> Priority: Major
> Attachments: zoo.cfg, zookeeper_server.log
>
>
> Although i'm able to authenticate successfully with the kerberoes account *"zookeeper/kafka-d1.eng.company.com@COMPANY.COM" , i still happen to encounter* AUTH_FAILED during client Authentication
> Following is the verification made from my end :
> # Checked DNS ( Both Forward and Backward)
> nslookup kafka-d1.eng.company.com
> Server: 172.16.2.3
> Address: 172.16.2.3#53
> Name: kafka-d1.eng.company.com
> Address: 10.14.61.17
> Reverse DNS
> nslookup 10.14.61.17
> Server: 172.16.2.3
> Address: 172.16.2.3#53
> 17.61.14.10.in-addr.arpa name = kafka-d1.eng.company.com.
>
> 2. Kerberoes Authentication
> kinit -kt /etc/keytabs/zookeeper.keytab -V zookeeper/kafka-d1.eng.company.com
> Using default cache: /tmp/krb5cc_0
> Using principal: zookeeper/kafka-d1.eng.company.com@COMPANY.COM
> Using keytab: /etc/keytabs/zookeeper.keytab
> Authenticated to Kerberos v5
>
> Below is the krb5 configuration File:
> cat /etc/krb5.conf
> [libdefaults]
> default_realm = COMPANY.COM
> dns_lookup_kdc = true
> dns_lookup_realm = true
> ticket_lifetime = 86400
> renew_lifetime = 604800
> forwardable = true
> default_tgs_enctypes = aes256-cts
> default_tkt_enctypes = aes256-cts
> permitted_enctypes = aes256-cts
> udp_preference_limit = 1
> kdc_timeout = 3000
> ignore_acceptor_hostname = true
> [realms]
> COMPANY.COM =
> { kdc = srv-ussc-dc01e.company.com admin_server = srv-exxx.company.com kdc = srv-exxxe.company.com }
> [domain_realm]
> kafka-d1.eng.company.com = COMPANY.COM
>
> *Error Message :[^zoo.cfg][^zookeeper_server.log]*
> {noformat}
> ./zkCli.sh -server kafka-d1.eng.company.com:2181
> Connecting to kafka-d1.eng.company.com:2181
> 2019-10-14 02:08:16,625 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
> 2019-10-14 02:08:16,628 [myid:] - INFO [main:Environment@100] - Client environment:host.name=kafka-d1.eng.company.com
> 2019-10-14 02:08:16,628 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_201
> 2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
> 2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_201/jre
> 2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/share/zookeeper/bin/../build/classes:/usr/share/zookeeper/bin/../build/lib/*.jar:/usr/share/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/share/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/share/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/share/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/share/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/share/zookeeper/bin/../zookeeper-3.4.10.jar:/usr/share/zookeeper/bin/../src/java/lib/*.jar:/usr/share/zookeeper/bin/../conf:
> 2019-10-14 02:08:16,630 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-327.el7.x86_64
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
> 2019-10-14 02:08:16,631 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/usr/share/zookeeper-3.4.10/bin
> 2019-10-14 02:08:16,632 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-d1.eng.company.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
> Welcome to ZooKeeper!
> JLine support is enabled
> Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/kafka-d1.eng.company.com@COMPANY.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false
> [zk: kafka-d1.eng.company.com:2181(CONNECTING) 0] principal is zookeeper/kafka-d1.eng.company.com@COMPANY.COM
> Will use keytab
> Commit Succeeded 2019-10-14 02:08:16,971 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):Login@295] - Client successfully logged in.
> 2019-10-14 02:08:16,973 [myid:] - INFO [Thread-1:Login$1@128] - TGT refresh thread started.
> 2019-10-14 02:08:16,975 [myid:] - INFO [Thread-1:Login@303] - TGT valid starting at: Mon Oct 14 02:08:16 EDT 2019
> 2019-10-14 02:08:16,976 [myid:] - INFO [Thread-1:Login@304] - TGT expires: Mon Oct 14 12:08:16 EDT 2019
> 2019-10-14 02:08:16,976 [myid:] - INFO [Thread-1:Login$1@183] - TGT refresh sleeping until: Mon Oct 14 10:08:57 EDT 2019
> 2019-10-14 02:08:16,977 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):SecurityUtils$1@124] - Client will use GSSAPI as SASL mechanism.
> 2019-10-14 02:08:16,988 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server kafka-d1.eng.company.com/10.14.61.17:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
> 2019-10-14 02:08:16,994 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@876] - Socket connection established to kafka-d1.eng.company.com/10.14.61.17:2181, initiating session
> 2019-10-14 02:08:17,002 [myid:] - INFO [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server kafka-d1.eng.company.com/10.14.61.17:2181, sessionid = 0x16dc8cbdb3b0002, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null
> 2019-10-14 02:08:17,024 [myid:] - ERROR [main-SendThread(kafka-d1.eng.company.com:2181):ZooKeeperSaslClient@247] - SASL authentication failed using login context 'Client'.WATCHER::WatchedEvent state:AuthFailed type:None path:null{noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)