You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "ZhenyuLi (Jira)" <ji...@apache.org> on 2026/03/25 19:20:00 UTC

[jira] [Updated] (ZOOKEEPER-5033) Quorum SASL authentication fails permanently after Login TGT refresh thread exits

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ZhenyuLi updated ZOOKEEPER-5033:
--------------------------------
    Description: 
When quorum SASL authentication is enabled ({{{}quorum.auth.enableSasl=true{}}}) with Kerberos, the {{Login}} class runs a background daemon thread to periodically 
refresh the TGT. This thread can silently exit in several scenarios:

1. Clock skew between the host and KDC 
2. TGT is not renewable
3. {{kinit -R}} fails after retry
4. {{reLogin()}} fails after retry 
5. {{nextRefresh}} is in the past

After the thread exits, the TGT cached in the {{Subject}} eventually expires. When a Follower/Observer later needs to reconnect to the Leader (e.g., after a network
partition or leader switch), {{SaslQuorumAuthLearner.authenticate()}} uses the stale credentials from {{learnerLogin.getSubject()}} and fails with 
{{{}SaslException{}}}.

The caller ({{{}QuorumPeer{}}} main loop) retries by going through {{{}LOOKING → FOLLOWING → connectToLeader → authenticate{}}}, but the {{authLearner}} object is created 
once in {{QuorumPeer.initialize()}} and never recreated. The same stale {{Login}} and {{Subject}} are reused, causing every retry to fail indefinitely.

*Trigger conditions (all must be met):*
 - {{quorum.auth.enableSasl=true}}
 - {{quorum.auth.learnerRequireSasl=true}} and {{quorum.auth.serverRequireSasl=true}}
 - Kerberos authentication (not DIGEST-MD5)
 - Login refresh thread exits due to one of the above scenarios
 - A reconnection event occurs after TGT expires

*Impact:* 
The affected server permanently loses the ability to join the quorum. If multiple servers are affected, quorum may be lost. The only 
recovery is to restart the process.

*Affected code paths:*
 - {{Learner.connectToLeader()}} → {{self.authLearner.authenticate()}} (Learner.java:354)
 - {{QuorumCnxManager.initiateConnection()}} → {{authLearner.authenticate()}} (QuorumCnxManager.java:506)
 - {{QuorumCnxManager.handleConnection()}} → {{authServer.authenticate()}} (QuorumCnxManager.java:633)
 - {{LearnerHandler}} constructor → {{authServer.authenticate()}} (LearnerHandler.java:291)

*Fix:* 
Add a {{forceReLogin()}} method to {{Login}} that re-logins immediately (bypassing the minimum time check), and call it from 
{{SaslQuorumAuthLearner.authenticate()}} and {{SaslQuorumAuthServer.authenticate()}} when authentication fails. This ensures the next authentication attempt uses 
fresh credentials.

  was:
When quorum SASL authentication is enabled ({{{}quorum.auth.enableSasl=true{}}}) with Kerberos, the {{Login}} class runs a background daemon thread to periodically 
refresh the TGT. This thread can silently exit in several scenarios:

1. Clock skew between the host and KDC ({{{}Login.java:185-193{}}}) 
2. TGT is not renewable ({{{}Login.java:148-160{}}})
3. {{kinit -R}} fails after retry ({{{}Login.java:236-242{}}}) 
4. {{reLogin()}} fails after retry ({{{}Login.java:268-270{}}}) 
5. {{nextRefresh}} is in the past ({{{}Login.java:207-214{}}})

After the thread exits, the TGT cached in the {{Subject}} eventually expires. When a Follower/Observer later needs to reconnect to the Leader (e.g., after a network
partition or leader switch), {{SaslQuorumAuthLearner.authenticate()}} uses the stale credentials from {{learnerLogin.getSubject()}} and fails with 
{{{}SaslException{}}}.

The caller ({{{}QuorumPeer{}}} main loop) retries by going through {{{}LOOKING → FOLLOWING → connectToLeader → authenticate{}}}, but the {{authLearner}} object is created 
once in {{QuorumPeer.initialize()}} and never recreated. The same stale {{Login}} and {{Subject}} are reused, causing every retry to fail indefinitely.

*Trigger conditions (all must be met):*
 - {{quorum.auth.enableSasl=true}}
 - {{quorum.auth.learnerRequireSasl=true}} and {{quorum.auth.serverRequireSasl=true}}
 - Kerberos authentication (not DIGEST-MD5)
 - Login refresh thread exits due to one of the above scenarios
 - A reconnection event occurs after TGT expires

*Impact:* 
The affected server permanently loses the ability to join the quorum. If multiple servers are affected, quorum may be lost. The only 
recovery is to restart the process.

*Affected code paths:*
 - {{Learner.connectToLeader()}} → {{self.authLearner.authenticate()}} (Learner.java:354)
 - {{QuorumCnxManager.initiateConnection()}} → {{authLearner.authenticate()}} (QuorumCnxManager.java:506)
 - {{QuorumCnxManager.handleConnection()}} → {{authServer.authenticate()}} (QuorumCnxManager.java:633)
 - {{LearnerHandler}} constructor → {{authServer.authenticate()}} (LearnerHandler.java:291)

*Fix:* 
Add a {{forceReLogin()}} method to {{Login}} that re-logins immediately (bypassing the minimum time check), and call it from 
{{SaslQuorumAuthLearner.authenticate()}} and {{SaslQuorumAuthServer.authenticate()}} when authentication fails. This ensures the next authentication attempt uses 
fresh credentials.


> Quorum SASL authentication fails permanently after Login TGT refresh thread exits 
> ----------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-5033
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5033
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.9.3
>            Reporter: ZhenyuLi
>            Priority: Major
>
> When quorum SASL authentication is enabled ({{{}quorum.auth.enableSasl=true{}}}) with Kerberos, the {{Login}} class runs a background daemon thread to periodically 
> refresh the TGT. This thread can silently exit in several scenarios:
> 1. Clock skew between the host and KDC 
> 2. TGT is not renewable
> 3. {{kinit -R}} fails after retry
> 4. {{reLogin()}} fails after retry 
> 5. {{nextRefresh}} is in the past
> After the thread exits, the TGT cached in the {{Subject}} eventually expires. When a Follower/Observer later needs to reconnect to the Leader (e.g., after a network
> partition or leader switch), {{SaslQuorumAuthLearner.authenticate()}} uses the stale credentials from {{learnerLogin.getSubject()}} and fails with 
> {{{}SaslException{}}}.
> The caller ({{{}QuorumPeer{}}} main loop) retries by going through {{{}LOOKING → FOLLOWING → connectToLeader → authenticate{}}}, but the {{authLearner}} object is created 
> once in {{QuorumPeer.initialize()}} and never recreated. The same stale {{Login}} and {{Subject}} are reused, causing every retry to fail indefinitely.
> *Trigger conditions (all must be met):*
>  - {{quorum.auth.enableSasl=true}}
>  - {{quorum.auth.learnerRequireSasl=true}} and {{quorum.auth.serverRequireSasl=true}}
>  - Kerberos authentication (not DIGEST-MD5)
>  - Login refresh thread exits due to one of the above scenarios
>  - A reconnection event occurs after TGT expires
> *Impact:* 
> The affected server permanently loses the ability to join the quorum. If multiple servers are affected, quorum may be lost. The only 
> recovery is to restart the process.
> *Affected code paths:*
>  - {{Learner.connectToLeader()}} → {{self.authLearner.authenticate()}} (Learner.java:354)
>  - {{QuorumCnxManager.initiateConnection()}} → {{authLearner.authenticate()}} (QuorumCnxManager.java:506)
>  - {{QuorumCnxManager.handleConnection()}} → {{authServer.authenticate()}} (QuorumCnxManager.java:633)
>  - {{LearnerHandler}} constructor → {{authServer.authenticate()}} (LearnerHandler.java:291)
> *Fix:* 
> Add a {{forceReLogin()}} method to {{Login}} that re-logins immediately (bypassing the minimum time check), and call it from 
> {{SaslQuorumAuthLearner.authenticate()}} and {{SaslQuorumAuthServer.authenticate()}} when authentication fails. This ensures the next authentication attempt uses 
> fresh credentials.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)