You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "ZhenyuLi (Jira)" <ji...@apache.org> on 2026/03/25 19:20:00 UTC
[jira] [Updated] (ZOOKEEPER-5033) Quorum SASL authentication fails permanently after Login TGT refresh thread exits
[ https://issues.apache.org/jira/browse/ZOOKEEPER-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ZhenyuLi updated ZOOKEEPER-5033:
--------------------------------
Description:
When quorum SASL authentication is enabled ({{{}quorum.auth.enableSasl=true{}}}) with Kerberos, the {{Login}} class runs a background daemon thread to periodically
refresh the TGT. This thread can silently exit in several scenarios:
1. Clock skew between the host and KDC
2. TGT is not renewable
3. {{kinit -R}} fails after retry
4. {{reLogin()}} fails after retry
5. {{nextRefresh}} is in the past
After the thread exits, the TGT cached in the {{Subject}} eventually expires. When a Follower/Observer later needs to reconnect to the Leader (e.g., after a network
partition or leader switch), {{SaslQuorumAuthLearner.authenticate()}} uses the stale credentials from {{learnerLogin.getSubject()}} and fails with
{{{}SaslException{}}}.
The caller ({{{}QuorumPeer{}}} main loop) retries by going through {{{}LOOKING → FOLLOWING → connectToLeader → authenticate{}}}, but the {{authLearner}} object is created
once in {{QuorumPeer.initialize()}} and never recreated. The same stale {{Login}} and {{Subject}} are reused, causing every retry to fail indefinitely.
*Trigger conditions (all must be met):*
- {{quorum.auth.enableSasl=true}}
- {{quorum.auth.learnerRequireSasl=true}} and {{quorum.auth.serverRequireSasl=true}}
- Kerberos authentication (not DIGEST-MD5)
- Login refresh thread exits due to one of the above scenarios
- A reconnection event occurs after TGT expires
*Impact:*
The affected server permanently loses the ability to join the quorum. If multiple servers are affected, quorum may be lost. The only
recovery is to restart the process.
*Affected code paths:*
- {{Learner.connectToLeader()}} → {{self.authLearner.authenticate()}} (Learner.java:354)
- {{QuorumCnxManager.initiateConnection()}} → {{authLearner.authenticate()}} (QuorumCnxManager.java:506)
- {{QuorumCnxManager.handleConnection()}} → {{authServer.authenticate()}} (QuorumCnxManager.java:633)
- {{LearnerHandler}} constructor → {{authServer.authenticate()}} (LearnerHandler.java:291)
*Fix:*
Add a {{forceReLogin()}} method to {{Login}} that re-logins immediately (bypassing the minimum time check), and call it from
{{SaslQuorumAuthLearner.authenticate()}} and {{SaslQuorumAuthServer.authenticate()}} when authentication fails. This ensures the next authentication attempt uses
fresh credentials.
was:
When quorum SASL authentication is enabled ({{{}quorum.auth.enableSasl=true{}}}) with Kerberos, the {{Login}} class runs a background daemon thread to periodically
refresh the TGT. This thread can silently exit in several scenarios:
1. Clock skew between the host and KDC ({{{}Login.java:185-193{}}})
2. TGT is not renewable ({{{}Login.java:148-160{}}})
3. {{kinit -R}} fails after retry ({{{}Login.java:236-242{}}})
4. {{reLogin()}} fails after retry ({{{}Login.java:268-270{}}})
5. {{nextRefresh}} is in the past ({{{}Login.java:207-214{}}})
After the thread exits, the TGT cached in the {{Subject}} eventually expires. When a Follower/Observer later needs to reconnect to the Leader (e.g., after a network
partition or leader switch), {{SaslQuorumAuthLearner.authenticate()}} uses the stale credentials from {{learnerLogin.getSubject()}} and fails with
{{{}SaslException{}}}.
The caller ({{{}QuorumPeer{}}} main loop) retries by going through {{{}LOOKING → FOLLOWING → connectToLeader → authenticate{}}}, but the {{authLearner}} object is created
once in {{QuorumPeer.initialize()}} and never recreated. The same stale {{Login}} and {{Subject}} are reused, causing every retry to fail indefinitely.
*Trigger conditions (all must be met):*
- {{quorum.auth.enableSasl=true}}
- {{quorum.auth.learnerRequireSasl=true}} and {{quorum.auth.serverRequireSasl=true}}
- Kerberos authentication (not DIGEST-MD5)
- Login refresh thread exits due to one of the above scenarios
- A reconnection event occurs after TGT expires
*Impact:*
The affected server permanently loses the ability to join the quorum. If multiple servers are affected, quorum may be lost. The only
recovery is to restart the process.
*Affected code paths:*
- {{Learner.connectToLeader()}} → {{self.authLearner.authenticate()}} (Learner.java:354)
- {{QuorumCnxManager.initiateConnection()}} → {{authLearner.authenticate()}} (QuorumCnxManager.java:506)
- {{QuorumCnxManager.handleConnection()}} → {{authServer.authenticate()}} (QuorumCnxManager.java:633)
- {{LearnerHandler}} constructor → {{authServer.authenticate()}} (LearnerHandler.java:291)
*Fix:*
Add a {{forceReLogin()}} method to {{Login}} that re-logins immediately (bypassing the minimum time check), and call it from
{{SaslQuorumAuthLearner.authenticate()}} and {{SaslQuorumAuthServer.authenticate()}} when authentication fails. This ensures the next authentication attempt uses
fresh credentials.
> Quorum SASL authentication fails permanently after Login TGT refresh thread exits
> ----------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-5033
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5033
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum, server
> Affects Versions: 3.9.3
> Reporter: ZhenyuLi
> Priority: Major
>
> When quorum SASL authentication is enabled ({{{}quorum.auth.enableSasl=true{}}}) with Kerberos, the {{Login}} class runs a background daemon thread to periodically
> refresh the TGT. This thread can silently exit in several scenarios:
> 1. Clock skew between the host and KDC
> 2. TGT is not renewable
> 3. {{kinit -R}} fails after retry
> 4. {{reLogin()}} fails after retry
> 5. {{nextRefresh}} is in the past
> After the thread exits, the TGT cached in the {{Subject}} eventually expires. When a Follower/Observer later needs to reconnect to the Leader (e.g., after a network
> partition or leader switch), {{SaslQuorumAuthLearner.authenticate()}} uses the stale credentials from {{learnerLogin.getSubject()}} and fails with
> {{{}SaslException{}}}.
> The caller ({{{}QuorumPeer{}}} main loop) retries by going through {{{}LOOKING → FOLLOWING → connectToLeader → authenticate{}}}, but the {{authLearner}} object is created
> once in {{QuorumPeer.initialize()}} and never recreated. The same stale {{Login}} and {{Subject}} are reused, causing every retry to fail indefinitely.
> *Trigger conditions (all must be met):*
> - {{quorum.auth.enableSasl=true}}
> - {{quorum.auth.learnerRequireSasl=true}} and {{quorum.auth.serverRequireSasl=true}}
> - Kerberos authentication (not DIGEST-MD5)
> - Login refresh thread exits due to one of the above scenarios
> - A reconnection event occurs after TGT expires
> *Impact:*
> The affected server permanently loses the ability to join the quorum. If multiple servers are affected, quorum may be lost. The only
> recovery is to restart the process.
> *Affected code paths:*
> - {{Learner.connectToLeader()}} → {{self.authLearner.authenticate()}} (Learner.java:354)
> - {{QuorumCnxManager.initiateConnection()}} → {{authLearner.authenticate()}} (QuorumCnxManager.java:506)
> - {{QuorumCnxManager.handleConnection()}} → {{authServer.authenticate()}} (QuorumCnxManager.java:633)
> - {{LearnerHandler}} constructor → {{authServer.authenticate()}} (LearnerHandler.java:291)
> *Fix:*
> Add a {{forceReLogin()}} method to {{Login}} that re-logins immediately (bypassing the minimum time check), and call it from
> {{SaslQuorumAuthLearner.authenticate()}} and {{SaslQuorumAuthServer.authenticate()}} when authentication fails. This ensures the next authentication attempt uses
> fresh credentials.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)