[jira] [Created] (KAFKA-7902) SASL/OAUTHBEARER can become unable to connect: javax.security.sasl.SaslException: Unable to find OAuth Bearer token in Subject's private credentials (size=2)

["Ron Dagostino (JIRA)" <ji...@apache.org>]

Ron Dagostino created KAFKA-7902:
------------------------------------

             Summary: SASL/OAUTHBEARER can become unable to connect: javax.security.sasl.SaslException: Unable to find OAuth Bearer token in Subject's private credentials (size=2) 
                 Key: KAFKA-7902
                 URL: https://issues.apache.org/jira/browse/KAFKA-7902
             Project: Kafka
          Issue Type: Bug
          Components: clients
    Affects Versions: 2.1.0, 2.0.1, 2.0.0
            Reporter: Ron Dagostino
            Assignee: Ron Dagostino


It is possible for a Java SASL/OAUTHBEARER client (either a non-broker producer/consumer client or a broker when acting as an inter-broker client) to end up in a state where it cannot connect to a new broker (or, if re-authentication as implemented by KIP-368 and merged for v2.2.0 were to be deployed and enabled, to be unable to re-authenticate). The error message looks like this:

{{Connection to node 1 failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Unable to find OAuth Bearer token in Subject's private credentials (size=2) [Caused by java.io.IOException: Unable to find OAuth Bearer token in Subject's private credentials (size=2)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.}}

The root cause of the problem begins at this point in the code:

[https://github.com/apache/kafka/blob/2.0/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/expiring/ExpiringCredentialRefreshingLogin.java#L378]:

The {{loginContext}} field doesn't get replaced with the old version stored away in the {{optionalLoginContextToLogout}} variable if/when the {{loginContext.login()}} call on line 381 throws an exception. *This is an unusual event* – the OAuth authorization server must be unavailable at the moment when the token refresh occurs – but when it does happen it puts the refresher thread instance in an invalid state because now its {{loginContext}} field represents the one that failed instead of the original one, which is now lost.  The current {{loginContext}} can't be logged out – it will throw an {{InvalidStateException}} if that is attempted because there is no token associated with it -- and the token associated with the login context that was lost can never be logged out and removed from the Subject's private credentials (because we don't retain a reference to it).  The net effect is that we end up with an extra token on the Subject's private credentials, which eventually results in the exception mentioned above when the client tries to authenticate to a broker.

So the chain of events is:

1) login failure upon token refresh causes the refresher thread's login context field to be incorrect, and the existing token on the Subject's private credentials will never be logged out/removed
 2) retry occurs in 10 seconds, potentially repeatedly until the authorization server is back online
 3) login succeeds, adding a second token to the Subject's private credentials (logout is then called on the login context set incorrectly in the most recent failure -- e.g. in step 1 -- which results in an exception, but this is not the real issue -- it is the 2 tokens on the Subject's private credentials that is the issue)
 4) At this point we now have 2 tokens on the Subject, and then at some point in the future the client tries to make a new connection, it sees the 2 tokens and throws an exception – BOOM! The client is now unable to connect (or re-authenticate if applicable) going forward.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)