[DISCUSS] SASL reauthentication, session expiry

[András Csaki <cs...@gmail.com>]

Hi Kafka Devs,

I'd like to discuss expected behavior and a potential bug with the SASL
reauthentication process.

I've opened KAFKA-13848 a while back and have a small project to reproduce
the problem here: https://github.com/acsaki/kafka-sasl-reauth

Briefly explained, OAuth clients remain able to produce/consume after they
have failed to reauthenticate, demonstrated with a short token expiry and a
killed Oauth server. :)

The problem seems to be in
SaslServerAuthenticator.ReauthInfo#calcCompletionTimesAndReturnSessionLifetimeMs
where sessionExpirationTimeNanos is only set when the session's calculated
lifetime is non-negative (token has not expired yet).

Because of this ReauthInfo#sessionExpirationTimeNanos remains null, in turn
making KafkaChannel#serverAuthenticationSessionExpired to always return
false, so SocketServer won't close the channel, leaving my producers and
consumers connected and happily producing and consuming.
You can see there's not much OAUTHBEARER specific in this behavior.

Looking at the if conditions in
calcCompletionTimesAndReturnSessionLifetimeMs it all seems rather
deliberate.
I've opened a very much work in progress and simplistic PR here:
https://github.com/apache/kafka/pull/12179
It only makes sure ReauthInfo#sessionExpirationTimeNanos gets set when
either credentials can expire or there's a max reauth time set. It actually
makes my producers with expired tokens die but it seems to break a lot of
assumptions in tests. (some of the tests I've started to fix but there are
many still broken)

I'd like to first discuss if this is indeed a problem worth investigating
more. Or maybe leaving clients with expired tokens connected is what we
want here so they may be able to reauthenticate eventually.


Best,
Andras

Re: [DISCUSS] SASL reauthentication, session expiry

[András Csaki <cs...@gmail.com>]

Hi,
Anyone to discuss this with?
I see a potential security issue in there since apparently clients can
produce/consume despite having an expired token. I have tried tuning
various timeouts to see if eventually they get disconnected to no avail.

One more likely case when this happens is when the client_secret changes
and a client having old credentials gets 401s constantly from the OAuth
server but remains connected to the broker.
It seems that all errors bubble up the ExpiringCredentialRefreshingLogin in
the Refresher thread that just keeps trying to relogin relentlessly.
There seems to be no way to handle these conditions in the actual user
code, no exceptions get back to my callbacks.

Best,
Andras

On Thu, May 19, 2022 at 10:30 AM András Csaki <cs...@gmail.com>
wrote:

> Hi Kafka Devs,
>
> I'd like to discuss expected behavior and a potential bug with the SASL
> reauthentication process.
>
> I've opened KAFKA-13848 a while back and have a small project to reproduce
> the problem here: https://github.com/acsaki/kafka-sasl-reauth
>
> Briefly explained, OAuth clients remain able to produce/consume after they
> have failed to reauthenticate, demonstrated with a short token expiry and a
> killed Oauth server. :)
>
> The problem seems to be in
> SaslServerAuthenticator.ReauthInfo#calcCompletionTimesAndReturnSessionLifetimeMs
> where sessionExpirationTimeNanos is only set when the session's calculated
> lifetime is non-negative (token has not expired yet).
>
> Because of this ReauthInfo#sessionExpirationTimeNanos remains null, in
> turn making KafkaChannel#serverAuthenticationSessionExpired to always
> return false, so SocketServer won't close the channel, leaving my producers
> and consumers connected and happily producing and consuming.
> You can see there's not much OAUTHBEARER specific in this behavior.
>
> Looking at the if conditions in
> calcCompletionTimesAndReturnSessionLifetimeMs it all seems rather
> deliberate.
> I've opened a very much work in progress and simplistic PR here:
> https://github.com/apache/kafka/pull/12179
> It only makes sure ReauthInfo#sessionExpirationTimeNanos gets set when
> either credentials can expire or there's a max reauth time set. It actually
> makes my producers with expired tokens die but it seems to break a lot of
> assumptions in tests. (some of the tests I've started to fix but there are
> many still broken)
>
> I'd like to first discuss if this is indeed a problem worth investigating
> more. Or maybe leaving clients with expired tokens connected is what we
> want here so they may be able to reauthenticate eventually.
>
>
> Best,
> Andras
>
>