You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Joel Shepherd <sh...@amazon.com> on 2025/06/30 16:37:35 UTC
[DISCUSS] CEP-50: Authentication Negotiation
Hello - We would like to propose CEP-50: Authentication Negotiation for
adoption by the community: <link> .
This CEP proposes minor changes to the initial handshake protocol
(OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
the node of the authenticators supported by the client, and changes in
the node's authentication-related areas to enable it to pick its
preferred authenticator for each client client connection. The CEP
explains why this approach is proposed, instead of implementing a
"negotiating authenticator".
Authentication negotiation will make it easier and safer for
administrators to migrate clusters to stronger authentication mechanisms
(including switching on authentication for a cluster that has been using
"allow-all" authentication) without downtime, and to support
environments where different clients prefer different authentication
mechanisms (e.g., username and password for ad-hoc cqlsh access, mutual
TLS for programmatic access, etc.), without having to pick a single
"lowest common denominator" authenticator for all. Additionally, the
proposed changes are intended to be backwards compatible for both
clients and nodes.
Thanks in advance for your time and feedback. Please keep the discussion
on this mailing list thread.
Thanks! -- Joel.
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Nate McCall <zz...@gmail.com>.
Thanks for submitting this, Joel. IMO this will be a valuable addition to
have and would benefit anyone running C* in a multi-cloud, multi-auth
environment (any big enterprise, really). Also echoing the appreciation for
such a detailed CEP!
Cheers,
-Nate
On Fri, Jul 4, 2025 at 6:39 AM Joel Shepherd <sh...@amazon.com> wrote:
> Thanks Andy - My hope/expectation is this would significantly reduce the
> amount of friction involved when either implementing or migrating to a new
> authenticator. I suspect it will benefit more complex environments too,
> when a single authenticator isn't ideal for all clients.
>
> At this point, the stickiest bits I've run into have involved logic that
> switches on "the" authenticator class, because of the hardcoded dependency
> on a specific authn implementation, and working out how to sanely extend
> the logic once it's possible for a single node to be using different
> authenticators for different client sessions. Solvable but will take some
> refactoring and might also generate debate about what the right behaviors
> are in that scenario. An example is the RoleManager which currently creates
> additional role attributes if the Password Authenticator is in use ...
> which, now that you mention it, I wonder if that's already broken for the
> MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>
> Anyway, thanks for the feedback -- Joel.
> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>
>
> Hi Joel,
>
> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>
> The capability to allow the server to support multiple authenticators
> would be very useful. CEP-34 added a
> 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously supporting
> both mTLS and Password authentication, primarily as a means to introduce
> mTLS auth without breaking password auth and also a possible gradual
> migration to mTLS, but this only works for combining these two particular
> authenticators, and also creates another authenticator to support.
>
> The migration to any new auth strategy would likely involve the need to
> simultaneously support an existing and new auth provider. I think the
> approach you describe is well described and should meet this need assuming
> users use a driver that supports it.
>
> In terms of the protocol, utilizing the capabilities of the existing
> OPTIONS, STARTUP and SUPPORTED messages to communicate what authenticators
> are supported/should be used is pretty clever as it shouldn't require a
> protocol version uprev, and hopefully wouldn't be too complicated for a
> driver to implement.
>
> Thanks,
> Andy
>
> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <sh...@amazon.com>
> wrote:
>
>> Erm ... and here's the CEP:
>>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>
>> (Thanks for the heads up, Abe ...)
>>
>> -- Joel.
>>
>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>> > Hello - We would like to propose CEP-50: Authentication Negotiation
>> > for adoption by the community: <link> .
>> >
>> > This CEP proposes minor changes to the initial handshake protocol
>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
>> > the node of the authenticators supported by the client, and changes in
>> > the node's authentication-related areas to enable it to pick its
>> > preferred authenticator for each client client connection. The CEP
>> > explains why this approach is proposed, instead of implementing a
>> > "negotiating authenticator".
>> >
>> > Authentication negotiation will make it easier and safer for
>> > administrators to migrate clusters to stronger authentication
>> > mechanisms (including switching on authentication for a cluster that
>> > has been using "allow-all" authentication) without downtime, and to
>> > support environments where different clients prefer different
>> > authentication mechanisms (e.g., username and password for ad-hoc
>> > cqlsh access, mutual TLS for programmatic access, etc.), without
>> > having to pick a single "lowest common denominator" authenticator for
>> > all. Additionally, the proposed changes are intended to be backwards
>> > compatible for both clients and nodes.
>> >
>> > Thanks in advance for your time and feedback. Please keep the
>> > discussion on this mailing list thread.
>> >
>> > Thanks! -- Joel.
>> >
>> >
>> >
>> >
>>
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Josh McKenzie <jm...@apache.org>.
> I'm concerned about config updates made through JMX not being persistent across restarts
Reasonably so yeah. I was thinking along the lines of "change to config that is persisted in config file", but now that you mention this I don't think we actually have a precedent for that yet. And the security implications of making durable config changes via a JMX endpoint or vtable mutation are... well, my gut reaction to that is aversion, let's put it that way. :)
Probably better to have the sidecar handle things like that as a control plane endpoint since it's in a good position architecturally to a) modify running node configs on co-located machines, b) persist those config changes to disk (and even (gasp) version them?!), and c) have auth on that.
On Thu, Jul 10, 2025, at 8:18 PM, Joel Shepherd wrote:
>
>
> On 7/8/2025 8:22 AM, Abe Ratnofsky wrote:
>> +1 (nb) from me as well. Would be nice to have a reference implementation of negotiated authentication in the Java driver; I’d happy to collaborate on that.
> I'd love some help with the driver[s]. I plan to do at least "an" implementation in the Java driver, but would really appreciate assistance in refining to the point that it could be considered a reference impl.
>
> Thanks! -- Joel.
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Joel Shepherd <sh...@amazon.com>.
On 7/8/2025 8:22 AM, Abe Ratnofsky wrote:
> +1 (nb) from me as well. Would be nice to have a reference implementation of negotiated authentication in the Java driver; I’d happy to collaborate on that.
I'd love some help with the driver[s]. I plan to do at least "an"
implementation in the Java driver, but would really appreciate
assistance in refining to the point that it could be considered a
reference impl.
Thanks! -- Joel.
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Abe Ratnofsky <ab...@aber.io>.
+1 (nb) from me as well. Would be nice to have a reference implementation of negotiated authentication in the Java driver; I’d happy to collaborate on that.
Really appreciate the amount of detail in this CEP as well, particularly for migration paths.
—
Abe
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Joel Shepherd <sh...@amazon.com>.
I've updated the CEP with a lot of your suggestions below.
One thing I'm holding off on is making "requiresAuthentication" a
JMX-controllable ("hot") property. I do like the idea of being able to
enable it without bouncing nodes. However, I'm concerned about config
updates made through JMX not being persistent across restarts,
potentially setting someone up to believe they have authentication in
place, only to discover that new or restarted nodes are allowing clients
to slide in without authentication (and with elevated permissions).
I don't know if settling that question is a blocker for the CEP ... in
any event, I'd like to give it more thought.
Thanks -- Joel.
On 7/10/2025 1:22 PM, Josh McKenzie wrote:
>
>> ... so that in the future new options could just be keys in the
>> multimap, which would not be considered a protocol change? Or placing
>> all the options -- current and future -- into the multimap?
> The former. Trying to think through defensive postures on the protocol
> to prevent future changes required to the client ecosystem, since at
> least right now it translates into 7+ different implementations for
> different language drivers to support new functionality.
>
> On Wed, Jul 9, 2025, at 8:18 PM, Joel Shepherd wrote:
>>
>> Hi Josh - Thanks for all the feedback: appreciate it. Responses to
>> specific points interwoven below ...
>>
>> On 7/9/2025 3:25 AM, Josh McKenzie wrote:
>>>
>>> Sorry for the delay in getting to this Joel. This is great work -
>>> really well thought out and put together. I'm a +1 on it; had the
>>> following observations or questions reviewing the CEP:
>>>
>>> Rather than going with a new discretely allowed option to a closed
>>> list of allowable options in STARTUP, what if we went the route we
>>> did with SUPPORTED and offered a variable [string multimap] so we
>>> could add future STARTUP options w/out fully revving the protocol
>>> spec in the future?
>>> (https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L462-L491
>>> <https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L462-L491>)
>>>
>> If I'm understanding, you're suggesting making a one-time protocol
>> change by adding something like:
>>
>> STARTUP: {
>> "CQL_VERSION": "3.11",
>> <other existing options>
>> "OPTIONS": <multimap of option name => value>
>> }
>>
>> ... so that in the future new options could just be keys in the
>> multimap, which would not be considered a protocol change? Or placing
>> all the options -- current and future -- into the multimap?
>>
>> I don't have a strong opinion on this. My weak opinion is that --
>> especially if the intent is to put all options in the multimap --
>> it's either going to require careful client-node coordination to roll
>> out the protocol change, or it's going to require the node to expect
>> and handle both flavors of the STARTUP message structure. Doable but
>> I'm not sure that part of the protocol changes so frequently that
>> it's worth the effort. I'm not going to argue if you or others have a
>> different and stronger opinion on it.
>>
>>>
>>> Any opinion on us deprecating the existing
>>> cassandra.yaml#authenticator: option to be renamed as
>>> "fallback_authenticator", and updating the text to reflect that? Or
>>> /default_authenticator/; something to denote through the naming that
>>> the role of that is secondary to the newly added
>>> /authenticator_negotiation/? I'm 50/50 on this; I don't love
>>> deprecation (knowing we're going to keep supporting the old
>>> nomenclature into perpetuity), but I do think the topic of auth is
>>> important enough that a little config disruption and maintenance
>>> here to steer new users to the new, more secure method of auth might
>>> warrant the change.
>>
>> I hadn't thought about that but I like the idea. I was striving for
>> maximum backwards compatibility, but you're right that the existing
>> key could continue to be supported, just not publicized. Going
>> forward, something like "default_authenticator" will be less
>> confusing than just "authenticator". I'll add it to the doc.
>>
>>>> `‘requireAuthentication’ should be set to true as soon as all
>>>> clients are using other authenticators.`
>>> A hot prop or mutable vtable (... did we ever do mutable / change
>>> config through vtables? Hm, looks like not yet:
>>> https://issues.apache.org/jira/browse/CASSANDRA-15254
>>> <https://issues.apache.org/jira/browse/CASSANDRA-15254>) so we could
>>> change this live on a cluster w/out bouncing would be nice. Be nicer
>>> if that change also coincided w/a change in config via 1 UX through
>>> the DB. ;) But that's a /different/ problem than what you're looking
>>> to address so worth deferring on that piece probably.
>>
>> Agree it'd be great to be able to enable authn and negotiation
>> without having to bounce nodes. Will add.
>>
>>> re: thundering herd risk - this tickled my memory. <snip> We have a
>>> separate executor specifically for auth messages in Dispatcher.java;
>>> might be worth keeping an eye on this to see if it proves to be a
>>> new bottleneck w/a more heavyweight negotiation on connection.
>>
>> Ah, thanks for pointing that out. I wouldn't expect STARTUP (which is
>> where negotiation will actually be resolved server-side) to be
>> significantly more expensive than it is today, but will keep an eye
>> on that. A second potential bottleneck might lie in the increased
>> overhead of the required OPTIONS/SUPPORTED handshake, which I believe
>> today is optional: the client can and often does initiate connection
>> with STARTUP alone. Again, while I wouldn't expect those to be very
>> expensive to handle, offloading them from the requestExecutor might
>> protect the workers serving the actual data plane traffic.
>>
>>> The JIRA link in the CEP just links to the ASF C* JIRA - I couldn't
>>> find a ticket for this CEP yet in JIRA. Generally we use that to
>>> link to a specific ticket; was a minor speed bump just FYI.
>>
>> I was deferring creating a JIRA until if/when the CEP was adopted,
>> but I'll create one today. Can always close it later.
>>
>>> And we have some prior art in the following:
>>> - CASSANDRA-13048: Support SASL mechanism negotiation in existing
>>> Authenticators
>>> - CASSANDRA-11471: Add SASL mechanism negotiation to the native protocol
>>>
>>> Might be good to link to those and once we get a JIRA up for CEP-50,
>>> we can flag those 2 as duplicates of it and close them out once it's
>>> done.
>>
>> Great: thanks for the pointers.
>>
>> One detail where CEP-50 varies a bit from the SASL RFC's description
>> of negotiation
>> (https://datatracker.ietf.org/doc/html/rfc4422#section-3.2
>> <https://datatracker.ietf.org/doc/html/rfc4422#section-3.2>) is that
>> the RFC suggests the server send the client a list of authenticators
>> to choose from, and the client responds with its chosen
>> authenticator. The RFC doesn't say that the protocol "MUST" or even
>> "SHOULD" operate this way, just the implementations "commonly" do. In
>> early drafts, I actually described the exchange as working this way,
>> but eventually backed away from it because it seems to create the
>> risk of a malicious or confused client picking the least secure
>> option offered by the server: the server loses some control. The
>> current proposal is that the client sends the server a list of auth
>> mechanisms that the client can support, and the server makes the
>> final decision about which to use, which discourages the client from
>> choosing the weakest usable option. (It doesn't outright prevent it
>> however.) I believe this is still SASL-compliant because SASL doesn't
>> mandate a particular exchange, but did want to call it out. I'll fold
>> this into the CEP as well.
>>
>> Thanks! -- Joel.
>>
>>>
>>> Overall - looks great. Again: +1 from me.
>>>
>>> On Tue, Jul 8, 2025, at 8:34 PM, Joel Shepherd wrote:
>>>>
>>>> Hi Doug - That's an interesting suggestion for a load test: I'll
>>>> include something like that in our plans.
>>>>
>>>> You're right about the logic in RoleManager: it should be doing the
>>>> right thing with MutualTlsWithPasswordFallbackAuthenticator.
>>>>
>>>> Thanks! -- Joel.
>>>>
>>>> On 7/8/2025 1:52 PM, Doug Rohrer wrote:
>>>>>
>>>>> +1 from me (I think committer +1s are "binding" on CEPs given our
>>>>> previous "how we do things" conversation, but either way, +1)...
>>>>>
>>>>> One interesting perf test to think about would be the difference
>>>>> between negotiated auth with MutualTlsAuthenticator and
>>>>> PasswordAuthenticator and the combined
>>>>> MutualTlsWithPasswordFallbackAuthenticator, as I think it'll
>>>>> provide a pretty good indication of the (hopefully negligible)
>>>>> performance difference when negotiating.
>>>>>
>>>>> Also, because I was curious about your last comment...
>>>>> MutualTlsWithPasswordFallbackAuthenticator _derives_ from
>>>>> PasswordAuthenticator, so *instanceof
>>>>> PasswordAuthenticator* checks return *true* for instances of it,
>>>>> and therefore (assuming you're talking about
>>>>> supportedOptions/alterableOptions in
>>>>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172
>>>>> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172>),
>>>>> it should work fine. Not that that helps _you_ solve your problem,
>>>>> but at least the existing classes should work.
>>>>>
>>>>> Thanks for putting the CEP together and working on the implementation!
>>>>>
>>>>> Doug
>>>>>
>>>>>> On Jul 3, 2025, at 2:38 PM, Joel Shepherd <sh...@amazon.com>
>>>>>> <ma...@amazon.com> wrote:
>>>>>>
>>>>>> Thanks Andy - My hope/expectation is this would significantly
>>>>>> reduce the amount of friction involved when either implementing
>>>>>> or migrating to a new authenticator. I suspect it will benefit
>>>>>> more complex environments too, when a single authenticator isn't
>>>>>> ideal for all clients.
>>>>>>
>>>>>> At this point, the stickiest bits I've run into have involved
>>>>>> logic that switches on "the" authenticator class, because of the
>>>>>> hardcoded dependency on a specific authn implementation, and
>>>>>> working out how to sanely extend the logic once it's possible for
>>>>>> a single node to be using different authenticators for different
>>>>>> client sessions. Solvable but will take some refactoring and
>>>>>> might also generate debate about what the right behaviors are in
>>>>>> that scenario. An example is the RoleManager which currently
>>>>>> creates additional role attributes if the Password Authenticator
>>>>>> is in use ... which, now that you mention it, I wonder if that's
>>>>>> already broken for the
>>>>>> MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>>>>>>
>>>>>> Anyway, thanks for the feedback -- Joel.
>>>>>>
>>>>>> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>>>>>>>
>>>>>>> Hi Joel,
>>>>>>>
>>>>>>> +1 (nb), I think this is a really good idea and well fleshed our
>>>>>>> CEP!
>>>>>>>
>>>>>>> The capability to allow the server to support multiple
>>>>>>> authenticators would be very useful. CEP-34 added a
>>>>>>> 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously
>>>>>>> supporting both mTLS and Password authentication, primarily as a
>>>>>>> means to introduce mTLS auth without breaking password auth and
>>>>>>> also a possible gradual migration to mTLS, but this only works
>>>>>>> for combining these two particular authenticators, and also
>>>>>>> creates another authenticator to support.
>>>>>>>
>>>>>>> The migration to any new auth strategy would likely involve the
>>>>>>> need to simultaneously support an existing and new auth
>>>>>>> provider. I think the approach you describe is well described
>>>>>>> and should meet this need assuming users use a driver that
>>>>>>> supports it.
>>>>>>>
>>>>>>> In terms of the protocol, utilizing the capabilities of the
>>>>>>> existing OPTIONS, STARTUP and SUPPORTED messages to communicate
>>>>>>> what authenticators are supported/should be used is pretty
>>>>>>> clever as it shouldn't require a protocol version uprev, and
>>>>>>> hopefully wouldn't be too complicated for a driver to implement.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andy
>>>>>>>
>>>>>>> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd
>>>>>>> <shepherd@amazon.com <ma...@amazon.com>> wrote:
>>>>>>>
>>>>>>> Erm ... and here's the CEP:
>>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation>
>>>>>>>
>>>>>>> (Thanks for the heads up, Abe ...)
>>>>>>>
>>>>>>> -- Joel.
>>>>>>>
>>>>>>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>>>>>>> > Hello - We would like to propose CEP-50: Authentication
>>>>>>> Negotiation
>>>>>>> > for adoption by the community: <link> .
>>>>>>> >
>>>>>>> > This CEP proposes minor changes to the initial handshake
>>>>>>> protocol
>>>>>>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a
>>>>>>> client to inform
>>>>>>> > the node of the authenticators supported by the client,
>>>>>>> and changes in
>>>>>>> > the node's authentication-related areas to enable it to
>>>>>>> pick its
>>>>>>> > preferred authenticator for each client client connection.
>>>>>>> The CEP
>>>>>>> > explains why this approach is proposed, instead of
>>>>>>> implementing a
>>>>>>> > "negotiating authenticator".
>>>>>>> >
>>>>>>> > Authentication negotiation will make it easier and safer for
>>>>>>> > administrators to migrate clusters to stronger authentication
>>>>>>> > mechanisms (including switching on authentication for a
>>>>>>> cluster that
>>>>>>> > has been using "allow-all" authentication) without
>>>>>>> downtime, and to
>>>>>>> > support environments where different clients prefer different
>>>>>>> > authentication mechanisms (e.g., username and password for
>>>>>>> ad-hoc
>>>>>>> > cqlsh access, mutual TLS for programmatic access, etc.),
>>>>>>> without
>>>>>>> > having to pick a single "lowest common denominator"
>>>>>>> authenticator for
>>>>>>> > all. Additionally, the proposed changes are intended to be
>>>>>>> backwards
>>>>>>> > compatible for both clients and nodes.
>>>>>>> >
>>>>>>> > Thanks in advance for your time and feedback. Please keep the
>>>>>>> > discussion on this mailing list thread.
>>>>>>> >
>>>>>>> > Thanks! -- Joel.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Josh McKenzie <jm...@apache.org>.
> ... so that in the future new options could just be keys in the multimap, which would not be considered a protocol change? Or placing all the options -- current and future -- into the multimap?
The former. Trying to think through defensive postures on the protocol to prevent future changes required to the client ecosystem, since at least right now it translates into 7+ different implementations for different language drivers to support new functionality.
On Wed, Jul 9, 2025, at 8:18 PM, Joel Shepherd wrote:
> Hi Josh - Thanks for all the feedback: appreciate it. Responses to specific points interwoven below ...
>
> On 7/9/2025 3:25 AM, Josh McKenzie wrote:
>>
>> Sorry for the delay in getting to this Joel. This is great work - really well thought out and put together. I'm a +1 on it; had the following observations or questions reviewing the CEP:
>>
>> Rather than going with a new discretely allowed option to a closed list of allowable options in STARTUP, what if we went the route we did with SUPPORTED and offered a variable [string multimap] so we could add future STARTUP options w/out fully revving the protocol spec in the future? (https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L462-L491)
>>
> If I'm understanding, you're suggesting making a one-time protocol change by adding something like:
>
>
> STARTUP: {
> "CQL_VERSION": "3.11",
> <other existing options>
> "OPTIONS": <multimap of option name => value>
> }
>
> ... so that in the future new options could just be keys in the multimap, which would not be considered a protocol change? Or placing all the options -- current and future -- into the multimap?
>
> I don't have a strong opinion on this. My weak opinion is that -- especially if the intent is to put all options in the multimap -- it's either going to require careful client-node coordination to roll out the protocol change, or it's going to require the node to expect and handle both flavors of the STARTUP message structure. Doable but I'm not sure that part of the protocol changes so frequently that it's worth the effort. I'm not going to argue if you or others have a different and stronger opinion on it.
>
>>
>> Any opinion on us deprecating the existing cassandra.yaml#authenticator: option to be renamed as "fallback_authenticator", and updating the text to reflect that? Or *default_authenticator*; something to denote through the naming that the role of that is secondary to the newly added *authenticator_negotiation*? I'm 50/50 on this; I don't love deprecation (knowing we're going to keep supporting the old nomenclature into perpetuity), but I do think the topic of auth is important enough that a little config disruption and maintenance here to steer new users to the new, more secure method of auth might warrant the change.
> I hadn't thought about that but I like the idea. I was striving for maximum backwards compatibility, but you're right that the existing key could continue to be supported, just not publicized. Going forward, something like "default_authenticator" will be less confusing than just "authenticator". I'll add it to the doc.
>
>>> `‘requireAuthentication’ should be set to true as soon as all clients are using other authenticators.`
>> A hot prop or mutable vtable (... did we ever do mutable / change config through vtables? Hm, looks like not yet: https://issues.apache.org/jira/browse/CASSANDRA-15254) so we could change this live on a cluster w/out bouncing would be nice. Be nicer if that change also coincided w/a change in config via 1 UX through the DB. ;) But that's a *different* problem than what you're looking to address so worth deferring on that piece probably.
> Agree it'd be great to be able to enable authn and negotiation without having to bounce nodes. Will add.
>
>> re: thundering herd risk - this tickled my memory. <snip> We have a separate executor specifically for auth messages in Dispatcher.java; might be worth keeping an eye on this to see if it proves to be a new bottleneck w/a more heavyweight negotiation on connection.
> Ah, thanks for pointing that out. I wouldn't expect STARTUP (which is where negotiation will actually be resolved server-side) to be significantly more expensive than it is today, but will keep an eye on that. A second potential bottleneck might lie in the increased overhead of the required OPTIONS/SUPPORTED handshake, which I believe today is optional: the client can and often does initiate connection with STARTUP alone. Again, while I wouldn't expect those to be very expensive to handle, offloading them from the requestExecutor might protect the workers serving the actual data plane traffic.
>
>> The JIRA link in the CEP just links to the ASF C* JIRA - I couldn't find a ticket for this CEP yet in JIRA. Generally we use that to link to a specific ticket; was a minor speed bump just FYI.
> I was deferring creating a JIRA until if/when the CEP was adopted, but I'll create one today. Can always close it later.
>
>> And we have some prior art in the following:
>> - CASSANDRA-13048: Support SASL mechanism negotiation in existing Authenticators
>> - CASSANDRA-11471: Add SASL mechanism negotiation to the native protocol
>>
>> Might be good to link to those and once we get a JIRA up for CEP-50, we can flag those 2 as duplicates of it and close them out once it's done.
> Great: thanks for the pointers.
>
> One detail where CEP-50 varies a bit from the SASL RFC's description of negotiation (https://datatracker.ietf.org/doc/html/rfc4422#section-3.2) is that the RFC suggests the server send the client a list of authenticators to choose from, and the client responds with its chosen authenticator. The RFC doesn't say that the protocol "MUST" or even "SHOULD" operate this way, just the implementations "commonly" do. In early drafts, I actually described the exchange as working this way, but eventually backed away from it because it seems to create the risk of a malicious or confused client picking the least secure option offered by the server: the server loses some control. The current proposal is that the client sends the server a list of auth mechanisms that the client can support, and the server makes the final decision about which to use, which discourages the client from choosing the weakest usable option. (It doesn't outright prevent it however.) I believe this is still SASL-compliant because SASL doesn't mandate a particular exchange, but did want to call it out. I'll fold this into the CEP as well.
>
> Thanks! -- Joel.
>
>>
>> Overall - looks great. Again: +1 from me.
>>
>> On Tue, Jul 8, 2025, at 8:34 PM, Joel Shepherd wrote:
>>> Hi Doug - That's an interesting suggestion for a load test: I'll include something like that in our plans.
>>>
>>> You're right about the logic in RoleManager: it should be doing the right thing with MutualTlsWithPasswordFallbackAuthenticator.
>>>
>>> Thanks! -- Joel.
>>>
>>> On 7/8/2025 1:52 PM, Doug Rohrer wrote:
>>>>
>>>> +1 from me (I think committer +1s are "binding" on CEPs given our previous "how we do things" conversation, but either way, +1)...
>>>>
>>>> One interesting perf test to think about would be the difference between negotiated auth with MutualTlsAuthenticator and PasswordAuthenticator and the combined MutualTlsWithPasswordFallbackAuthenticator, as I think it'll provide a pretty good indication of the (hopefully negligible) performance difference when negotiating.
>>>>
>>>> Also, because I was curious about your last comment... MutualTlsWithPasswordFallbackAuthenticator _derives_ from PasswordAuthenticator, so *instanceof PasswordAuthenticator* checks return *true* for instances of it, and therefore (assuming you're talking about supportedOptions/alterableOptions in https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172), it should work fine. Not that that helps _you_ solve your problem, but at least the existing classes should work.
>>>>
>>>> Thanks for putting the CEP together and working on the implementation!
>>>>
>>>> Doug
>>>>
>>>>> On Jul 3, 2025, at 2:38 PM, Joel Shepherd <sh...@amazon.com> wrote:
>>>>>
>>>>> Thanks Andy - My hope/expectation is this would significantly reduce the amount of friction involved when either implementing or migrating to a new authenticator. I suspect it will benefit more complex environments too, when a single authenticator isn't ideal for all clients.
>>>>>
>>>>> At this point, the stickiest bits I've run into have involved logic that switches on "the" authenticator class, because of the hardcoded dependency on a specific authn implementation, and working out how to sanely extend the logic once it's possible for a single node to be using different authenticators for different client sessions. Solvable but will take some refactoring and might also generate debate about what the right behaviors are in that scenario. An example is the RoleManager which currently creates additional role attributes if the Password Authenticator is in use ... which, now that you mention it, I wonder if that's already broken for the MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>>>>>
>>>>> Anyway, thanks for the feedback -- Joel.
>>>>>
>>>>> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>>>>>>
>>>>>> Hi Joel,
>>>>>>
>>>>>> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>>>>>>
>>>>>> The capability to allow the server to support multiple authenticators would be very useful. CEP-34 added a 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously supporting both mTLS and Password authentication, primarily as a means to introduce mTLS auth without breaking password auth and also a possible gradual migration to mTLS, but this only works for combining these two particular authenticators, and also creates another authenticator to support.
>>>>>>
>>>>>> The migration to any new auth strategy would likely involve the need to simultaneously support an existing and new auth provider. I think the approach you describe is well described and should meet this need assuming users use a driver that supports it.
>>>>>>
>>>>>> In terms of the protocol, utilizing the capabilities of the existing OPTIONS, STARTUP and SUPPORTED messages to communicate what authenticators are supported/should be used is pretty clever as it shouldn't require a protocol version uprev, and hopefully wouldn't be too complicated for a driver to implement.
>>>>>>
>>>>>> Thanks,
>>>>>> Andy
>>>>>>
>>>>>> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <sh...@amazon.com> wrote:
>>>>>>> Erm ... and here's the CEP:
>>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>>>>>>
>>>>>>> (Thanks for the heads up, Abe ...)
>>>>>>>
>>>>>>> -- Joel.
>>>>>>>
>>>>>>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>>>>>>> > Hello - We would like to propose CEP-50: Authentication Negotiation
>>>>>>> > for adoption by the community: <link> .
>>>>>>> >
>>>>>>> > This CEP proposes minor changes to the initial handshake protocol
>>>>>>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
>>>>>>> > the node of the authenticators supported by the client, and changes in
>>>>>>> > the node's authentication-related areas to enable it to pick its
>>>>>>> > preferred authenticator for each client client connection. The CEP
>>>>>>> > explains why this approach is proposed, instead of implementing a
>>>>>>> > "negotiating authenticator".
>>>>>>> >
>>>>>>> > Authentication negotiation will make it easier and safer for
>>>>>>> > administrators to migrate clusters to stronger authentication
>>>>>>> > mechanisms (including switching on authentication for a cluster that
>>>>>>> > has been using "allow-all" authentication) without downtime, and to
>>>>>>> > support environments where different clients prefer different
>>>>>>> > authentication mechanisms (e.g., username and password for ad-hoc
>>>>>>> > cqlsh access, mutual TLS for programmatic access, etc.), without
>>>>>>> > having to pick a single "lowest common denominator" authenticator for
>>>>>>> > all. Additionally, the proposed changes are intended to be backwards
>>>>>>> > compatible for both clients and nodes.
>>>>>>> >
>>>>>>> > Thanks in advance for your time and feedback. Please keep the
>>>>>>> > discussion on this mailing list thread.
>>>>>>> >
>>>>>>> > Thanks! -- Joel.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Joel Shepherd <sh...@amazon.com>.
Hi Josh - Thanks for all the feedback: appreciate it. Responses to
specific points interwoven below ...
On 7/9/2025 3:25 AM, Josh McKenzie wrote:
>
> Sorry for the delay in getting to this Joel. This is great work -
> really well thought out and put together. I'm a +1 on it; had the
> following observations or questions reviewing the CEP:
>
> Rather than going with a new discretely allowed option to a closed
> list of allowable options in STARTUP, what if we went the route we did
> with SUPPORTED and offered a variable [string multimap] so we could
> add future STARTUP options w/out fully revving the protocol spec in
> the future?
> (https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L462-L491)
>
If I'm understanding, you're suggesting making a one-time protocol
change by adding something like:
STARTUP: {
"CQL_VERSION": "3.11",
<other existing options>
"OPTIONS": <multimap of option name => value>
}
... so that in the future new options could just be keys in the
multimap, which would not be considered a protocol change? Or placing
all the options -- current and future -- into the multimap?
I don't have a strong opinion on this. My weak opinion is that --
especially if the intent is to put all options in the multimap -- it's
either going to require careful client-node coordination to roll out the
protocol change, or it's going to require the node to expect and handle
both flavors of the STARTUP message structure. Doable but I'm not sure
that part of the protocol changes so frequently that it's worth the
effort. I'm not going to argue if you or others have a different and
stronger opinion on it.
> Any opinion on us deprecating the existing
> cassandra.yaml#authenticator: option to be renamed as
> "fallback_authenticator", and updating the text to reflect that? Or
> /default_authenticator/; something to denote through the naming that
> the role of that is secondary to the newly added
> /authenticator_negotiation/? I'm 50/50 on this; I don't love
> deprecation (knowing we're going to keep supporting the old
> nomenclature into perpetuity), but I do think the topic of auth is
> important enough that a little config disruption and maintenance here
> to steer new users to the new, more secure method of auth might
> warrant the change.
I hadn't thought about that but I like the idea. I was striving for
maximum backwards compatibility, but you're right that the existing key
could continue to be supported, just not publicized. Going forward,
something like "default_authenticator" will be less confusing than just
"authenticator". I'll add it to the doc.
>> `‘requireAuthentication’ should be set to true as soon as all clients
>> are using other authenticators.`
> A hot prop or mutable vtable (... did we ever do mutable / change
> config through vtables? Hm, looks like not yet:
> https://issues.apache.org/jira/browse/CASSANDRA-15254) so we could
> change this live on a cluster w/out bouncing would be nice. Be nicer
> if that change also coincided w/a change in config via 1 UX through
> the DB. ;) But that's a /different/ problem than what you're looking
> to address so worth deferring on that piece probably.
Agree it'd be great to be able to enable authn and negotiation without
having to bounce nodes. Will add.
> re: thundering herd risk - this tickled my memory. <snip> We have a
> separate executor specifically for auth messages in Dispatcher.java;
> might be worth keeping an eye on this to see if it proves to be a new
> bottleneck w/a more heavyweight negotiation on connection.
Ah, thanks for pointing that out. I wouldn't expect STARTUP (which is
where negotiation will actually be resolved server-side) to be
significantly more expensive than it is today, but will keep an eye on
that. A second potential bottleneck might lie in the increased overhead
of the required OPTIONS/SUPPORTED handshake, which I believe today is
optional: the client can and often does initiate connection with STARTUP
alone. Again, while I wouldn't expect those to be very expensive to
handle, offloading them from the requestExecutor might protect the
workers serving the actual data plane traffic.
> The JIRA link in the CEP just links to the ASF C* JIRA - I couldn't
> find a ticket for this CEP yet in JIRA. Generally we use that to link
> to a specific ticket; was a minor speed bump just FYI.
I was deferring creating a JIRA until if/when the CEP was adopted, but
I'll create one today. Can always close it later.
> And we have some prior art in the following:
> - CASSANDRA-13048: Support SASL mechanism negotiation in existing
> Authenticators
> - CASSANDRA-11471: Add SASL mechanism negotiation to the native protocol
>
> Might be good to link to those and once we get a JIRA up for CEP-50,
> we can flag those 2 as duplicates of it and close them out once it's done.
Great: thanks for the pointers.
One detail where CEP-50 varies a bit from the SASL RFC's description of
negotiation (https://datatracker.ietf.org/doc/html/rfc4422#section-3.2)
is that the RFC suggests the server send the client a list of
authenticators to choose from, and the client responds with its chosen
authenticator. The RFC doesn't say that the protocol "MUST" or even
"SHOULD" operate this way, just the implementations "commonly" do. In
early drafts, I actually described the exchange as working this way, but
eventually backed away from it because it seems to create the risk of a
malicious or confused client picking the least secure option offered by
the server: the server loses some control. The current proposal is that
the client sends the server a list of auth mechanisms that the client
can support, and the server makes the final decision about which to use,
which discourages the client from choosing the weakest usable option.
(It doesn't outright prevent it however.) I believe this is still
SASL-compliant because SASL doesn't mandate a particular exchange, but
did want to call it out. I'll fold this into the CEP as well.
Thanks! -- Joel.
>
> Overall - looks great. Again: +1 from me.
>
> On Tue, Jul 8, 2025, at 8:34 PM, Joel Shepherd wrote:
>>
>> Hi Doug - That's an interesting suggestion for a load test: I'll
>> include something like that in our plans.
>>
>> You're right about the logic in RoleManager: it should be doing the
>> right thing with MutualTlsWithPasswordFallbackAuthenticator.
>>
>> Thanks! -- Joel.
>>
>> On 7/8/2025 1:52 PM, Doug Rohrer wrote:
>>>
>>> +1 from me (I think committer +1s are "binding" on CEPs given our
>>> previous "how we do things" conversation, but either way, +1)...
>>>
>>> One interesting perf test to think about would be the difference
>>> between negotiated auth with MutualTlsAuthenticator and
>>> PasswordAuthenticator and the combined
>>> MutualTlsWithPasswordFallbackAuthenticator, as I think it'll provide
>>> a pretty good indication of the (hopefully negligible) performance
>>> difference when negotiating.
>>>
>>> Also, because I was curious about your last comment...
>>> MutualTlsWithPasswordFallbackAuthenticator _derives_ from
>>> PasswordAuthenticator, so *instanceof PasswordAuthenticator* checks
>>> return *true* for instances of it, and therefore (assuming you're
>>> talking about supportedOptions/alterableOptions in
>>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172
>>> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172>),
>>> it should work fine. Not that that helps _you_ solve your problem,
>>> but at least the existing classes should work.
>>>
>>> Thanks for putting the CEP together and working on the implementation!
>>>
>>> Doug
>>>
>>>> On Jul 3, 2025, at 2:38 PM, Joel Shepherd <sh...@amazon.com>
>>>> <ma...@amazon.com> wrote:
>>>>
>>>> Thanks Andy - My hope/expectation is this would significantly
>>>> reduce the amount of friction involved when either implementing or
>>>> migrating to a new authenticator. I suspect it will benefit more
>>>> complex environments too, when a single authenticator isn't ideal
>>>> for all clients.
>>>>
>>>> At this point, the stickiest bits I've run into have involved logic
>>>> that switches on "the" authenticator class, because of the
>>>> hardcoded dependency on a specific authn implementation, and
>>>> working out how to sanely extend the logic once it's possible for a
>>>> single node to be using different authenticators for different
>>>> client sessions. Solvable but will take some refactoring and might
>>>> also generate debate about what the right behaviors are in that
>>>> scenario. An example is the RoleManager which currently creates
>>>> additional role attributes if the Password Authenticator is in use
>>>> ... which, now that you mention it, I wonder if that's already
>>>> broken for the MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>>>>
>>>> Anyway, thanks for the feedback -- Joel.
>>>>
>>>> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>>>>>
>>>>> Hi Joel,
>>>>>
>>>>> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>>>>>
>>>>> The capability to allow the server to support multiple
>>>>> authenticators would be very useful. CEP-34 added a
>>>>> 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously
>>>>> supporting both mTLS and Password authentication, primarily as a
>>>>> means to introduce mTLS auth without breaking password auth and
>>>>> also a possible gradual migration to mTLS, but this only works for
>>>>> combining these two particular authenticators, and also creates
>>>>> another authenticator to support.
>>>>>
>>>>> The migration to any new auth strategy would likely involve the
>>>>> need to simultaneously support an existing and new auth provider.
>>>>> I think the approach you describe is well described and should
>>>>> meet this need assuming users use a driver that supports it.
>>>>>
>>>>> In terms of the protocol, utilizing the capabilities of the
>>>>> existing OPTIONS, STARTUP and SUPPORTED messages to communicate
>>>>> what authenticators are supported/should be used is pretty clever
>>>>> as it shouldn't require a protocol version uprev, and hopefully
>>>>> wouldn't be too complicated for a driver to implement.
>>>>>
>>>>> Thanks,
>>>>> Andy
>>>>>
>>>>> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd
>>>>> <shepherd@amazon.com <ma...@amazon.com>> wrote:
>>>>>
>>>>> Erm ... and here's the CEP:
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation>
>>>>>
>>>>> (Thanks for the heads up, Abe ...)
>>>>>
>>>>> -- Joel.
>>>>>
>>>>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>>>>> > Hello - We would like to propose CEP-50: Authentication
>>>>> Negotiation
>>>>> > for adoption by the community: <link> .
>>>>> >
>>>>> > This CEP proposes minor changes to the initial handshake
>>>>> protocol
>>>>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client
>>>>> to inform
>>>>> > the node of the authenticators supported by the client, and
>>>>> changes in
>>>>> > the node's authentication-related areas to enable it to pick
>>>>> its
>>>>> > preferred authenticator for each client client connection.
>>>>> The CEP
>>>>> > explains why this approach is proposed, instead of
>>>>> implementing a
>>>>> > "negotiating authenticator".
>>>>> >
>>>>> > Authentication negotiation will make it easier and safer for
>>>>> > administrators to migrate clusters to stronger authentication
>>>>> > mechanisms (including switching on authentication for a
>>>>> cluster that
>>>>> > has been using "allow-all" authentication) without downtime,
>>>>> and to
>>>>> > support environments where different clients prefer different
>>>>> > authentication mechanisms (e.g., username and password for
>>>>> ad-hoc
>>>>> > cqlsh access, mutual TLS for programmatic access, etc.),
>>>>> without
>>>>> > having to pick a single "lowest common denominator"
>>>>> authenticator for
>>>>> > all. Additionally, the proposed changes are intended to be
>>>>> backwards
>>>>> > compatible for both clients and nodes.
>>>>> >
>>>>> > Thanks in advance for your time and feedback. Please keep the
>>>>> > discussion on this mailing list thread.
>>>>> >
>>>>> > Thanks! -- Joel.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Josh McKenzie <jm...@apache.org>.
Sorry for the delay in getting to this Joel. This is great work - really well thought out and put together. I'm a +1 on it; had the following observations or questions reviewing the CEP:
Rather than going with a new discretely allowed option to a closed list of allowable options in STARTUP, what if we went the route we did with SUPPORTED and offered a variable [string multimap] so we could add future STARTUP options w/out fully revving the protocol spec in the future? (https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L462-L491)
Any opinion on us deprecating the existing cassandra.yaml#authenticator: option to be renamed as "fallback_authenticator", and updating the text to reflect that? Or *default_authenticator*; something to denote through the naming that the role of that is secondary to the newly added *authenticator_negotiation*? I'm 50/50 on this; I don't love deprecation (knowing we're going to keep supporting the old nomenclature into perpetuity), but I do think the topic of auth is important enough that a little config disruption and maintenance here to steer new users to the new, more secure method of auth might warrant the change.
> `‘requireAuthentication’ should be set to true as soon as all clients are using other authenticators.`
A hot prop or mutable vtable (... did we ever do mutable / change config through vtables? Hm, looks like not yet: https://issues.apache.org/jira/browse/CASSANDRA-15254) so we could change this live on a cluster w/out bouncing would be nice. Be nicer if that change also coincided w/a change in config via 1 UX through the DB. ;) But that's a *different* problem than what you're looking to address so worth deferring on that piece probably.
re: thundering herd risk - this tickled my memory. We did some work on pre-caching things in the Auth cache and I *vaguely recall* getting it into its own ThreadPool; I was upstreaming other peoples' code so it's not as durable in my memory as usual. https://issues.apache.org/jira/browse/CASSANDRA-16958 and https://issues.apache.org/jira/browse/CASSANDRA-16957 come to mind. Ah, there we are: https://issues.apache.org/jira/browse/CASSANDRA-17812. We have a separate executor specifically for auth messages in Dispatcher.java; might be worth keeping an eye on this to see if it proves to be a new bottleneck w/a more heavyweight negotiation on connection.
The JIRA link in the CEP just links to the ASF C* JIRA - I couldn't find a ticket for this CEP yet in JIRA. Generally we use that to link to a specific ticket; was a minor speed bump just FYI.
And we have some prior art in the following:
- CASSANDRA-13048: Support SASL mechanism negotiation in existing Authenticators
- CASSANDRA-11471: Add SASL mechanism negotiation to the native protocol
Might be good to link to those and once we get a JIRA up for CEP-50, we can flag those 2 as duplicates of it and close them out once it's done.
Overall - looks great. Again: +1 from me.
On Tue, Jul 8, 2025, at 8:34 PM, Joel Shepherd wrote:
> Hi Doug - That's an interesting suggestion for a load test: I'll include something like that in our plans.
>
> You're right about the logic in RoleManager: it should be doing the right thing with MutualTlsWithPasswordFallbackAuthenticator.
>
> Thanks! -- Joel.
>
> On 7/8/2025 1:52 PM, Doug Rohrer wrote:
>>
>> +1 from me (I think committer +1s are "binding" on CEPs given our previous "how we do things" conversation, but either way, +1)...
>>
>> One interesting perf test to think about would be the difference between negotiated auth with MutualTlsAuthenticator and PasswordAuthenticator and the combined MutualTlsWithPasswordFallbackAuthenticator, as I think it'll provide a pretty good indication of the (hopefully negligible) performance difference when negotiating.
>>
>> Also, because I was curious about your last comment... MutualTlsWithPasswordFallbackAuthenticator _derives_ from PasswordAuthenticator, so *instanceof PasswordAuthenticator* checks return *true* for instances of it, and therefore (assuming you're talking about supportedOptions/alterableOptions in https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172), it should work fine. Not that that helps _you_ solve your problem, but at least the existing classes should work.
>>
>> Thanks for putting the CEP together and working on the implementation!
>>
>> Doug
>>
>>> On Jul 3, 2025, at 2:38 PM, Joel Shepherd <sh...@amazon.com> wrote:
>>>
>>> Thanks Andy - My hope/expectation is this would significantly reduce the amount of friction involved when either implementing or migrating to a new authenticator. I suspect it will benefit more complex environments too, when a single authenticator isn't ideal for all clients.
>>>
>>> At this point, the stickiest bits I've run into have involved logic that switches on "the" authenticator class, because of the hardcoded dependency on a specific authn implementation, and working out how to sanely extend the logic once it's possible for a single node to be using different authenticators for different client sessions. Solvable but will take some refactoring and might also generate debate about what the right behaviors are in that scenario. An example is the RoleManager which currently creates additional role attributes if the Password Authenticator is in use ... which, now that you mention it, I wonder if that's already broken for the MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>>>
>>> Anyway, thanks for the feedback -- Joel.
>>>
>>> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>>>>
>>>> Hi Joel,
>>>>
>>>> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>>>>
>>>> The capability to allow the server to support multiple authenticators would be very useful. CEP-34 added a 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously supporting both mTLS and Password authentication, primarily as a means to introduce mTLS auth without breaking password auth and also a possible gradual migration to mTLS, but this only works for combining these two particular authenticators, and also creates another authenticator to support.
>>>>
>>>> The migration to any new auth strategy would likely involve the need to simultaneously support an existing and new auth provider. I think the approach you describe is well described and should meet this need assuming users use a driver that supports it.
>>>>
>>>> In terms of the protocol, utilizing the capabilities of the existing OPTIONS, STARTUP and SUPPORTED messages to communicate what authenticators are supported/should be used is pretty clever as it shouldn't require a protocol version uprev, and hopefully wouldn't be too complicated for a driver to implement.
>>>>
>>>> Thanks,
>>>> Andy
>>>>
>>>> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <sh...@amazon.com> wrote:
>>>>> Erm ... and here's the CEP:
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>>>>
>>>>> (Thanks for the heads up, Abe ...)
>>>>>
>>>>> -- Joel.
>>>>>
>>>>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>>>>> > Hello - We would like to propose CEP-50: Authentication Negotiation
>>>>> > for adoption by the community: <link> .
>>>>> >
>>>>> > This CEP proposes minor changes to the initial handshake protocol
>>>>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
>>>>> > the node of the authenticators supported by the client, and changes in
>>>>> > the node's authentication-related areas to enable it to pick its
>>>>> > preferred authenticator for each client client connection. The CEP
>>>>> > explains why this approach is proposed, instead of implementing a
>>>>> > "negotiating authenticator".
>>>>> >
>>>>> > Authentication negotiation will make it easier and safer for
>>>>> > administrators to migrate clusters to stronger authentication
>>>>> > mechanisms (including switching on authentication for a cluster that
>>>>> > has been using "allow-all" authentication) without downtime, and to
>>>>> > support environments where different clients prefer different
>>>>> > authentication mechanisms (e.g., username and password for ad-hoc
>>>>> > cqlsh access, mutual TLS for programmatic access, etc.), without
>>>>> > having to pick a single "lowest common denominator" authenticator for
>>>>> > all. Additionally, the proposed changes are intended to be backwards
>>>>> > compatible for both clients and nodes.
>>>>> >
>>>>> > Thanks in advance for your time and feedback. Please keep the
>>>>> > discussion on this mailing list thread.
>>>>> >
>>>>> > Thanks! -- Joel.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Joel Shepherd <sh...@amazon.com>.
Hi Doug - That's an interesting suggestion for a load test: I'll include
something like that in our plans.
You're right about the logic in RoleManager: it should be doing the
right thing with MutualTlsWithPasswordFallbackAuthenticator.
Thanks! -- Joel.
On 7/8/2025 1:52 PM, Doug Rohrer wrote:
>
> +1 from me (I think committer +1s are "binding" on CEPs given our
> previous "how we do things" conversation, but either way, +1)...
>
> One interesting perf test to think about would be the difference
> between negotiated auth with MutualTlsAuthenticator and
> PasswordAuthenticator and the combined
> MutualTlsWithPasswordFallbackAuthenticator, as I think it'll provide a
> pretty good indication of the (hopefully negligible) performance
> difference when negotiating.
>
> Also, because I was curious about your last comment...
> MutualTlsWithPasswordFallbackAuthenticator _derives_ from
> PasswordAuthenticator, so *instanceof PasswordAuthenticator* checks
> return *true* for instances of it, and therefore (assuming you're
> talking about supportedOptions/alterableOptions in
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172),
> it should work fine. Not that that helps _you_ solve your problem, but
> at least the existing classes should work.
>
> Thanks for putting the CEP together and working on the implementation!
>
> Doug
>
>> On Jul 3, 2025, at 2:38 PM, Joel Shepherd <sh...@amazon.com> wrote:
>>
>> Thanks Andy - My hope/expectation is this would significantly reduce
>> the amount of friction involved when either implementing or migrating
>> to a new authenticator. I suspect it will benefit more complex
>> environments too, when a single authenticator isn't ideal for all
>> clients.
>>
>> At this point, the stickiest bits I've run into have involved logic
>> that switches on "the" authenticator class, because of the hardcoded
>> dependency on a specific authn implementation, and working out how to
>> sanely extend the logic once it's possible for a single node to be
>> using different authenticators for different client sessions.
>> Solvable but will take some refactoring and might also generate
>> debate about what the right behaviors are in that scenario. An
>> example is the RoleManager which currently creates additional role
>> attributes if the Password Authenticator is in use ... which, now
>> that you mention it, I wonder if that's already broken for the
>> MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>>
>> Anyway, thanks for the feedback -- Joel.
>>
>> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>>>
>>> Hi Joel,
>>>
>>> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>>>
>>> The capability to allow the server to support multiple
>>> authenticators would be very useful. CEP-34 added a
>>> 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously
>>> supporting both mTLS and Password authentication, primarily as a
>>> means to introduce mTLS auth without breaking password auth and also
>>> a possible gradual migration to mTLS, but this only works for
>>> combining these two particular authenticators, and also creates
>>> another authenticator to support.
>>>
>>> The migration to any new auth strategy would likely involve the need
>>> to simultaneously support an existing and new auth provider. I
>>> think the approach you describe is well described and should meet
>>> this need assuming users use a driver that supports it.
>>>
>>> In terms of the protocol, utilizing the capabilities of the existing
>>> OPTIONS, STARTUP and SUPPORTED messages to communicate what
>>> authenticators are supported/should be used is pretty clever as it
>>> shouldn't require a protocol version uprev, and hopefully wouldn't
>>> be too complicated for a driver to implement.
>>>
>>> Thanks,
>>> Andy
>>>
>>> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <sh...@amazon.com>
>>> wrote:
>>>
>>> Erm ... and here's the CEP:
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>>
>>> (Thanks for the heads up, Abe ...)
>>>
>>> -- Joel.
>>>
>>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>>> > Hello - We would like to propose CEP-50: Authentication
>>> Negotiation
>>> > for adoption by the community: <link> .
>>> >
>>> > This CEP proposes minor changes to the initial handshake protocol
>>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client
>>> to inform
>>> > the node of the authenticators supported by the client, and
>>> changes in
>>> > the node's authentication-related areas to enable it to pick its
>>> > preferred authenticator for each client client connection. The
>>> CEP
>>> > explains why this approach is proposed, instead of implementing a
>>> > "negotiating authenticator".
>>> >
>>> > Authentication negotiation will make it easier and safer for
>>> > administrators to migrate clusters to stronger authentication
>>> > mechanisms (including switching on authentication for a
>>> cluster that
>>> > has been using "allow-all" authentication) without downtime,
>>> and to
>>> > support environments where different clients prefer different
>>> > authentication mechanisms (e.g., username and password for ad-hoc
>>> > cqlsh access, mutual TLS for programmatic access, etc.), without
>>> > having to pick a single "lowest common denominator"
>>> authenticator for
>>> > all. Additionally, the proposed changes are intended to be
>>> backwards
>>> > compatible for both clients and nodes.
>>> >
>>> > Thanks in advance for your time and feedback. Please keep the
>>> > discussion on this mailing list thread.
>>> >
>>> > Thanks! -- Joel.
>>> >
>>> >
>>> >
>>> >
>>>
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Doug Rohrer <dr...@apple.com>.
+1 from me (I think committer +1s are "binding" on CEPs given our previous "how we do things" conversation, but either way, +1)...
One interesting perf test to think about would be the difference between negotiated auth with MutualTlsAuthenticator and PasswordAuthenticator and the combined MutualTlsWithPasswordFallbackAuthenticator, as I think it'll provide a pretty good indication of the (hopefully negligible) performance difference when negotiating.
Also, because I was curious about your last comment... MutualTlsWithPasswordFallbackAuthenticator derives from PasswordAuthenticator, so instanceof PasswordAuthenticator checks return true for instances of it, and therefore (assuming you're talking about supportedOptions/alterableOptions in https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L167-L172), it should work fine. Not that that helps you solve your problem, but at least the existing classes should work.
Thanks for putting the CEP together and working on the implementation!
Doug
> On Jul 3, 2025, at 2:38 PM, Joel Shepherd <sh...@amazon.com> wrote:
>
> Thanks Andy - My hope/expectation is this would significantly reduce the amount of friction involved when either implementing or migrating to a new authenticator. I suspect it will benefit more complex environments too, when a single authenticator isn't ideal for all clients.
>
> At this point, the stickiest bits I've run into have involved logic that switches on "the" authenticator class, because of the hardcoded dependency on a specific authn implementation, and working out how to sanely extend the logic once it's possible for a single node to be using different authenticators for different client sessions. Solvable but will take some refactoring and might also generate debate about what the right behaviors are in that scenario. An example is the RoleManager which currently creates additional role attributes if the Password Authenticator is in use ... which, now that you mention it, I wonder if that's already broken for the MutualTlsWithPasswordFallbackAuthenticator. Hmm.
>
> Anyway, thanks for the feedback -- Joel.
>
> On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>>
>> Hi Joel,
>>
>> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>>
>> The capability to allow the server to support multiple authenticators would be very useful. CEP-34 added a 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously supporting both mTLS and Password authentication, primarily as a means to introduce mTLS auth without breaking password auth and also a possible gradual migration to mTLS, but this only works for combining these two particular authenticators, and also creates another authenticator to support.
>>
>> The migration to any new auth strategy would likely involve the need to simultaneously support an existing and new auth provider. I think the approach you describe is well described and should meet this need assuming users use a driver that supports it.
>>
>> In terms of the protocol, utilizing the capabilities of the existing OPTIONS, STARTUP and SUPPORTED messages to communicate what authenticators are supported/should be used is pretty clever as it shouldn't require a protocol version uprev, and hopefully wouldn't be too complicated for a driver to implement.
>>
>> Thanks,
>> Andy
>>
>> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <shepherd@amazon.com <ma...@amazon.com>> wrote:
>>> Erm ... and here's the CEP:
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>>>
>>> (Thanks for the heads up, Abe ...)
>>>
>>> -- Joel.
>>>
>>> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
>>> > Hello - We would like to propose CEP-50: Authentication Negotiation
>>> > for adoption by the community: <link> .
>>> >
>>> > This CEP proposes minor changes to the initial handshake protocol
>>> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
>>> > the node of the authenticators supported by the client, and changes in
>>> > the node's authentication-related areas to enable it to pick its
>>> > preferred authenticator for each client client connection. The CEP
>>> > explains why this approach is proposed, instead of implementing a
>>> > "negotiating authenticator".
>>> >
>>> > Authentication negotiation will make it easier and safer for
>>> > administrators to migrate clusters to stronger authentication
>>> > mechanisms (including switching on authentication for a cluster that
>>> > has been using "allow-all" authentication) without downtime, and to
>>> > support environments where different clients prefer different
>>> > authentication mechanisms (e.g., username and password for ad-hoc
>>> > cqlsh access, mutual TLS for programmatic access, etc.), without
>>> > having to pick a single "lowest common denominator" authenticator for
>>> > all. Additionally, the proposed changes are intended to be backwards
>>> > compatible for both clients and nodes.
>>> >
>>> > Thanks in advance for your time and feedback. Please keep the
>>> > discussion on this mailing list thread.
>>> >
>>> > Thanks! -- Joel.
>>> >
>>> >
>>> >
>>> >
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Joel Shepherd <sh...@amazon.com>.
Thanks Andy - My hope/expectation is this would significantly reduce the
amount of friction involved when either implementing or migrating to a
new authenticator. I suspect it will benefit more complex environments
too, when a single authenticator isn't ideal for all clients.
At this point, the stickiest bits I've run into have involved logic that
switches on "the" authenticator class, because of the hardcoded
dependency on a specific authn implementation, and working out how to
sanely extend the logic once it's possible for a single node to be using
different authenticators for different client sessions. Solvable but
will take some refactoring and might also generate debate about what the
right behaviors are in that scenario. An example is the RoleManager
which currently creates additional role attributes if the Password
Authenticator is in use ... which, now that you mention it, I wonder if
that's already broken for the
MutualTlsWithPasswordFallbackAuthenticator. Hmm.
Anyway, thanks for the feedback -- Joel.
On 7/3/2025 7:52 AM, Tolbert, Andy wrote:
>
> Hi Joel,
>
> +1 (nb), I think this is a really good idea and well fleshed our CEP!
>
> The capability to allow the server to support multiple authenticators
> would be very useful. CEP-34 added a
> 'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously
> supporting both mTLS and Password authentication, primarily as a means
> to introduce mTLS auth without breaking password auth and also a
> possible gradual migration to mTLS, but this only works for combining
> these two particular authenticators, and also creates another
> authenticator to support.
>
> The migration to any new auth strategy would likely involve the need
> to simultaneously support an existing and new auth provider. I think
> the approach you describe is well described and should meet this need
> assuming users use a driver that supports it.
>
> In terms of the protocol, utilizing the capabilities of the existing
> OPTIONS, STARTUP and SUPPORTED messages to communicate what
> authenticators are supported/should be used is pretty clever as it
> shouldn't require a protocol version uprev, and hopefully wouldn't be
> too complicated for a driver to implement.
>
> Thanks,
> Andy
>
> On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <sh...@amazon.com>
> wrote:
>
> Erm ... and here's the CEP:
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>
> (Thanks for the heads up, Abe ...)
>
> -- Joel.
>
> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
> > Hello - We would like to propose CEP-50: Authentication Negotiation
> > for adoption by the community: <link> .
> >
> > This CEP proposes minor changes to the initial handshake protocol
> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to
> inform
> > the node of the authenticators supported by the client, and
> changes in
> > the node's authentication-related areas to enable it to pick its
> > preferred authenticator for each client client connection. The CEP
> > explains why this approach is proposed, instead of implementing a
> > "negotiating authenticator".
> >
> > Authentication negotiation will make it easier and safer for
> > administrators to migrate clusters to stronger authentication
> > mechanisms (including switching on authentication for a cluster
> that
> > has been using "allow-all" authentication) without downtime, and to
> > support environments where different clients prefer different
> > authentication mechanisms (e.g., username and password for ad-hoc
> > cqlsh access, mutual TLS for programmatic access, etc.), without
> > having to pick a single "lowest common denominator"
> authenticator for
> > all. Additionally, the proposed changes are intended to be
> backwards
> > compatible for both clients and nodes.
> >
> > Thanks in advance for your time and feedback. Please keep the
> > discussion on this mailing list thread.
> >
> > Thanks! -- Joel.
> >
> >
> >
> >
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by "Tolbert, Andy" <x...@andrewtolbert.com>.
Hi Joel,
+1 (nb), I think this is a really good idea and well fleshed our CEP!
The capability to allow the server to support multiple authenticators would
be very useful. CEP-34 added a
'MutualTlsWithPasswordFallbackAuthenticator' for simultaneously supporting
both mTLS and Password authentication, primarily as a means to introduce
mTLS auth without breaking password auth and also a possible gradual
migration to mTLS, but this only works for combining these two particular
authenticators, and also creates another authenticator to support.
The migration to any new auth strategy would likely involve the need to
simultaneously support an existing and new auth provider. I think the
approach you describe is well described and should meet this need assuming
users use a driver that supports it.
In terms of the protocol, utilizing the capabilities of the existing
OPTIONS, STARTUP and SUPPORTED messages to communicate what authenticators
are supported/should be used is pretty clever as it shouldn't require a
protocol version uprev, and hopefully wouldn't be too complicated for a
driver to implement.
Thanks,
Andy
On Mon, Jun 30, 2025 at 11:44 AM Joel Shepherd <sh...@amazon.com> wrote:
> Erm ... and here's the CEP:
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
>
> (Thanks for the heads up, Abe ...)
>
> -- Joel.
>
> On 6/30/2025 9:37 AM, Joel Shepherd wrote:
> > Hello - We would like to propose CEP-50: Authentication Negotiation
> > for adoption by the community: <link> .
> >
> > This CEP proposes minor changes to the initial handshake protocol
> > (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
> > the node of the authenticators supported by the client, and changes in
> > the node's authentication-related areas to enable it to pick its
> > preferred authenticator for each client client connection. The CEP
> > explains why this approach is proposed, instead of implementing a
> > "negotiating authenticator".
> >
> > Authentication negotiation will make it easier and safer for
> > administrators to migrate clusters to stronger authentication
> > mechanisms (including switching on authentication for a cluster that
> > has been using "allow-all" authentication) without downtime, and to
> > support environments where different clients prefer different
> > authentication mechanisms (e.g., username and password for ad-hoc
> > cqlsh access, mutual TLS for programmatic access, etc.), without
> > having to pick a single "lowest common denominator" authenticator for
> > all. Additionally, the proposed changes are intended to be backwards
> > compatible for both clients and nodes.
> >
> > Thanks in advance for your time and feedback. Please keep the
> > discussion on this mailing list thread.
> >
> > Thanks! -- Joel.
> >
> >
> >
> >
>
Re: [DISCUSS] CEP-50: Authentication Negotiation
Posted by Joel Shepherd <sh...@amazon.com>.
Erm ... and here's the CEP:
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-50%3A+Authentication+Negotiation
(Thanks for the heads up, Abe ...)
-- Joel.
On 6/30/2025 9:37 AM, Joel Shepherd wrote:
> Hello - We would like to propose CEP-50: Authentication Negotiation
> for adoption by the community: <link> .
>
> This CEP proposes minor changes to the initial handshake protocol
> (OPTIONS, SUPPORTED and STARTUP messages) to enable a client to inform
> the node of the authenticators supported by the client, and changes in
> the node's authentication-related areas to enable it to pick its
> preferred authenticator for each client client connection. The CEP
> explains why this approach is proposed, instead of implementing a
> "negotiating authenticator".
>
> Authentication negotiation will make it easier and safer for
> administrators to migrate clusters to stronger authentication
> mechanisms (including switching on authentication for a cluster that
> has been using "allow-all" authentication) without downtime, and to
> support environments where different clients prefer different
> authentication mechanisms (e.g., username and password for ad-hoc
> cqlsh access, mutual TLS for programmatic access, etc.), without
> having to pick a single "lowest common denominator" authenticator for
> all. Additionally, the proposed changes are intended to be backwards
> compatible for both clients and nodes.
>
> Thanks in advance for your time and feedback. Please keep the
> discussion on this mailing list thread.
>
> Thanks! -- Joel.
>
>
>
>