You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Gaurav Agarwal <ga...@gmail.com> on 2024/02/12 19:44:31 UTC

[Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

Dear Cassandra Community,

I'm excited to share a proposal for a new feature that I believe would
significantly enhance the platform's security and operational flexibility: *a
flexible authentication mechanism implemented through a feature flag *.

Currently, enforcing authentication in Cassandra requires a disruptive,
full-cluster restart, posing significant risks in live environments. My
proposal, the *auth_enforcement_flag*, addresses this challenge by offering
three modes:

*Hard:* Enforces strict authentication with detailed logging.
*Soft:* Monitors connection attempts (valid and invalid) without enforcing
authentication.
*None:* Maintains the current Cassandra behavior.

This flag enables:
*Minimized downtime: *Seamless authentication rollout without service
interruptions.
*Enhanced security:* Detailed logs for improved threat detection and
troubleshooting.
*Gradual adoption:* Phased implementation with real-world feedback
integration.

I believe this feature provides substantial benefits for both users and
administrators. Please see the detailed proposal here: Introducing flexible
authentication mechanism
<https://docs.google.com/document/d/1w649JAJdhVNQwQ9btXaaUopXlGjia6Sfgv281U9U7HY>

I warmly invite the community to review this proposal and share your
valuable feedback. I'm eager to discuss its potential impact and
collaborate on making Cassandra even better.

Thank you for your time and consideration.

Sincerely,
Gaurav Agarwal
Software Engineer at Uber

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

Posted by Dinesh Joshi <dj...@apache.org>.
Hi Gaurav,

Thank you for the document. I read through it and wasn't entirely clear
about the problem you're trying to solve.

If you're talking about enabling authentication for the very first time on
a cluster which does not have any authentication then there are different
ways of handling it.

> Minimized downtime: Seamless authentication rollout without service
interruptions.

I am not sure why would the app or the cluster take any downtime as rolling
restarts in Cassandra are safe and do not cause downtime. If Cassandra
doesn't require authentication, specifying credentials in the driver should
not cause issues. So users are safe to transition from an unauthenticated
to an authenticated cluster.

> Enhanced security: Detailed logs for improved threat detection and
troubleshooting.

Allowing client connections to proceed with invalid credentials poses a
serious security risk. Cassandra 4.0+ has Audit Logging which IIRC does log
invalid login attempts (if not, it would be trivial to add them).

CASSANDRA-11471 allows for improved protocol negotiation which would allow
you to specify a list of acceptable authentication mechanisms but I don't
think this is being worked on at the moment.

That said, right now you can certainly implement your own Authenticator
which implements your custom logic without any changes to Cassandra itself.

Dinesh


On Mon, Feb 12, 2024 at 11:45 AM Gaurav Agarwal <ga...@gmail.com>
wrote:

> Dear Cassandra Community,
>
> I'm excited to share a proposal for a new feature that I believe would
> significantly enhance the platform's security and operational flexibility: *a
> flexible authentication mechanism implemented through a feature flag *.
>
> Currently, enforcing authentication in Cassandra requires a disruptive,
> full-cluster restart, posing significant risks in live environments. My
> proposal, the *auth_enforcement_flag*, addresses this challenge by
> offering three modes:
>
> *Hard:* Enforces strict authentication with detailed logging.
> *Soft:* Monitors connection attempts (valid and invalid) without
> enforcing authentication.
> *None:* Maintains the current Cassandra behavior.
>
> This flag enables:
> *Minimized downtime: *Seamless authentication rollout without service
> interruptions.
> *Enhanced security:* Detailed logs for improved threat detection and
> troubleshooting.
> *Gradual adoption:* Phased implementation with real-world feedback
> integration.
>
> I believe this feature provides substantial benefits for both users and
> administrators. Please see the detailed proposal here: Introducing
> flexible authentication mechanism
> <https://docs.google.com/document/d/1w649JAJdhVNQwQ9btXaaUopXlGjia6Sfgv281U9U7HY>
>
> I warmly invite the community to review this proposal and share your
> valuable feedback. I'm eager to discuss its potential impact and
> collaborate on making Cassandra even better.
>
> Thank you for your time and consideration.
>
> Sincerely,
> Gaurav Agarwal
> Software Engineer at Uber
>

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

Posted by Jacek Lewandowski <le...@gmail.com>.
Hi,

I think what Gaurav means is what we know at DataStax as transitional
authenticator, which temporarily allows for partially enabled
authentication - when the system allows the clients to authenticate but
does not enforce it.

All in all, that should be included in CEP-31 - also CEP-31 aims to let the
administrators enable/disable and reconfigure authentication without a
restart so we could discuss whether such transitional mode would be needed
at all in that case.

Thanks,
- - -- --- ----- -------- -------------
Jacek Lewandowski


wt., 13 lut 2024 o 07:04 Jeff Jirsa <jj...@gmail.com> napisał(a):

> Auth is one of those things that needs to be a bit more concrete
>
> In the scenario you describe, you already have an option to deploy the
> auth in piece partially during the rollout (pause halfway through) in the
> cluster and look for asymmetric connections, and the option to drop in a
> new Authenticator jar in the class path that does the flexible auth you
> describe
>
> I fear that the extra flexibility this allows for 1% of operations exposes
> people to long term problems
>
> Have you considered just implementing the feature flag you describe using
> the existing plugin infrastructure ?
>
> On Feb 12, 2024, at 9:47 PM, Gaurav Agarwal <ga...@gmail.com>
> wrote:
>
> 
> Dear Dinesh and Abe,
>
> Thank you for reviewing the document on enabling Cassandra authentication.
> I apologize that I didn't initially include the following failure scenarios
> where this feature could be particularly beneficial (I've included them
> now):
>
> *Below are the failure scenarios:*
>
>    - Incorrect credentials: If a client accidentally uses the wrong
>    username/password combination during the rollout, While restarting the
>    server to enable authentication, it will refuse connections with incorrect
>    credentials. This can temporarily interrupt the service until correct
>    credentials are sent.
>    - Missed service auth updates: In a large-scale system, a service "X"
>    might miss the credential update during rollout. After some server nodes
>    restart, service "X" might finally realize it needs correct credentials,
>    but it's too late. Nodes are already expecting authorized requests, and
>    this mismatch causes "X" to stop working on auth enabled and restarted
>    nodes.
>    - Infrequent traffic:  Suppose one of the services only interacts with
>    the server once a week. Suppose it starts sending requests with incorrect
>    credentials after authentication is enabled. Since the entire cluster is
>    now running on authentication, the service's outdated credentials cause it
>    to be denied access, resulting in a service-wide outage.
>
>
> The overall aim of the proposed feature flag would allow clients to
> connect momentarily without authentication during the rollout, mitigating
> these risks and ensuring a smoother transition.
>
> Thanks in advance for your continued review of the proposal.
>
>
>
> On Mon, Feb 12, 2024 at 2:24 PM Abe Ratnofsky <ab...@aber.io> wrote:
>
>> Hey Guarav,
>>
>> Thanks for your proposal.
>>
>> > disruptive, full-cluster restart, posing significant risks in live
>> environments
>>
>> For configuration that isn't hot-reloadable, like providing a new
>> IAuthenticator implementation, a rolling restart is required. But rolling
>> restarts are zero-downtime and safe in production, as long as you pace them
>> accordingly.
>>
>> In general, changing authenticators is a risky thing because it requires
>> coordination with clients. To mitigate this risk and support clients while
>> they transition between authenticators, I like the approach taken by
>> MutualTlsWithPasswordFallbackAuthenticator:
>>
>> https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34
>>
>> If client certificates are available, then use those, otherwise use the
>> existing PasswordAuthenticator that clients are already using. The existing
>> IAuthenticator interface supports this transitional behavior well.
>>
>> Your proposal to include a new configuration for auth_enforcement_flag
>> doesn't clearly cover how to transition from one authenticator to another.
>> It says:
>>
>> > Soft: Operates in a monitoring mode without enforcing authentication
>>
>> Most users use authentication today, so auth_enforcement_flag=Soft would
>> allow unauthenticated clients to connect to the database.
>>
>> --
>> Abe
>>
>> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal <ga...@gmail.com>
>> wrote:
>>
>> Dear Cassandra Community,
>>
>> I'm excited to share a proposal for a new feature that I believe would
>> significantly enhance the platform's security and operational flexibility: *a
>> flexible authentication mechanism implemented through a feature flag *.
>>
>> Currently, enforcing authentication in Cassandra requires a disruptive,
>> full-cluster restart, posing significant risks in live environments. My
>> proposal, the *auth_enforcement_flag*, addresses this challenge by
>> offering three modes:
>>
>> *Hard:* Enforces strict authentication with detailed logging.
>> *Soft:* Monitors connection attempts (valid and invalid) without
>> enforcing authentication.
>> *None:* Maintains the current Cassandra behavior.
>>
>> This flag enables:
>> *Minimized downtime: *Seamless authentication rollout without service
>> interruptions.
>> *Enhanced security:* Detailed logs for improved threat detection and
>> troubleshooting.
>> *Gradual adoption:* Phased implementation with real-world feedback
>> integration.
>>
>> I believe this feature provides substantial benefits for both users and
>> administrators. Please see the detailed proposal here: Introducing
>> flexible authentication mechanism
>> <https://docs.google.com/document/d/1w649JAJdhVNQwQ9btXaaUopXlGjia6Sfgv281U9U7HY>
>>
>> I warmly invite the community to review this proposal and share your
>> valuable feedback. I'm eager to discuss its potential impact and
>> collaborate on making Cassandra even better.
>>
>> Thank you for your time and consideration.
>>
>> Sincerely,
>> Gaurav Agarwal
>> Software Engineer at Uber
>>
>>
>>

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

Posted by Jeff Jirsa <jj...@gmail.com>.
Auth is one of those things that needs to be a bit more concrete

  

In the scenario you describe, you already have an option to deploy the auth in
piece partially during the rollout (pause halfway through) in the cluster and
look for asymmetric connections, and the option to drop in a new Authenticator
jar in the class path that does the flexible auth you describe

  

I fear that the extra flexibility this allows for 1% of operations exposes
people to long term problems

  

Have you considered just implementing the feature flag you describe using the
existing plugin infrastructure ?  

  

> On Feb 12, 2024, at 9:47 PM, Gaurav Agarwal <ga...@gmail.com> wrote:  
>  
>

> 
>
> Dear Dinesh and Abe,  
>  
> Thank you for reviewing the document on enabling Cassandra authentication. I
> apologize that I didn't initially include the following failure scenarios
> where this feature could be particularly beneficial (I've included them
> now):  
>
>
>  
>
>
>  **Below are the failure scenarios:**  
>
>
>   * Incorrect credentials: If a client accidentally uses the wrong
> username/password combination during the rollout, While restarting the
> server to enable authentication, it will refuse connections with incorrect
> credentials. This can temporarily interrupt the service until correct
> credentials are sent.
>   * Missed service auth updates: In a large-scale system, a service "X"
> might miss the credential update during rollout. After some server nodes
> restart, service "X" might finally realize it needs correct credentials, but
> it's too late. Nodes are already expecting authorized requests, and this
> mismatch causes "X" to stop working on auth enabled and restarted nodes.
>   * Infrequent traffic:  Suppose one of the services only interacts with the
> server once a week. Suppose it starts sending requests with incorrect
> credentials after authentication is enabled. Since the entire cluster is now
> running on authentication, the service's outdated credentials cause it to be
> denied access, resulting in a service-wide outage.  
>
>

>
>  
> The overall aim of the proposed feature flag would allow clients to connect
> momentarily without authentication during the rollout, mitigating these
> risks and ensuring a smoother transition.
>
>  
>
>
> Thanks in advance for your continued review of the proposal.  
>
>
>  
>
>
>  
>
>  
>
>
> On Mon, Feb 12, 2024 at 2:24 PM Abe Ratnofsky
> <[abe@aber.io](mailto:abe@aber.io)> wrote:  
>
>

>> Hey Guarav,

>>

>>  
>
>>

>> Thanks for your proposal.

>>

>>  
>
>>

>> > disruptive, full-cluster restart, posing significant risks in live
environments

>>

>>  
>
>>

>> For configuration that isn't hot-reloadable, like providing a new
IAuthenticator implementation, a rolling restart is required. But rolling
restarts are zero-downtime and safe in production, as long as you pace them
accordingly.

>>

>>  
>
>>

>> In general, changing authenticators is a risky thing because it requires
coordination with clients. To mitigate this risk and support clients while
they transition between authenticators, I like the approach taken by
MutualTlsWithPasswordFallbackAuthenticator:

>>

>>
<https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34>

>>

>>  
>
>>

>> If client certificates are available, then use those, otherwise use the
existing PasswordAuthenticator that clients are already using. The existing
IAuthenticator interface supports this transitional behavior well.

>>

>>  
>
>>

>> Your proposal to include a new configuration for auth_enforcement_flag
doesn't clearly cover how to transition from one authenticator to another. It
says:

>>

>>  
>
>>

>> > Soft: Operates in a monitoring mode without enforcing authentication

>>

>>  
>
>>

>> Most users use authentication today, so auth_enforcement_flag=Soft would
allow unauthenticated clients to connect to the database.

>>

>>  
>
>>

>> \--

>>

>> Abe

>>

>>  
>
>>

>>> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal
<[gaurav.iit219@gmail.com](mailto:gaurav.iit219@gmail.com)> wrote:

>>>

>>>  
>
>>>

>>> Dear Cassandra Community,  
>  
> I'm excited to share a proposal for a new feature that I believe would
> significantly enhance the platform's security and operational flexibility:
> _a flexible authentication mechanism implemented through a feature flag_.  
>  
> Currently, enforcing authentication in Cassandra requires a disruptive,
> full-cluster restart, posing significant risks in live environments. My
> proposal, the _auth_enforcement_flag_ , addresses this challenge by offering
> three modes:  
>  
>  _Hard:_ Enforces strict authentication with detailed logging.  
>  _Soft:_ Monitors connection attempts (valid and invalid) without enforcing
> authentication.  
>  _None:_ Maintains the current Cassandra behavior.  
>  
> This flag enables:  
>  **Minimized downtime:** Seamless authentication rollout without service
> interruptions.  
>  **Enhanced security:** Detailed logs for improved threat detection and
> troubleshooting.  
>  **Gradual adoption:** Phased implementation with real-world feedback
> integration.  
>  
> I believe this feature provides substantial benefits for both users and
> administrators. Please see the detailed proposal here: [Introducing flexible
> authentication
> mechanism](https://docs.google.com/document/d/1w649JAJdhVNQwQ9btXaaUopXlGjia6Sfgv281U9U7HY)  
>  
> I warmly invite the community to review this proposal and share your
> valuable feedback. I'm eager to discuss its potential impact and collaborate
> on making Cassandra even better.  
>  
> Thank you for your time and consideration.  
>  
> Sincerely,  
> Gaurav Agarwal
>>>

>>> Software Engineer at Uber

>>

>>  
>


Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

Posted by Gaurav Agarwal <ga...@gmail.com>.
Dear Dinesh and Abe,

Thank you for reviewing the document on enabling Cassandra authentication.
I apologize that I didn't initially include the following failure scenarios
where this feature could be particularly beneficial (I've included them
now):

*Below are the failure scenarios:*

   - Incorrect credentials: If a client accidentally uses the wrong
   username/password combination during the rollout, While restarting the
   server to enable authentication, it will refuse connections with incorrect
   credentials. This can temporarily interrupt the service until correct
   credentials are sent.
   - Missed service auth updates: In a large-scale system, a service "X"
   might miss the credential update during rollout. After some server nodes
   restart, service "X" might finally realize it needs correct credentials,
   but it's too late. Nodes are already expecting authorized requests, and
   this mismatch causes "X" to stop working on auth enabled and restarted
   nodes.
   - Infrequent traffic:  Suppose one of the services only interacts with
   the server once a week. Suppose it starts sending requests with incorrect
   credentials after authentication is enabled. Since the entire cluster is
   now running on authentication, the service's outdated credentials cause it
   to be denied access, resulting in a service-wide outage.


The overall aim of the proposed feature flag would allow clients to connect
momentarily without authentication during the rollout, mitigating these
risks and ensuring a smoother transition.

Thanks in advance for your continued review of the proposal.



On Mon, Feb 12, 2024 at 2:24 PM Abe Ratnofsky <ab...@aber.io> wrote:

> Hey Guarav,
>
> Thanks for your proposal.
>
> > disruptive, full-cluster restart, posing significant risks in live
> environments
>
> For configuration that isn't hot-reloadable, like providing a new
> IAuthenticator implementation, a rolling restart is required. But rolling
> restarts are zero-downtime and safe in production, as long as you pace them
> accordingly.
>
> In general, changing authenticators is a risky thing because it requires
> coordination with clients. To mitigate this risk and support clients while
> they transition between authenticators, I like the approach taken by
> MutualTlsWithPasswordFallbackAuthenticator:
>
> https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34
>
> If client certificates are available, then use those, otherwise use the
> existing PasswordAuthenticator that clients are already using. The existing
> IAuthenticator interface supports this transitional behavior well.
>
> Your proposal to include a new configuration for auth_enforcement_flag
> doesn't clearly cover how to transition from one authenticator to another.
> It says:
>
> > Soft: Operates in a monitoring mode without enforcing authentication
>
> Most users use authentication today, so auth_enforcement_flag=Soft would
> allow unauthenticated clients to connect to the database.
>
> --
> Abe
>
> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal <ga...@gmail.com>
> wrote:
>
> Dear Cassandra Community,
>
> I'm excited to share a proposal for a new feature that I believe would
> significantly enhance the platform's security and operational flexibility: *a
> flexible authentication mechanism implemented through a feature flag *.
>
> Currently, enforcing authentication in Cassandra requires a disruptive,
> full-cluster restart, posing significant risks in live environments. My
> proposal, the *auth_enforcement_flag*, addresses this challenge by
> offering three modes:
>
> *Hard:* Enforces strict authentication with detailed logging.
> *Soft:* Monitors connection attempts (valid and invalid) without
> enforcing authentication.
> *None:* Maintains the current Cassandra behavior.
>
> This flag enables:
> *Minimized downtime: *Seamless authentication rollout without service
> interruptions.
> *Enhanced security:* Detailed logs for improved threat detection and
> troubleshooting.
> *Gradual adoption:* Phased implementation with real-world feedback
> integration.
>
> I believe this feature provides substantial benefits for both users and
> administrators. Please see the detailed proposal here: Introducing
> flexible authentication mechanism
> <https://docs.google.com/document/d/1w649JAJdhVNQwQ9btXaaUopXlGjia6Sfgv281U9U7HY>
>
> I warmly invite the community to review this proposal and share your
> valuable feedback. I'm eager to discuss its potential impact and
> collaborate on making Cassandra even better.
>
> Thank you for your time and consideration.
>
> Sincerely,
> Gaurav Agarwal
> Software Engineer at Uber
>
>
>

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

Posted by Abe Ratnofsky <ab...@aber.io>.
Hey Guarav,

Thanks for your proposal.

> disruptive, full-cluster restart, posing significant risks in live environments

For configuration that isn't hot-reloadable, like providing a new IAuthenticator implementation, a rolling restart is required. But rolling restarts are zero-downtime and safe in production, as long as you pace them accordingly.

In general, changing authenticators is a risky thing because it requires coordination with clients. To mitigate this risk and support clients while they transition between authenticators, I like the approach taken by MutualTlsWithPasswordFallbackAuthenticator:
https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34

If client certificates are available, then use those, otherwise use the existing PasswordAuthenticator that clients are already using. The existing IAuthenticator interface supports this transitional behavior well.

Your proposal to include a new configuration for auth_enforcement_flag doesn't clearly cover how to transition from one authenticator to another. It says:

> Soft: Operates in a monitoring mode without enforcing authentication

Most users use authentication today, so auth_enforcement_flag=Soft would allow unauthenticated clients to connect to the database.

--
Abe

> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal <ga...@gmail.com> wrote:
> 
> Dear Cassandra Community,
> 
> I'm excited to share a proposal for a new feature that I believe would significantly enhance the platform's security and operational flexibility: a flexible authentication mechanism implemented through a feature flag .
> 
> Currently, enforcing authentication in Cassandra requires a disruptive, full-cluster restart, posing significant risks in live environments. My proposal, the auth_enforcement_flag, addresses this challenge by offering three modes:
> 
> Hard: Enforces strict authentication with detailed logging.
> Soft: Monitors connection attempts (valid and invalid) without enforcing authentication.
> None: Maintains the current Cassandra behavior.
> 
> This flag enables:
> Minimized downtime: Seamless authentication rollout without service interruptions.
> Enhanced security: Detailed logs for improved threat detection and troubleshooting.
> Gradual adoption: Phased implementation with real-world feedback integration.
> 
> I believe this feature provides substantial benefits for both users and administrators. Please see the detailed proposal here: Introducing flexible authentication mechanism <https://docs.google.com/document/d/1w649JAJdhVNQwQ9btXaaUopXlGjia6Sfgv281U9U7HY>
> 
> I warmly invite the community to review this proposal and share your valuable feedback. I'm eager to discuss its potential impact and collaborate on making Cassandra even better.
> 
> Thank you for your time and consideration.
> 
> Sincerely,
> Gaurav Agarwal
> Software Engineer at Uber