You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jeff Jirsa <jj...@gmail.com> on 2025/08/01 06:45:25 UTC
Re: [DISCUSS] CEP-51: Support Include Semantics for cassandra.yaml
Hey folks
Helping nudge this along - any dissent? Anyone with concerns they’d like
addressed before we move towards a vote?
\- Jeff
> On Jul 26, 2025, at 12:35 PM, Ekaterina Dimitrova <e....@gmail.com>
> wrote:
>
>
>
>
> Hi,
>
>
>
>
> Just wanted to mention that we already have some config related to duplicate
> keys in cassandra.yaml. (That doesn’t mean we cannot extend/improve things
> further, of course)
>
> From our docs:
>
>
>
>
> [CASSANDRA-17379](https://issues.apache.org/jira/browse/CASSANDRA-17379) was
> opened to improve the user experience and deprecate the overloading. By
> default, we refuse starting Cassandra with a config containing both old and
> new config keys for the same parameter. Start Cassandra with
> `-Dcassandra.allow_new_old_config_keys=true` to override. For historical
> reasons duplicate config keys in `cassandra.yaml` are allowed by default,
> start Cassandra with `-Dcassandra.allow_duplicate_config_keys=false` to
> disallow this. Please note that `key_cache_save_period`,
> `row_cache_save_period`, `counter_cache_save_period` will be affected only
> by `-Dcassandra.allow_duplicate_config_keys`.
>
>
>
>
> There are people who actively use the overloading in their environments so
> backward compatibility is important.
>
>
>
>
> Best regards,
>
> Ekaterina
>
>
>
>
> On Wed, 23 Jul 2025 at 4:05, Johnny Miller
> <[johnny.p.miller@gmail.com](mailto:johnny.p.miller@gmail.com)> wrote:
>
>
>> That's a really good idea! I have updated the CEP to include this
duplicate_key_policy functionality and corresponding test scenarios.
>>
>>
>
>>
>> On Tue, 22 Jul 2025 at 16:35, Patrick McFadin
<[pmcfadin@gmail.com](mailto:pmcfadin@gmail.com)> wrote:
>
>>
>>> Another hit from the DevOps request backlog. I'm glad this has finally
turned into something formal. This will make CI/CD much easier. One thing I
hope this fosters more of is the sharing of configs. For example, "here are my
recommended storage settings for EBS."
>
> The CEP aborts on any duplicate key. For the people doing GitOps, there will
> be a need for layering a read-only baseline `cassandra.yaml` with
> environment-specific or secret files. This is exactly the way Argo CD/Helm
> handles value precedence. This is typical in cloud native environments. I
> propose a modification that allows an *opt-in* policy switch:
>
> duplicate_key_policy: { ABORT (default) | LAST_WINS | WARN }
>
> * ABORT – current behaviour, startup fails on duplicates.
> * WARN – duplicates allowed, last key wins, but every override is logged.
> * LAST_WINS – same as WARN, but without log
>>>
>>>
>
>>>
>>> On Mon, Jul 21, 2025 at 7:01 AM Josh McKenzie
<[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>
>>>
>>>> __
>>>>
>>>>> I suppose the only concern would be maintaining this version in
alignment with what's going into the main cassandra.yaml as part of the
regular development.
>>>>
>>>> Seems like it'd be relatively easy to script something that'll generate
modularized config files based on the reference cassandra.yaml by classifying
different parameters based on file grouping. Not to scope creep or add on to
the CEP or anything, just thinking out loud; as follow up work it could be
useful.
>
> From a technical perspective having a bi-directional sync that'd just dump
> things into a "overflow" file from monolithic -> modular, and tacking on at
> the end of cassandra.yaml under an overflow section for things not
> classified in the script from modular -> monolithic config shouldn't be too
> complex. If that proved stable, integrating that into the build process and
> even adding a checkstyle job target warning on non-classified configuration
> parameters could tighten the whole thing up and give us a minimally invasive
> way to support both through a transition.
>>>>
>>>>
>
>>>>
>>>> On Mon, Jul 21, 2025, at 9:41 AM, Johnny Miller wrote:
>>>>
>>>>> I have added the section "Reference Example Configuration" - will see
what the feedback on this is. I suppose the only concern would be maintaining
this version in alignment with what's going into the main cassandra.yaml as
part of the regular development.
>>>>>
>>>>>
>
>>>>>
>>>>> On Mon, 21 Jul 2025 at 15:24, Josh McKenzie
<[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:
>>>>>
>>>>>> __
>
>>>>>>
>>>>>> That sounds useful to me.
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> I'd like to see us move to "modularized by default"; our current config
being [2839 lines of
.yaml](https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml) is
a bad experience for both new and old users. Starting with examples of the new
paradigm and then refactoring config out over time for the default config is a
path forward I'd support.
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> On Mon, Jul 21, 2025, at 9:18 AM, Johnny Miller wrote:
>>>>>>
>>>>>>> One feature I was thinking of adding to the CEP was to have an example
yaml config setup using the includes with the config grouped logically so
people have a reference example in the conf? Would this be a good idea?
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> On Fri, 18 Jul 2025 at 19:57, Johnny Miller
<[johnny.p.miller@gmail.com](mailto:johnny.p.miller@gmail.com)> wrote:
>>>>>>>
>>>>>>>> Hello 👋
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> We would like to propose CEP-51: Support Include Semantics for
cassandra.yaml for adoption by the community:
>>>>>>>>
>>>>>>>>
<https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-51+Support+Include+Semantics+for+cassandra.yaml>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> This CEP proposes adding completely optional include directives to
Cassandra's configuration system, allowing users who need it to split their
cassandra.yaml into multiple files for better security, organization, and
deployment flexibility. No changes are made to the default cassandra.yaml, and
this feature is entirely opt-in.
>>>>>>>>
>>>>>>>> The proposed include directives (include, include_if_exists, and
include_dir) enable organizations to:
>>>>>>>>
>>>>>>>> * Apply the principle of least privilege by separating sensitive
security configurations into files with restricted permissions
>>>>>>>> * Better organize large configuration files by logical subsystems
>>>>>>>> * Simplify configuration management in environments where different
teams manage different aspects of the cluster
>>>>>>>> * Follow established patterns already present in PostgreSQL, MySQL,
Redis, NGINX, and other widely-used systems
>>>>>>>>
>>>>>>>>
>>>>>>>> Key design principles:
>>>>>>>>
>>>>>>>> * Zero impact on users who don't use the feature
>>>>>>>> * No recursive includes (only the main cassandra.yaml can contain
include directives)
>>>>>>>> * No duplicate configuration keys allowed (each setting must appear
in exactly one file)
>>>>>>>> * Clear error messages for troubleshooting
>>>>>>>>
>>>>>>>>
>>>>>>>> This enhancement addresses real operational challenges faced by
organisations with strict security requirements or complex deployment needs,
while maintaining complete backward compatibility and requiring no changes to
existing deployments.
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Thanks in advance for your time and feedback. Please keep the
discussion on this mailing list thread.
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Johnny
>>>>>>
>>>>>>
>
>>>>
>>>>
>
Re: [DISCUSS] CEP-51: Support Include Semantics for cassandra.yaml
Posted by Štefan Miklošovič <sm...@apache.org>.
More as a curiosity, I think people might be aware of this technique
already, there is a way to specify your custom ConfigurationLoader by
system property as Cassandra starts so they might. e.g. point to this
(1). (Look into DatabaseDescriptor.loadConfig for details).
What (1) does is that it reads the "cassandra.config" property which
contains partial yamls, delimited by ":".
Then it will use (2) which will act as if all these files were in one
file together but they are in fact in separate ones.
Just a way to show people that a monster yaml we have now can be
already split. This is a very simplistic approach but if somebody
wants to have something quick they can.
(1) https://github.com/instaclustr/cassandra-operator/blob/master/java/cassandra-4-k8s-addons/src/main/java/com/instaclustr/cassandra/k8s/ConcatenatedYamlConfigurationLoader.java
(2) https://github.com/instaclustr/cassandra-operator/blob/master/java/cassandra-4-k8s-addons/src/main/java/com/instaclustr/cassandra/k8s/ConcatenatedYamlConfigurationLoader.java#L112C44-L112C62
On Sat, Sep 13, 2025 at 12:01 AM Jordan West <jo...@gmail.com> wrote:
>
> Following up on this. Sounds like we can take this to a vote? I’m happy to sheperd if another committee or PMC member is not available
>
> On Fri, Aug 1, 2025 at 01:46 Jeff Jirsa <jj...@gmail.com> wrote:
>>
>> Hey folks
>>
>> Helping nudge this along - any dissent? Anyone with concerns they’d like addressed before we move towards a vote?
>>
>> - Jeff
>>
>> On Jul 26, 2025, at 12:35 PM, Ekaterina Dimitrova <e....@gmail.com> wrote:
>>
>>
>>
>> Hi,
>>
>> Just wanted to mention that we already have some config related to duplicate keys in cassandra.yaml. (That doesn’t mean we cannot extend/improve things further, of course)
>> From our docs:
>>
>> CASSANDRA-17379 was opened to improve the user experience and deprecate the overloading. By default, we refuse starting Cassandra with a config containing both old and new config keys for the same parameter. Start Cassandra with -Dcassandra.allow_new_old_config_keys=true to override. For historical reasons duplicate config keys in cassandra.yaml are allowed by default, start Cassandra with -Dcassandra.allow_duplicate_config_keys=false to disallow this. Please note that key_cache_save_period, row_cache_save_period, counter_cache_save_period will be affected only by -Dcassandra.allow_duplicate_config_keys.
>>
>> There are people who actively use the overloading in their environments so backward compatibility is important.
>>
>> Best regards,
>> Ekaterina
>>
>> On Wed, 23 Jul 2025 at 4:05, Johnny Miller <jo...@gmail.com> wrote:
>>>
>>> That's a really good idea! I have updated the CEP to include this duplicate_key_policy functionality and corresponding test scenarios.
>>>
>>> On Tue, 22 Jul 2025 at 16:35, Patrick McFadin <pm...@gmail.com> wrote:
>>>>
>>>> Another hit from the DevOps request backlog. I'm glad this has finally turned into something formal. This will make CI/CD much easier. One thing I hope this fosters more of is the sharing of configs. For example, "here are my recommended storage settings for EBS."
>>>>
>>>> The CEP aborts on any duplicate key. For the people doing GitOps, there will be a need for layering a read-only baseline `cassandra.yaml` with environment-specific or secret files. This is exactly the way Argo CD/Helm handles value precedence. This is typical in cloud native environments. I propose a modification that allows an *opt-in* policy switch:
>>>>
>>>> duplicate_key_policy: { ABORT (default) | LAST_WINS | WARN }
>>>>
>>>> * ABORT – current behaviour, startup fails on duplicates.
>>>> * WARN – duplicates allowed, last key wins, but every override is logged.
>>>> * LAST_WINS – same as WARN, but without log
>>>>
>>>> On Mon, Jul 21, 2025 at 7:01 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>>
>>>>> I suppose the only concern would be maintaining this version in alignment with what's going into the main cassandra.yaml as part of the regular development.
>>>>>
>>>>> Seems like it'd be relatively easy to script something that'll generate modularized config files based on the reference cassandra.yaml by classifying different parameters based on file grouping. Not to scope creep or add on to the CEP or anything, just thinking out loud; as follow up work it could be useful.
>>>>>
>>>>> From a technical perspective having a bi-directional sync that'd just dump things into a "overflow" file from monolithic -> modular, and tacking on at the end of cassandra.yaml under an overflow section for things not classified in the script from modular -> monolithic config shouldn't be too complex. If that proved stable, integrating that into the build process and even adding a checkstyle job target warning on non-classified configuration parameters could tighten the whole thing up and give us a minimally invasive way to support both through a transition.
>>>>>
>>>>> On Mon, Jul 21, 2025, at 9:41 AM, Johnny Miller wrote:
>>>>>
>>>>> I have added the section "Reference Example Configuration" - will see what the feedback on this is. I suppose the only concern would be maintaining this version in alignment with what's going into the main cassandra.yaml as part of the regular development.
>>>>>
>>>>> On Mon, 21 Jul 2025 at 15:24, Josh McKenzie <jm...@apache.org> wrote:
>>>>>
>>>>>
>>>>> That sounds useful to me.
>>>>>
>>>>> I'd like to see us move to "modularized by default"; our current config being 2839 lines of .yaml is a bad experience for both new and old users. Starting with examples of the new paradigm and then refactoring config out over time for the default config is a path forward I'd support.
>>>>>
>>>>> On Mon, Jul 21, 2025, at 9:18 AM, Johnny Miller wrote:
>>>>>
>>>>> One feature I was thinking of adding to the CEP was to have an example yaml config setup using the includes with the config grouped logically so people have a reference example in the conf? Would this be a good idea?
>>>>>
>>>>> On Fri, 18 Jul 2025 at 19:57, Johnny Miller <jo...@gmail.com> wrote:
>>>>>
>>>>> Hello 👋
>>>>>
>>>>> We would like to propose CEP-51: Support Include Semantics for cassandra.yaml for adoption by the community:
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-51+Support+Include+Semantics+for+cassandra.yaml
>>>>>
>>>>> This CEP proposes adding completely optional include directives to Cassandra's configuration system, allowing users who need it to split their cassandra.yaml into multiple files for better security, organization, and deployment flexibility. No changes are made to the default cassandra.yaml, and this feature is entirely opt-in.
>>>>> The proposed include directives (include, include_if_exists, and include_dir) enable organizations to:
>>>>>
>>>>> Apply the principle of least privilege by separating sensitive security configurations into files with restricted permissions
>>>>> Better organize large configuration files by logical subsystems
>>>>> Simplify configuration management in environments where different teams manage different aspects of the cluster
>>>>> Follow established patterns already present in PostgreSQL, MySQL, Redis, NGINX, and other widely-used systems
>>>>>
>>>>> Key design principles:
>>>>>
>>>>> Zero impact on users who don't use the feature
>>>>> No recursive includes (only the main cassandra.yaml can contain include directives)
>>>>> No duplicate configuration keys allowed (each setting must appear in exactly one file)
>>>>> Clear error messages for troubleshooting
>>>>>
>>>>> This enhancement addresses real operational challenges faced by organisations with strict security requirements or complex deployment needs, while maintaining complete backward compatibility and requiring no changes to existing deployments.
>>>>>
>>>>> Thanks in advance for your time and feedback. Please keep the discussion on this mailing list thread.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Johnny
>>>>>
>>>>>
>>>>>
Re: [DISCUSS] CEP-51: Support Include Semantics for cassandra.yaml
Posted by Jordan West <jo...@gmail.com>.
Following up on this. Sounds like we can take this to a vote? I’m happy to
sheperd if another committee or PMC member is not available
On Fri, Aug 1, 2025 at 01:46 Jeff Jirsa <jj...@gmail.com> wrote:
> Hey folks
>
> Helping nudge this along - any dissent? Anyone with concerns they’d like
> addressed before we move towards a vote?
>
> - Jeff
>
> On Jul 26, 2025, at 12:35 PM, Ekaterina Dimitrova <e....@gmail.com>
> wrote:
>
>
>
> Hi,
>
> Just wanted to mention that we already have some config related to
> duplicate keys in cassandra.yaml. (That doesn’t mean we cannot
> extend/improve things further, of course)
> From our docs:
>
> CASSANDRA-17379 <https://issues.apache.org/jira/browse/CASSANDRA-17379> was
> opened to improve the user experience and deprecate the overloading. By
> default, we refuse starting Cassandra with a config containing both old and
> new config keys for the same parameter. Start Cassandra with
> -Dcassandra.allow_new_old_config_keys=true to override. For historical
> reasons duplicate config keys in cassandra.yaml are allowed by default,
> start Cassandra with -Dcassandra.allow_duplicate_config_keys=false to
> disallow this. Please note that key_cache_save_period,
> row_cache_save_period, counter_cache_save_period will be affected only by
> -Dcassandra.allow_duplicate_config_keys.
>
> There are people who actively use the overloading in their environments so
> backward compatibility is important.
>
> Best regards,
> Ekaterina
>
> On Wed, 23 Jul 2025 at 4:05, Johnny Miller <jo...@gmail.com>
> wrote:
>
>> That's a really good idea! I have updated the CEP to include this
>> duplicate_key_policy functionality and corresponding test scenarios.
>>
>> On Tue, 22 Jul 2025 at 16:35, Patrick McFadin <pm...@gmail.com> wrote:
>>
>>> Another hit from the DevOps request backlog. I'm glad this has
>>> finally turned into something formal. This will make CI/CD much easier. One
>>> thing I hope this fosters more of is the sharing of configs. For example,
>>> "here are my recommended storage settings for EBS."
>>>
>>> The CEP aborts on any duplicate key. For the people doing GitOps, there
>>> will be a need for layering a read-only baseline `cassandra.yaml` with
>>> environment-specific or secret files. This is exactly the way Argo CD/Helm
>>> handles value precedence. This is typical in cloud native environments. I
>>> propose a modification that allows an *opt-in* policy switch:
>>>
>>> duplicate_key_policy: { ABORT (default) | LAST_WINS | WARN }
>>>
>>> * ABORT – current behaviour, startup fails on duplicates.
>>> * WARN – duplicates allowed, last key wins, but every override is
>>> logged.
>>> * LAST_WINS – same as WARN, but without log
>>>
>>> On Mon, Jul 21, 2025 at 7:01 AM Josh McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>>> I suppose the only concern would be maintaining this version in
>>>> alignment with what's going into the main cassandra.yaml as part of the
>>>> regular development.
>>>>
>>>> Seems like it'd be relatively easy to script something that'll generate
>>>> modularized config files based on the reference cassandra.yaml by
>>>> classifying different parameters based on file grouping. Not to scope creep
>>>> or add on to the CEP or anything, just thinking out loud; as follow up work
>>>> it could be useful.
>>>>
>>>> From a technical perspective having a bi-directional sync that'd just
>>>> dump things into a "overflow" file from monolithic -> modular, and tacking
>>>> on at the end of cassandra.yaml under an overflow section for things not
>>>> classified in the script from modular -> monolithic config shouldn't be too
>>>> complex. If that proved stable, integrating that into the build process and
>>>> even adding a checkstyle job target warning on non-classified configuration
>>>> parameters could tighten the whole thing up and give us a minimally
>>>> invasive way to support both through a transition.
>>>>
>>>> On Mon, Jul 21, 2025, at 9:41 AM, Johnny Miller wrote:
>>>>
>>>> I have added the section "Reference Example Configuration" - will see
>>>> what the feedback on this is. I suppose the only concern would be
>>>> maintaining this version in alignment with what's going into the main
>>>> cassandra.yaml as part of the regular development.
>>>>
>>>> On Mon, 21 Jul 2025 at 15:24, Josh McKenzie <jm...@apache.org>
>>>> wrote:
>>>>
>>>>
>>>> That sounds useful to me.
>>>>
>>>> I'd like to see us move to "modularized by default"; our current config
>>>> being 2839 lines of .yaml
>>>> <https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml>
>>>> is a bad experience for both new and old users. Starting with examples of
>>>> the new paradigm and then refactoring config out over time for the default
>>>> config is a path forward I'd support.
>>>>
>>>> On Mon, Jul 21, 2025, at 9:18 AM, Johnny Miller wrote:
>>>>
>>>> One feature I was thinking of adding to the CEP was to have an example
>>>> yaml config setup using the includes with the config grouped logically so
>>>> people have a reference example in the conf? Would this be a good idea?
>>>>
>>>> On Fri, 18 Jul 2025 at 19:57, Johnny Miller <jo...@gmail.com>
>>>> wrote:
>>>>
>>>> Hello 👋
>>>>
>>>> We would like to propose CEP-51: Support Include Semantics for
>>>> cassandra.yaml for adoption by the community:
>>>>
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-51+Support+Include+Semantics+for+cassandra.yaml
>>>>
>>>> This CEP proposes adding completely optional include directives to
>>>> Cassandra's configuration system, allowing users who need it to split their
>>>> cassandra.yaml into multiple files for better security, organization, and
>>>> deployment flexibility. No changes are made to the default cassandra.yaml,
>>>> and this feature is entirely opt-in.
>>>> The proposed include directives (include, include_if_exists, and
>>>> include_dir) enable organizations to:
>>>>
>>>> - Apply the principle of least privilege by separating sensitive
>>>> security configurations into files with restricted permissions
>>>> - Better organize large configuration files by logical subsystems
>>>> - Simplify configuration management in environments where different
>>>> teams manage different aspects of the cluster
>>>> - Follow established patterns already present in PostgreSQL, MySQL,
>>>> Redis, NGINX, and other widely-used systems
>>>>
>>>> Key design principles:
>>>>
>>>> - Zero impact on users who don't use the feature
>>>> - No recursive includes (only the main cassandra.yaml can contain
>>>> include directives)
>>>> - No duplicate configuration keys allowed (each setting must appear
>>>> in exactly one file)
>>>> - Clear error messages for troubleshooting
>>>>
>>>> This enhancement addresses real operational challenges faced by
>>>> organisations with strict security requirements or complex deployment
>>>> needs, while maintaining complete backward compatibility and requiring no
>>>> changes to existing deployments.
>>>>
>>>> Thanks in advance for your time and feedback. Please keep the
>>>> discussion on this mailing list thread.
>>>>
>>>> Regards,
>>>>
>>>> Johnny
>>>>
>>>>
>>>>
>>>>