You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by David Capwell <dc...@apple.com.INVALID> on 2021/12/03 18:33:41 UTC
Re: [DISCUSS] Nested YAML configs for new features
Thanks everyone for the feedback! If I am reading this properly I am seeing the following
* Good with nested configs
* Good with YAML layer supporting flat structure (possible foo.bar.baz for the path foo: {bar: {baz: 42}}), how this relates with Settings table should be resolved, but there is a open ticket for this (enhance our YAML CASSANDRA-17166, and support updates to Settings vtable CASSANDRA-15254)
* Where/How we group is an open question, maybe we move this to a JIRA as follow up work to CASSANDRA-15234?
> We’re also mixing terminology already, with limits/thresholds and fail/abort.
Spoke with Ekaterina about this, and not solved in 15234; lets move this to a follow up JIRA for 15234?
> On Nov 30, 2021, at 6:08 AM, Ekaterina Dimitrova <e....@gmail.com> wrote:
>
> Thank you for confirming as I misread your email at first :-)
> I had a chat with David last week and I don’t think his plan is reworking
> of 15234 but incremental improvements on top of it.
> Regarding config, after spending time cleaning around and looking more into
> detail my only appeal is:
> - Centralized management and not 5 places to change things when you add new
> config so we are less error-prone
> - Documenting things for people who add new config or for our users (I
> promised and I will do it for 15234 but it will be good to continue doing
> it with any further changes down the road)
> - be careful with breaking changes
>
> Thank you
> Ekaterina
>
> On Tue, 30 Nov 2021 at 8:59, benedict@apache.org <be...@apache.org>
> wrote:
>
>> I mean that it has been waiting for months, is ready to go, and I don’t
>> want to hold you up any longer.
>>
>> From: Ekaterina Dimitrova <e....@gmail.com>
>> Date: Tuesday, 30 November 2021 at 13:44
>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
>> Subject: Re: [DISCUSS] Nested YAML configs for new features
>> “
>> IMO 15234 has sailed – it’s been held up for a long time, and was brought
>> to this list for discussion with no engagement. Ekaterina is long overdue
>> being able to commit her work. “
>>
>>
>> Sailed? I submitted the patch a week ago for review. Not sure how to
>> understand this statement. Can elaborate, please?
>>
>> On Tue, 30 Nov 2021 at 8:09, benedict@apache.org <be...@apache.org>
>> wrote:
>>
>>> The problem with scoping this to “features” is that we end up with at
>> best
>>> local coherence. The config file as a whole will end up just as
>> incoherent
>>> through its design evolution as it has historically.
>>>
>>> If you take a look at my proposed layout for the overall config, there is
>>> a “limits” section that specifies thresholds for reporting warnings and
>>> errors for various scenario. In this case, we probably don’t also want
>>> per-feature limits? We’re also mixing terminology already, with
>>> limits/thresholds and fail/abort.
>>>
>>> It’s a lot of work to come up with a coherent and intuitive config
>> layout.
>>> We probably want to at least create some documentation in-tree
>> stipulating
>>> terminology with respect to plurals, verbs/nouns, and specific terms
>>> (period, abort, limit, datacenter vs dc, etc), but ideally we would have
>> a
>>> common end goal for the config file.
>>>
>>>> leave non-features to CASSANDRA-15234
>>>
>>> IMO 15234 has sailed – it’s been held up for a long time, and was brought
>>> to this list for discussion with no engagement. Ekaterina is long overdue
>>> being able to commit her work.
>>>
>>>
>>> From: David Capwell <dc...@apple.com.INVALID>
>>> Date: Monday, 29 November 2021 at 23:44
>>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
>>> Subject: Re: [DISCUSS] Nested YAML configs for new features
>>>> but I would hate to repeat the mistakes of our past by evolving the
>>> config in a new direction without any coherent overarching design.
>>>
>>> At the start I asked to keep the thread local to new features, but to
>> more
>>> flesh out an “overarching design” maybe we should increase the “desired”
>>> scope to be “feature” (and leave non-features to CASSANDRA-15234 -
>>> Standardise config and JVM parameters)? Aka, do we think the following
>> is
>>> more ideal (configs scoped to a feature)
>>>
>>> hinted_handoff:
>>> enabled: true
>>> disabled_datacenters:
>>> - DC1
>>> - DC2
>>> max_window: 3h
>>> flush_period: 10s
>>> max_file_size: 128mb
>>> compression:
>>> class_name: LZ4Compressor
>>> parameters:
>>> a: b
>>>
>>> track_warnings:
>>> enabled: true
>>> local_read_size:
>>> warn_threshold: 1mb
>>> abort_threshold: 10mb
>>> coordinator_read_size:
>>> warn_threshold: 5mb
>>> abort_threshold: 20mb
>>>
>>>
>>> OR
>>>
>>> # I had to rename hint configs as there was 0 consistent naming
>>> hinted_handoff_enabled: true
>>> hinted_handoff_disabled_datacenters:
>>> - 'DC1'
>>> - 'DC2'
>>> hinted_handoff_max_window: 3h
>>> hinted_handoff_max_file_size: 128mb
>>> hinted_handoff_flush_period: 10s
>>> hinted_handoff_compression:
>>> class_name: LZ4Compressor
>>> parameters:
>>> a: b
>>>
>>> track_warnings_enabled: true
>>> track_warnings_local_read_size_warn_threshold: 1mb
>>> track_warnings_local_read_size_abort_threshold: 10mb
>>> track_warnings_coordinator_read_size_warn_threshold: 5mb
>>> track_warnings_coordinator_read_size_abort_threshold: 20mb
>>>
>>>
>>> The main issue I have with flat structure is that we have no way to
>>> enforce standard naming; if you look at the hint example there were at
>>> least 3 naming conventions (CASSANDRA-15234 is to clean this up, but can
>> we
>>> actually maintain that?). And one of the core reasons track_warnings
>> went
>>> nested was that warn/abort some times became warn/fail and threshold some
>>> times was thresholds…. By embracing nested structure we can actually
>>> enforce consistency, with flat we have no way to maintain consistency.
>>>
>>> Additionally by embracing the nested structure we can accept a flat one
>> as
>>> well (PR in CASSANDRA-17166 shows this working) if users desire it; so we
>>> get the consistency of nested, and the “grep” benefits of flat.
>>>
>>>
>>>> On Nov 29, 2021, at 2:17 PM, benedict@apache.org wrote:
>>>>
>>>> If we’re thinking of moving towards nested configuration, then before
>>> employing the approach further we would ideally consider what a fully
>>> nested config looks like for the project. Ekaterina has done a lot to
>> clean
>>> up inconsistent naming, but I would hate to repeat the mistakes of our
>> past
>>> by evolving the config in a new direction without any coherent
>> overarching
>>> design.
>>>>
>>>> In case anyone missed it in the earlier discussion, this was my attempt
>>> to prototype a nested config:
>>>
>> https://github.com/belliottsmith/cassandra/blob/5f80d1c0d38873b7a27dc137656d8b81f8e6bbd7/conf/cassandra_nocomment.yaml
>>>>
>>>> I don’t have any specific attachment to it, but settling on some
>>> approximate scheme would be helpful IMO.
>>>>
>>>> From: David Capwell <dc...@apple.com.INVALID>
>>>> Date: Monday, 29 November 2021 at 20:38
>>>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
>>>> Subject: Re: [DISCUSS] Nested YAML configs for new features
>>>>> What should our default example cassandra.yaml file use (flat or
>>> nested)? Currently default shows nested
>>>>
>>>> Was told this statement was confusing, so trying to clarify. At the
>>> moment we do not allow a nested config to be expressed in any way outside
>>> of nesting it (excluding YAML’s ability to inline objects), so if we did
>>> allow flat config representation of nested configs, then this would be a
>>> brand new feature; we currently show the nested structure in
>> cassandra.yaml
>>>>
>>>>> On Nov 29, 2021, at 11:58 AM, David Capwell
>> <dc...@apple.com.INVALID>
>>> wrote:
>>>>>
>>>>> Thanks everyone for the comments, I hope below is a good summary of
>> all
>>> the talking points?
>>>>>
>>>>> We already use nested configs (networking, seed provider, commit
>>> log/hint compression, back pressure, etc.)
>>>>> Flat configs are easier for grep, but can be solved with grep -A/-B
>>> and/or yq
>>>>> It would be possible to support flat versions of our configs in
>>> cassandra.yaml (in addition to the nested versions)
>>>>> "Settings" vtable currently uses the "_" separator (example of
>>> encryption/audit log). Switching to "." Would be a change in behavior
>>> which may impact some users
>>>>> "." Separator for nested configs are common in other systems (yq,
>>> elastic search, etc.)
>>>>> "Structured / nested config is easier for human eyes to read"... "Flat
>>> config is harder for human eyes but easy for simple scripts"
>>>>> For learning what configs are enabled, cassandra.yaml isn't the best
>>> interface as it may not reflect the actual configs; we can better expose
>>> this in CQL and/or Sidecar
>>>>> What should our default example cassandra.yaml file use (flat or
>>> nested)? Currently default shows nested
>>>>> When projecting the Config into CQL, we may want to consider UDTs to
>>> represent the complex types
>>>>> Current limitations in CQL make nested structures hard to work with,
>> it
>>> may be worth wild to expand CQL support for nested structures.
>>>>>
>>>>> I also took a quick stab at enhancing our cassandra.yaml logic to: 1)
>>> be reusable outside of yaml parsing, 2) support setters (we currently do,
>>> but setters must be snake case… I fixed that)…, 3) support both nested
>> and
>>> structured, 4) support ignoring fields in a consistent way (Settings
>> vtable
>>> will include things SnakeYAML won’t and visa-versa).
>>>>>
>>>>> https://github.com/apache/cassandra/pull/1335 <
>>> https://github.com/apache/cassandra/pull/1335><
>> https://github.com/apache/cassandra/pull/1335%3e>. This PR is NOT a final
>>> ready to merge thing, but instead a POC to show how we can solve a lot of
>>> the core problems in a consistent and reusable manner.
>>>>>
>>>>> The following cassandra.yaml was used to show both worlds would work
>>> fine in the config (and compliment each other)
>>>>>
>>>>> track_warnings:
>>>>> enabled: true
>>>>> # nested relative to the local level (TrackWarnings)
>>>>> coordinator_read_size.warn_threshold_kb: 1024
>>>>> local_read_size.abort_threshold_kb: 1024
>>>>> row_index_size:
>>>>> warn_threshold_kb: 1024
>>>>> abort_threshold_kb: 1024
>>>>> # nested relative to the top level
>>>>> track_warnings.coordinator_read_size.abort_threshold_kb: 42
>>>>>
>>>>> For the “Settings” vtable, a new Loader interface was added to get all
>>> the properties, and Properties.flatten would turn every property into a
>>> “flatten” version (isScalar (isPrimitive or not hasSubProperties) or
>>> isCollection). This doesn’t solve 100% of the issues that vtable has
>>> (types such as Duration would need additional translation as they are
>>> Scalar but need a translation from String -> Duration), and doesn’t solve
>>> the fact the table currently uses “_”.
>>>>>
>>>>>> On Nov 29, 2021, at 10:11 AM, benedict@apache.org wrote:
>>>>>>
>>>>>> I meant to imply we should improve our UDT usability to support this
>>> kind of querying, essentially – but that if we support a simple
>>> text->property setup we might want to offer LIKE support so we can search
>>> them (via simple filtering, not any index) – which is actually pretty
>> easy
>>> to provide.
>>>>>>
>>>>>> I think we should aim to provide users all the facilities they need
>> to
>>> interact with config via vtables. If the user requires external tooling,
>> it
>>> suggests a weakness in CQL that we should address, and maybe help the
>> user
>>> in other scenario too…
>>>>>>
>>>>>> From: Joseph Lynch <jo...@gmail.com>
>>>>>> Date: Monday, 29 November 2021 at 17:32
>>>>>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
>>>>>> Subject: Re: [DISCUSS] Nested YAML configs for new features
>>>>>> On Mon, Nov 29, 2021 at 11:51 AM benedict@apache.org
>>>>>> <be...@apache.org> wrote:
>>>>>>>
>>>>>>> Maybe we can make our query language more expressive 😊
>>>>>>>
>>>>>>> We might anyway want to introduce e.g. a LIKE filtering option to
>>> find/discover flattened config parameters?
>>>>>>
>>>>>> This sounds more complicated than just having the settings virtual
>>>>>> table return text (dot encoded) -> text (json) and probably not even
>>>>>> that much more useful. A full table scan on the settings table could
>>>>>> return all top level keys (strings before the first dot) and if we
>>>>>> just return a valid json string then users can bring their own
>>>>>> querying capabilities via jq [1], or one line of code in almost any
>>>>>> programming language (especially python, perl, etc ...).
>>>>>>
>>>>>> Alternatively if we want to modify the grammar it seems supporting
>>>>>> structured data querying on text fields would maybe be more
>> preferable
>>>>>> to LIKE since you could get what you want without a grammar change
>> and
>>>>>> if we could generalize to any text column it would be amazingly
>> useful
>>>>>> elsewhere to users. For example, we could emulate jq's query syntax
>> in
>>>>>> the select which is, imo, best-in-class for quickly querying into
>>>>>> nearest structures. Assuming a key (text) -> value (json) schema:
>>>>>>
>>>>>> 'a' -> "{'b': [{'c': {'d': 4}}]}",
>>>>>>
>>>>>> SELECT json(value).b.0.c.d FROM settings WHERE key = 'a';
>>>>>>
>>>>>> To have exactly jq syntax (but harder to parse) it would be:
>>>>>>
>>>>>> SELECT json(value).b[0].c.d FROM settings WHERE key = 'a';
>>>>>>
>>>>>> Since we're not indexing the structured data in any way, filtering
>>>>>> before selection probably doesn't give us much performance
>> improvement
>>>>>> as we'd still have to parse the whole text field in most cases.
>>>>>>
>>>>>> -Joey
>>>>>>
>>>>>> [1] https://stedolan.github.io/jq/
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: [DISCUSS] Nested YAML configs for new features
Posted by Ekaterina Dimitrova <e....@gmail.com>.
Please find my comments inline below
* Good with nested configs - the question is how they will be introduced
and maintained? I wouldn't advocate for maintaining more than yaml file but
probably as you once mentioned some time ago (if I remember correctly) -
having one format as default and just documenting the support for the other
one/ones. Now which is the default one is a different topic.
* Where/How we group is an open question, maybe we move this to a JIRA as
follow up work to CASSANDRA-15234? - not part of CASSANDRA-15234 as per all
the discussions, already in review (thank you for your first quick round
btw, appreciate it!)
Spoke with Ekaterina about this, and not solved in 15234; let's move this
to a follow up JIRA for 15234? - For the broader audience, currently what I
solve around naming in CASSANDRA-15234 is removing the unit suffix and
moving to the format noun_verb the config parameters names. After all
discussions and realizing the great interest and variety of opinions, I
tried really to split more tickets from CASSANDRA-15234 and to keep
primarily the new custom types and the new framework with backward
compatibility as the main body of work, good also for the reviewers. Last
year I came up with the idea of reorganizing the config file a bit which
led to discussions. So considering my previous point about splitting to a
more incremental approach considering the variety of opinions, I suggested
when submitting for review to open a new ticket for that new organization
of our config file. Probably we can add the abort/fail and any other
similar concerns/questions there post CASSANDRA-15234?
On Fri, 3 Dec 2021 at 13:34, David Capwell <dc...@apple.com.invalid>
wrote:
> Thanks everyone for the feedback! If I am reading this properly I am
> seeing the following
>
> * Good with nested configs
> * Good with YAML layer supporting flat structure (possible foo.bar.baz for
> the path foo: {bar: {baz: 42}}), how this relates with Settings table
> should be resolved, but there is a open ticket for this (enhance our YAML
> CASSANDRA-17166, and support updates to Settings vtable CASSANDRA-15254)
> * Where/How we group is an open question, maybe we move this to a JIRA as
> follow up work to CASSANDRA-15234?
>
> > We’re also mixing terminology already, with limits/thresholds and
> fail/abort.
>
> Spoke with Ekaterina about this, and not solved in 15234; lets move this
> to a follow up JIRA for 15234?
>
> > On Nov 30, 2021, at 6:08 AM, Ekaterina Dimitrova <e....@gmail.com>
> wrote:
> >
> > Thank you for confirming as I misread your email at first :-)
> > I had a chat with David last week and I don’t think his plan is reworking
> > of 15234 but incremental improvements on top of it.
> > Regarding config, after spending time cleaning around and looking more
> into
> > detail my only appeal is:
> > - Centralized management and not 5 places to change things when you add
> new
> > config so we are less error-prone
> > - Documenting things for people who add new config or for our users (I
> > promised and I will do it for 15234 but it will be good to continue doing
> > it with any further changes down the road)
> > - be careful with breaking changes
> >
> > Thank you
> > Ekaterina
> >
> > On Tue, 30 Nov 2021 at 8:59, benedict@apache.org <be...@apache.org>
> > wrote:
> >
> >> I mean that it has been waiting for months, is ready to go, and I don’t
> >> want to hold you up any longer.
> >>
> >> From: Ekaterina Dimitrova <e....@gmail.com>
> >> Date: Tuesday, 30 November 2021 at 13:44
> >> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> >> Subject: Re: [DISCUSS] Nested YAML configs for new features
> >> “
> >> IMO 15234 has sailed – it’s been held up for a long time, and was
> brought
> >> to this list for discussion with no engagement. Ekaterina is long
> overdue
> >> being able to commit her work. “
> >>
> >>
> >> Sailed? I submitted the patch a week ago for review. Not sure how to
> >> understand this statement. Can elaborate, please?
> >>
> >> On Tue, 30 Nov 2021 at 8:09, benedict@apache.org <be...@apache.org>
> >> wrote:
> >>
> >>> The problem with scoping this to “features” is that we end up with at
> >> best
> >>> local coherence. The config file as a whole will end up just as
> >> incoherent
> >>> through its design evolution as it has historically.
> >>>
> >>> If you take a look at my proposed layout for the overall config, there
> is
> >>> a “limits” section that specifies thresholds for reporting warnings and
> >>> errors for various scenario. In this case, we probably don’t also want
> >>> per-feature limits? We’re also mixing terminology already, with
> >>> limits/thresholds and fail/abort.
> >>>
> >>> It’s a lot of work to come up with a coherent and intuitive config
> >> layout.
> >>> We probably want to at least create some documentation in-tree
> >> stipulating
> >>> terminology with respect to plurals, verbs/nouns, and specific terms
> >>> (period, abort, limit, datacenter vs dc, etc), but ideally we would
> have
> >> a
> >>> common end goal for the config file.
> >>>
> >>>> leave non-features to CASSANDRA-15234
> >>>
> >>> IMO 15234 has sailed – it’s been held up for a long time, and was
> brought
> >>> to this list for discussion with no engagement. Ekaterina is long
> overdue
> >>> being able to commit her work.
> >>>
> >>>
> >>> From: David Capwell <dc...@apple.com.INVALID>
> >>> Date: Monday, 29 November 2021 at 23:44
> >>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> >>> Subject: Re: [DISCUSS] Nested YAML configs for new features
> >>>> but I would hate to repeat the mistakes of our past by evolving the
> >>> config in a new direction without any coherent overarching design.
> >>>
> >>> At the start I asked to keep the thread local to new features, but to
> >> more
> >>> flesh out an “overarching design” maybe we should increase the
> “desired”
> >>> scope to be “feature” (and leave non-features to CASSANDRA-15234 -
> >>> Standardise config and JVM parameters)? Aka, do we think the following
> >> is
> >>> more ideal (configs scoped to a feature)
> >>>
> >>> hinted_handoff:
> >>> enabled: true
> >>> disabled_datacenters:
> >>> - DC1
> >>> - DC2
> >>> max_window: 3h
> >>> flush_period: 10s
> >>> max_file_size: 128mb
> >>> compression:
> >>> class_name: LZ4Compressor
> >>> parameters:
> >>> a: b
> >>>
> >>> track_warnings:
> >>> enabled: true
> >>> local_read_size:
> >>> warn_threshold: 1mb
> >>> abort_threshold: 10mb
> >>> coordinator_read_size:
> >>> warn_threshold: 5mb
> >>> abort_threshold: 20mb
> >>>
> >>>
> >>> OR
> >>>
> >>> # I had to rename hint configs as there was 0 consistent naming
> >>> hinted_handoff_enabled: true
> >>> hinted_handoff_disabled_datacenters:
> >>> - 'DC1'
> >>> - 'DC2'
> >>> hinted_handoff_max_window: 3h
> >>> hinted_handoff_max_file_size: 128mb
> >>> hinted_handoff_flush_period: 10s
> >>> hinted_handoff_compression:
> >>> class_name: LZ4Compressor
> >>> parameters:
> >>> a: b
> >>>
> >>> track_warnings_enabled: true
> >>> track_warnings_local_read_size_warn_threshold: 1mb
> >>> track_warnings_local_read_size_abort_threshold: 10mb
> >>> track_warnings_coordinator_read_size_warn_threshold: 5mb
> >>> track_warnings_coordinator_read_size_abort_threshold: 20mb
> >>>
> >>>
> >>> The main issue I have with flat structure is that we have no way to
> >>> enforce standard naming; if you look at the hint example there were at
> >>> least 3 naming conventions (CASSANDRA-15234 is to clean this up, but
> can
> >> we
> >>> actually maintain that?). And one of the core reasons track_warnings
> >> went
> >>> nested was that warn/abort some times became warn/fail and threshold
> some
> >>> times was thresholds…. By embracing nested structure we can actually
> >>> enforce consistency, with flat we have no way to maintain consistency.
> >>>
> >>> Additionally by embracing the nested structure we can accept a flat one
> >> as
> >>> well (PR in CASSANDRA-17166 shows this working) if users desire it; so
> we
> >>> get the consistency of nested, and the “grep” benefits of flat.
> >>>
> >>>
> >>>> On Nov 29, 2021, at 2:17 PM, benedict@apache.org wrote:
> >>>>
> >>>> If we’re thinking of moving towards nested configuration, then before
> >>> employing the approach further we would ideally consider what a fully
> >>> nested config looks like for the project. Ekaterina has done a lot to
> >> clean
> >>> up inconsistent naming, but I would hate to repeat the mistakes of our
> >> past
> >>> by evolving the config in a new direction without any coherent
> >> overarching
> >>> design.
> >>>>
> >>>> In case anyone missed it in the earlier discussion, this was my
> attempt
> >>> to prototype a nested config:
> >>>
> >>
> https://github.com/belliottsmith/cassandra/blob/5f80d1c0d38873b7a27dc137656d8b81f8e6bbd7/conf/cassandra_nocomment.yaml
> >>>>
> >>>> I don’t have any specific attachment to it, but settling on some
> >>> approximate scheme would be helpful IMO.
> >>>>
> >>>> From: David Capwell <dc...@apple.com.INVALID>
> >>>> Date: Monday, 29 November 2021 at 20:38
> >>>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> >>>> Subject: Re: [DISCUSS] Nested YAML configs for new features
> >>>>> What should our default example cassandra.yaml file use (flat or
> >>> nested)? Currently default shows nested
> >>>>
> >>>> Was told this statement was confusing, so trying to clarify. At the
> >>> moment we do not allow a nested config to be expressed in any way
> outside
> >>> of nesting it (excluding YAML’s ability to inline objects), so if we
> did
> >>> allow flat config representation of nested configs, then this would be
> a
> >>> brand new feature; we currently show the nested structure in
> >> cassandra.yaml
> >>>>
> >>>>> On Nov 29, 2021, at 11:58 AM, David Capwell
> >> <dc...@apple.com.INVALID>
> >>> wrote:
> >>>>>
> >>>>> Thanks everyone for the comments, I hope below is a good summary of
> >> all
> >>> the talking points?
> >>>>>
> >>>>> We already use nested configs (networking, seed provider, commit
> >>> log/hint compression, back pressure, etc.)
> >>>>> Flat configs are easier for grep, but can be solved with grep -A/-B
> >>> and/or yq
> >>>>> It would be possible to support flat versions of our configs in
> >>> cassandra.yaml (in addition to the nested versions)
> >>>>> "Settings" vtable currently uses the "_" separator (example of
> >>> encryption/audit log). Switching to "." Would be a change in behavior
> >>> which may impact some users
> >>>>> "." Separator for nested configs are common in other systems (yq,
> >>> elastic search, etc.)
> >>>>> "Structured / nested config is easier for human eyes to read"...
> "Flat
> >>> config is harder for human eyes but easy for simple scripts"
> >>>>> For learning what configs are enabled, cassandra.yaml isn't the best
> >>> interface as it may not reflect the actual configs; we can better
> expose
> >>> this in CQL and/or Sidecar
> >>>>> What should our default example cassandra.yaml file use (flat or
> >>> nested)? Currently default shows nested
> >>>>> When projecting the Config into CQL, we may want to consider UDTs to
> >>> represent the complex types
> >>>>> Current limitations in CQL make nested structures hard to work with,
> >> it
> >>> may be worth wild to expand CQL support for nested structures.
> >>>>>
> >>>>> I also took a quick stab at enhancing our cassandra.yaml logic to: 1)
> >>> be reusable outside of yaml parsing, 2) support setters (we currently
> do,
> >>> but setters must be snake case… I fixed that)…, 3) support both nested
> >> and
> >>> structured, 4) support ignoring fields in a consistent way (Settings
> >> vtable
> >>> will include things SnakeYAML won’t and visa-versa).
> >>>>>
> >>>>> https://github.com/apache/cassandra/pull/1335 <
> >>> https://github.com/apache/cassandra/pull/1335><
> >> https://github.com/apache/cassandra/pull/1335%3e>. This PR is NOT a
> final
> >>> ready to merge thing, but instead a POC to show how we can solve a lot
> of
> >>> the core problems in a consistent and reusable manner.
> >>>>>
> >>>>> The following cassandra.yaml was used to show both worlds would work
> >>> fine in the config (and compliment each other)
> >>>>>
> >>>>> track_warnings:
> >>>>> enabled: true
> >>>>> # nested relative to the local level (TrackWarnings)
> >>>>> coordinator_read_size.warn_threshold_kb: 1024
> >>>>> local_read_size.abort_threshold_kb: 1024
> >>>>> row_index_size:
> >>>>> warn_threshold_kb: 1024
> >>>>> abort_threshold_kb: 1024
> >>>>> # nested relative to the top level
> >>>>> track_warnings.coordinator_read_size.abort_threshold_kb: 42
> >>>>>
> >>>>> For the “Settings” vtable, a new Loader interface was added to get
> all
> >>> the properties, and Properties.flatten would turn every property into a
> >>> “flatten” version (isScalar (isPrimitive or not hasSubProperties) or
> >>> isCollection). This doesn’t solve 100% of the issues that vtable has
> >>> (types such as Duration would need additional translation as they are
> >>> Scalar but need a translation from String -> Duration), and doesn’t
> solve
> >>> the fact the table currently uses “_”.
> >>>>>
> >>>>>> On Nov 29, 2021, at 10:11 AM, benedict@apache.org wrote:
> >>>>>>
> >>>>>> I meant to imply we should improve our UDT usability to support this
> >>> kind of querying, essentially – but that if we support a simple
> >>> text->property setup we might want to offer LIKE support so we can
> search
> >>> them (via simple filtering, not any index) – which is actually pretty
> >> easy
> >>> to provide.
> >>>>>>
> >>>>>> I think we should aim to provide users all the facilities they need
> >> to
> >>> interact with config via vtables. If the user requires external
> tooling,
> >> it
> >>> suggests a weakness in CQL that we should address, and maybe help the
> >> user
> >>> in other scenario too…
> >>>>>>
> >>>>>> From: Joseph Lynch <jo...@gmail.com>
> >>>>>> Date: Monday, 29 November 2021 at 17:32
> >>>>>> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> >>>>>> Subject: Re: [DISCUSS] Nested YAML configs for new features
> >>>>>> On Mon, Nov 29, 2021 at 11:51 AM benedict@apache.org
> >>>>>> <be...@apache.org> wrote:
> >>>>>>>
> >>>>>>> Maybe we can make our query language more expressive 😊
> >>>>>>>
> >>>>>>> We might anyway want to introduce e.g. a LIKE filtering option to
> >>> find/discover flattened config parameters?
> >>>>>>
> >>>>>> This sounds more complicated than just having the settings virtual
> >>>>>> table return text (dot encoded) -> text (json) and probably not even
> >>>>>> that much more useful. A full table scan on the settings table could
> >>>>>> return all top level keys (strings before the first dot) and if we
> >>>>>> just return a valid json string then users can bring their own
> >>>>>> querying capabilities via jq [1], or one line of code in almost any
> >>>>>> programming language (especially python, perl, etc ...).
> >>>>>>
> >>>>>> Alternatively if we want to modify the grammar it seems supporting
> >>>>>> structured data querying on text fields would maybe be more
> >> preferable
> >>>>>> to LIKE since you could get what you want without a grammar change
> >> and
> >>>>>> if we could generalize to any text column it would be amazingly
> >> useful
> >>>>>> elsewhere to users. For example, we could emulate jq's query syntax
> >> in
> >>>>>> the select which is, imo, best-in-class for quickly querying into
> >>>>>> nearest structures. Assuming a key (text) -> value (json) schema:
> >>>>>>
> >>>>>> 'a' -> "{'b': [{'c': {'d': 4}}]}",
> >>>>>>
> >>>>>> SELECT json(value).b.0.c.d FROM settings WHERE key = 'a';
> >>>>>>
> >>>>>> To have exactly jq syntax (but harder to parse) it would be:
> >>>>>>
> >>>>>> SELECT json(value).b[0].c.d FROM settings WHERE key = 'a';
> >>>>>>
> >>>>>> Since we're not indexing the structured data in any way, filtering
> >>>>>> before selection probably doesn't give us much performance
> >> improvement
> >>>>>> as we'd still have to parse the whole text field in most cases.
> >>>>>>
> >>>>>> -Joey
> >>>>>>
> >>>>>> [1] https://stedolan.github.io/jq/
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>