Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Robert Yokota <ra...@gmail.com>]

Hi,

I'm late to this thread, but would like to comment on the double dot/double
asterisk syntax.

Unfortunately double dot is often used in JSON Path as a descendant
selector, see https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html

I think a better notation would be to use backticks to quote any names with
a dot, such as  root.`child.with.dot`.  See Jsonata syntax for example:
http://docs.jsonata.org/simple

Robert

On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
quilcate.jorge@gmail.com> wrote:

> Thanks Chris! I have updated the KIP to include this fix.
>
> I will keep the array as a potential improvement at the moment, and out of
> the scope of this KIP.
>
> Thanks,
> Jorge.
>
> On Tue, 28 Jun 2022 at 23:19, Chris Egerton <fe...@gmail.com>
> wrote:
>
> > Hi Jorge,
> >
> > Apologies for the long delay, had my own KIP-related work to focus on.
> >
> > I think it's fine to include array accesses but it's not a blocker. I'm
> +1
> > either way. On that front though, I think the MaskField section might
> need
> > to be updated as it still mentions arrays and deep-scan?
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> > quilcate.jorge@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > I have update the KIP to the previous state voted, including the
> > > configuration change from `field.style` to `field.syntax.version`.
> > >
> > > I'll bump the vote thread as well to see if there's agreement on adding
> > > this feature to Connect.
> > >
> > > Cheers,
> > > Jorge.
> > >
> > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > > quilcate.jorge@gmail.com> wrote:
> > >
> > > > Thanks, Chris. Your feedback is much appreciated!
> > > >
> > > > I see how the current proposal might be underestimating some edge
> > cases.
> > > > I'm happy to move the design for deep-scan and multi-values to future
> > > > developments related with this KIP and reduce its scope, though open
> > for
> > > > more feedback.
> > > >
> > > > Also, just to be sure, are you proposing also to not include array
> > access
> > > > at this stage?
> > > >
> > > > Thanks,
> > > > Jorge.
> > > >
> > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <fearthecellos@gmail.com
> >
> > > > wrote:
> > > >
> > > >> Hi Jorge,
> > > >>
> > > >> I've done some more thinking and I hate to say it, but I think the
> > > syntax
> > > >> does need to be expanded. Right now it's clear what "a.b" refers to
> > and
> > > >> what "a..b" refers to, but what about "a...b"? Is that referring to
> > > >> subfield ".b" of field "a", or subfield "b" of field "a."? This gets
> > > even
> > > >> more complicated when thinking about fields whose names are
> > exclusively
> > > >> made up of dots.
> > > >>
> > > >> I'm also a little hesitant to mix the cases of multi-value paths and
> > > deep
> > > >> scans. What if you only want to access one subfield deep for an SMT,
> > > >> instead of recursing through all the children of a given field? It's
> > > akin
> > > >> to the distinction between * and ** with file globbing patterns, and
> > > there
> > > >> could be a substantial performance difference if you have
> > heavily-nested
> > > >> fields.
> > > >>
> > > >> Ultimately, I think that if the proposed "field.syntax.version"
> > property
> > > >> sits well with people, it might be better to reduce the scope of the
> > KIP
> > > >> back to the original proposal and just focus on adding support for
> > > >> explicitly-specified nested values, with no multi-value paths
> > > whatsoever,
> > > >> knowing that we have an easy way to introduce new syntax and
> features
> > in
> > > >> the future. (We could probably leave the "a...b" case for that next
> > > >> version
> > > >> too.)
> > > >>
> > > >> I was a huge fan of this KIP before we started trying to address
> more
> > > >> complex use cases, and although I don't want to write those off, I
> > think
> > > >> we
> > > >> may have bitten off more than we can chew in time for the 3.3.0
> > release
> > > >> and
> > > >> would hate to see this KIP get delayed as a result.
> > > >>
> > > >> I'd be really curious to hear from Joshua and Tom on this front,
> > though.
> > > >> Is
> > > >> it acceptable to move more incrementally here and settle on the
> syntax
> > > >> version property as our means of introducing new features, or is it
> > > >> preferable to implement things monolithically and try to get
> > everything
> > > >> (or
> > > >> at least, as much as possible) right the first time?
> > > >>
> > > >> Thanks again for your continued effort on this KIP!
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Chris
> > > >>
> > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate Otoya <
> > > >> quilcate.jorge@gmail.com> wrote:
> > > >>
> > > >> > Thanks, Chris!
> > > >> >
> > > >> > Please, find my comments below:
> > > >> >
> > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> fearthecellos@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Jorge,
> > > >> > >
> > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > >> > >
> > > >> > > 1. Under the "Accessing multiple values by deep-scan" header
> it's
> > > >> stated
> > > >> > > that "If deep-scan is used, it must have only one field after
> the
> > > >> > asterisk
> > > >> > > level.". However, in example 3 for the Cast SMT and other
> examples
> > > for
> > > >> > > other SMTs, the spec contains a field of "*.child.k2", which
> > appears
> > > >> to
> > > >> > > have two fields after the asterisk level. I may be
> > misunderstanding
> > > >> the
> > > >> > > proposal, but it seems like the two contradict each other.
> > > >> > >
> > > >> >
> > > >> > Thanks for catching this. I have clarified it by removing this
> > > >> restriction.
> > > >> > Also, have extended the deep-scan scenarios.
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > 2. I'm a little unclear on why we need the special handling for
> > > arrays
> > > >> > > where, for an array field "a", the field name "a" can be treated
> > as
> > > >> > either
> > > >> > > the array itself, or every element in the array. Is there a
> reason
> > > we
> > > >> > can't
> > > >> > > use the field name "a.*" to handle the latter case, and "a" to
> > > handle
> > > >> the
> > > >> > > former?
> > > >> > >
> > > >> >
> > > >> > Agree, this is confusing. I like the `a.*` approach to access
> array
> > > >> items.
> > > >> > I have added this to the proposal.
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > 3. How would a user specify that they'd like to access a field
> > with
> > > >> the
> > > >> > > literal name "*"?
> > > >> > >
> > > >> >
> > > >> > Good one. I'm proposing an approach similar to how it's proposed
> to
> > > >> escape
> > > >> > dots, with a double-asterisk. Curious on your thoughts around
> this.
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > 4. For the Cast SMT, do you think it might bite some people if
> > > fields
> > > >> > that
> > > >> > > can't be cast correctly are silently ignored? I'm imagining the
> > case
> > > >> > where
> > > >> > > none of the fields in a multi-path expression can be cast
> > correctly
> > > >> and
> > > >> > it
> > > >> > > ends up eating half of someone's day to track down why their SMT
> > > isn't
> > > >> > > doing anything.
> > > >> > >
> > > >> >
> > > >> > If I understand correctly, this challenge could be relevant across
> > > SMTs.
> > > >> > At the moment, most/all? SMTs just silently ignore.
> > > >> > Was thinking about adding a flag `field.on.path.not.found` to
> either
> > > >> ignore
> > > >> > or fail when no paths are found. What do your think?
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > 5. For the ExtractField and ValueToKey SMTs, what happens if a
> > > >> deep-scan
> > > >> > > field name is used, but only one field is found? Is the
> resulting
> > > >> field
> > > >> > > still an array, or is it just the single field that was found?
> > (FWIW
> > > >> I'm
> > > >> > > leaning towards keeping it an array just to keep schemas
> > consistent
> > > >> in a
> > > >> > > pipeline in case the number of fields found fluctuates across
> > > >> records.)
> > > >> > >
> > > >> > > Agree. Will clarify that an array is always produced even for 1
> > or 0
> > > >> > fields are found.
> > > >> >
> > > >> >
> > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that "As this SMT
> > > affects
> > > >> > only
> > > >> > > existing fields, additional configurations will not be
> required.".
> > > >> Given
> > > >> > > the new "field.syntax.version" property, this part should
> probably
> > > be
> > > >> > > removed.
> > > >> > >
> > > >> > Agree.
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > 7. Is recursive descent intentionally excluded? That was an
> > > important
> > > >> > part
> > > >> > > of Joshua's KIP and his feedback on this KIP; I think it's worth
> > > >> pursuing
> > > >> > > if we can.
> > > >> > >
> > > >> >
> > > >> > My understanding from Joshua's feedback is that by including
> support
> > > for
> > > >> > deep-scan, we are already covering the recursive functionality.
> > > Though,
> > > >> I
> > > >> > may be missing something.
> > > >> >
> > > >> > >
> > > >> > >
> > > >> > > Cheers,
> > > >> > >
> > > >> > > Chris
> > > >> > >
> > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate Otoya <
> > > >> > > quilcate.jorge@gmail.com> wrote:
> > > >> > >
> > > >> > > > Thanks, Chris!
> > > >> > > >
> > > >> > > > I have updated the KIP with the rejected alternatives updated.
> > > >> Also, I
> > > >> > > have
> > > >> > > > drafted the support for arrays and deep scans as part of the
> > > >> proposed
> > > >> > > > notation to make it more complete, even though these can be
> > > >> implemented
> > > >> > > in
> > > >> > > > multiple PRs.
> > > >> > > >
> > > >> > > > Looking forward to your feedback.
> > > >> > > >
> > > >> > > > Cheers,
> > > >> > > > Jorge.
> > > >> > > >
> > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > >> fearthecellos@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Jorge,
> > > >> > > > >
> > > >> > > > > I really appreciate the effort you've made to simplify the
> > > syntax
> > > >> and
> > > >> > > > > feature set of a JSONPath-based approach as much as
> possible.
> > > I'm
> > > >> > still
> > > >> > > > > hesitant to continue with it, though.
> > > >> > > > >
> > > >> > > > > 1. The syntax is much less friendly. Just compare
> > > >> "top.mid.bottom" to
> > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses JSONPath or
> > > even
> > > >> > JSON,
> > > >> > > > and
> > > >> > > > > the learning curve for the former is going to be steeper.
> The
> > > >> > examples
> > > >> > > in
> > > >> > > > > the new KIP draft you published demonstrate this pretty
> well,
> > > and
> > > >> > this
> > > >> > > is
> > > >> > > > > without diving into the details of what escape syntax would
> > look
> > > >> > like.
> > > >> > > > >
> > > >> > > > > 2. The three new major features that this syntax adds (array
> > > >> > accesses,
> > > >> > > > deep
> > > >> > > > > scans, and multi-value paths) could all be added pretty
> easily
> > > to
> > > >> the
> > > >> > > dot
> > > >> > > > > notation syntax without introducing brackets and dollar
> signs.
> > > >> Array
> > > >> > > > > accesses can be described using the same syntax as
> struct/map
> > > >> field
> > > >> > > > access,
> > > >> > > > > deep scans can be described using '*', and multi-value paths
> > can
> > > >> be
> > > >> > > > > described by referencing the name of a field that's expected
> > to
> > > >> have
> > > >> > > > > children. These are all top-of-the-head ideas and can
> probably
> > > be
> > > >> > > > refined,
> > > >> > > > > but hopefully they demonstrate that we can keep the syntax
> > > simple
> > > >> > > without
> > > >> > > > > sacrificing features. Of course, the question of leaving
> room
> > > for
> > > >> > > future
> > > >> > > > > features might arise... given that these are the
> > out-of-the-box
> > > >> SMTs
> > > >> > > that
> > > >> > > > > are likely to be the first that many people encounter, I'd
> err
> > > on
> > > >> the
> > > >> > > > side
> > > >> > > > > of simplicity and a gentle learning curve; if people need to
> > get
> > > >> more
> > > >> > > out
> > > >> > > > > of transforms, the option to implement their own is still
> > there.
> > > >> If
> > > >> > we
> > > >> > > > can
> > > >> > > > > address 95% of use cases with something easy to use, it's
> not
> > > >> worth
> > > >> > > > making
> > > >> > > > > the feature harder for everyone to use just to accommodate
> the
> > > >> > > remaining
> > > >> > > > > 5%.
> > > >> > > > >
> > > >> > > > > 3. The advantage of leveraging an existing syntax is
> twofold:
> > > >> users
> > > >> > who
> > > >> > > > are
> > > >> > > > > already familiar with that syntax don't need to learn a new
> > > >> syntax,
> > > >> > and
> > > >> > > > > maintainers of the syntax get to leverage existing libraries
> > and
> > > >> > > > > documentation for the syntax. With the current proposal,
> > reusing
> > > >> > > > libraries
> > > >> > > > > is off the table, which means that we're going to have to
> > parse
> > > >> this
> > > >> > > > > ourselves (including all the escape syntax edge cases), and
> we
> > > >> won't
> > > >> > be
> > > >> > > > > able to automatically leverage new features added to that
> > > syntax.
> > > >> And
> > > >> > > > given
> > > >> > > > > how stripped-down the syntax is in comparison to full
> > JSONPath,
> > > >> > there's
> > > >> > > > > still going to be a learning curve for users who are already
> > > >> familiar
> > > >> > > > with
> > > >> > > > > it, and we'll still have to document how Connect's variant
> of
> > > >> > JSONPath
> > > >> > > > > works either instead of or in addition to just linking to a
> > > >> > > > well-maintained
> > > >> > > > > third-party docs site.
> > > >> > > > >
> > > >> > > > > On the topic of "field.path" and "include.path" vs.
> > > >> "field.style", I
> > > >> > > > > actually think that a single property per SMT is cleaner and
> > > >> simpler.
> > > >> > > > > Allowing users to mix and match styles within the same SMT
> > > config
> > > >> > seems
> > > >> > > > > like a recipe for confusion, and with a single property that
> > > >> dictates
> > > >> > > > field
> > > >> > > > > syntax behavior, we leave the door open to change the
> default
> > > at a
> > > >> > > later
> > > >> > > > > date. We could even fully remove support for plain field
> > > notation
> > > >> at
> > > >> > > some
> > > >> > > > > point and still be able to retain the simple property names
> of
> > > >> > "field",
> > > >> > > > > "include", etc. instead of forcing people to use
> "field.path"
> > > and
> > > >> the
> > > >> > > > like.
> > > >> > > > > That said, the term "field.style" and the permitted values
> > might
> > > >> be a
> > > >> > > > > little ambiguous. One alternative, though a little
> > heavy-handed,
> > > >> is
> > > >> > to
> > > >> > > > > change it to "field.syntax.version" with permitted values of
> > > "V1"
> > > >> > > > (default,
> > > >> > > > > equivalent to "field.style = plain") and "V2" (equivalent to
> > > >> > > > "field.style =
> > > >> > > > > nested"). This would leave us room in the future to make
> > further
> > > >> > > changes
> > > >> > > > to
> > > >> > > > > the syntax without having to come up with new names,
> although
> > it
> > > >> does
> > > >> > > > > sacrifice a little bit in that the permitted values are no
> > > longer
> > > >> > > > > self-describing.
> > > >> > > > >
> > > >> > > > > Cheers,
> > > >> > > > >
> > > >> > > > > Chris
> > > >> > > > >
> > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban Quilcate
> Otoya <
> > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > >> > > > >
> > > >> > > > > > Thank you all for your feedback, and sorry for the long
> wait
> > > >> for a
> > > >> > > > reply.
> > > >> > > > > >
> > > >> > > > > > I would like to explore the idea of
> JSONPath-inspired/subset
> > > >> > > notation a
> > > >> > > > > bit
> > > >> > > > > > further:
> > > >> > > > > >
> > > >> > > > > > It will need to be a much-reduced version of JSONPath:
> > > >> > > > > > - No full support for JsonPath therefore an additional
> > > >> dependency.
> > > >> > > > > > - All paths must start with '$'
> > > >> > > > > > - No functions support or other operators allowed.
> > > >> > > > > > - JsonPath dotted and square-bracket notations can be
> > > supported
> > > >> to
> > > >> > > > avoid
> > > >> > > > > > escaping dots or other characters: `$a.b.c` and
> > > `$['a.b']['c']`
> > > >> -
> > > >> > or
> > > >> > > we
> > > >> > > > > > could only support the second one as it's more complete.
> > > >> > > > > > - Add support for arrays with `[<integer>]`, e.g.
> `$a.[1].b`
> > > >> > > > > > - Add support for multiple-value paths using array access
> > > >> `$a.*.b`
> > > >> > or
> > > >> > > > > deep
> > > >> > > > > > scan `$a..b`.
> > > >> > > > > >     - Some SMTs that will benefit from this: `MaskField`,
> > > >> `Cast`,
> > > >> > > > > > `ReplaceField`.
> > > >> > > > > > - We could introduce a `path[s]` config under the current
> > > >> > > > configurations
> > > >> > > > > to
> > > >> > > > > > apply this feature so no compatibility issues are
> > introduced:
> > > >> e.g.
> > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > >> > > > > >
> > > >> > > > > > With these,  100, 101, 102, and 103 will be effectively
> > > solved.
> > > >> > > > > >
> > > >> > > > > > The main challenge that I see at the moment is that by
> being
> > > >> JSON
> > > >> > > > > > path-like, there may be some edge cases that I can't
> foresee
> > > at
> > > >> the
> > > >> > > > > moment,
> > > >> > > > > > that could make this hard to implement, test, and maintain
> > in
> > > >> the
> > > >> > > long
> > > >> > > > > run.
> > > >> > > > > >
> > > >> > > > > > I'll appreciate your feedback on how this JSONPath-based
> > > >> > alternative
> > > >> > > > > > compares to the dotted notation initially proposed, and if
> > it
> > > >> > matches
> > > >> > > > > your
> > > >> > > > > > feedback.
> > > >> > > > > > I have drafted a copy of the KIP to change the examples to
> > > >> JSONPath
> > > >> > > > > > approach and validate some differences:
> > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > >> > > > > >
> > > >> > > > > > About 104. I agreed with Chris. We can handle this as part
> > of
> > > a
> > > >> new
> > > >> > > > KIP.
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Jorge.
> > > >> > > > > >
> > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > >> > fearthecellos@gmail.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Joshua,
> > > >> > > > > > >
> > > >> > > > > > > I have a few reservations about using JsonPath notation
> > > here.
> > > >> > > > > > >
> > > >> > > > > > > 1. There's likely to be a substantial performance
> penalty
> > > for
> > > >> > > > > converting
> > > >> > > > > > > between the Kafka Connect format and something that a
> > > JsonPath
> > > >> > > > library
> > > >> > > > > > > would understand.
> > > >> > > > > > >
> > > >> > > > > > > 2. The complexity of the feature will be significantly
> > > >> higher. It
> > > >> > > > will
> > > >> > > > > be
> > > >> > > > > > > harder to test, document, and implement. There will be
> > many
> > > >> more
> > > >> > > edge
> > > >> > > > > > cases
> > > >> > > > > > > to consider and support, and we'll be on the hook to
> > handle
> > > >> any
> > > >> > > bugs
> > > >> > > > or
> > > >> > > > > > > inconsistencies that arise, either as a result of our
> use
> > of
> > > >> the
> > > >> > > > > JsonPath
> > > >> > > > > > > library we choose, or as a result of bugs in that
> library.
> > > >> > > > > > >
> > > >> > > > > > > 3. It's not clear that JsonPath is superior or even
> > suitable
> > > >> for
> > > >> > > some
> > > >> > > > > of
> > > >> > > > > > > the SMTs proposed here. What would be the advantages of
> > > >> JsonPath
> > > >> > > with
> > > >> > > > > the
> > > >> > > > > > > InsertField or HoistField SMTs?
> > > >> > > > > > >
> > > >> > > > > > > I also don't think that adding dot notation is
> unfriendly
> > to
> > > >> > users;
> > > >> > > > > many
> > > >> > > > > > > have proposed this type of syntax in the past, and it's
> > > >> > frequently
> > > >> > > > used
> > > >> > > > > > in
> > > >> > > > > > > informal discussions to refer to nested fields. If the
> > > >> proposed
> > > >> > > > syntax
> > > >> > > > > > was
> > > >> > > > > > > not already intuitive then a case against deciding on
> our
> > > own
> > > >> > might
> > > >> > > > be
> > > >> > > > > > more
> > > >> > > > > > > convincing, but as things stand, simple dot notation is
> > > likely
> > > >> > > going
> > > >> > > > to
> > > >> > > > > > be
> > > >> > > > > > > easier for users to understand than JsonPath syntax.
> > > >> > > > > > >
> > > >> > > > > > > Cheers,
> > > >> > > > > > >
> > > >> > > > > > > Chris
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua Grisham <
> > > >> > > > grishamjosh@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hello all!
> > > >> > > > > > > >
> > > >> > > > > > > > Sorry that I come a bit later to the party here, but I
> > am
> > > >> the
> > > >> > one
> > > >> > > > who
> > > >> > > > > > > wrote
> > > >> > > > > > > > KIP-683 [1] for recursive support (just simply looping
> > > >> through
> > > >> > > all
> > > >> > > > > > child
> > > >> > > > > > > > non-primitive structures for the same matching
> name(s))
> > > >> which
> > > >> > is
> > > >> > > a
> > > >> > > > > > > slightly
> > > >> > > > > > > > different way to try and solve a similar requirement
> --
> > > >> > > > unfortunately
> > > >> > > > > > at
> > > >> > > > > > > > the time the dev community was not quite as active and
> > > then
> > > >> I
> > > >> > > also
> > > >> > > > > got
> > > >> > > > > > > busy
> > > >> > > > > > > > with work and just life in general so wasn't able to
> > > follow
> > > >> up
> > > >> > or
> > > >> > > > > push
> > > >> > > > > > > it.
> > > >> > > > > > > >
> > > >> > > > > > > > I do think it is a very good idea to have some kind of
> > > >> > path-like
> > > >> > > > > > > expression
> > > >> > > > > > > > to be able to specifically address a nested field, as
> I
> > > can
> > > >> see
> > > >> > > > that
> > > >> > > > > > the
> > > >> > > > > > > > simple "recursive" case could potentially result in
> > > >> unwanted or
> > > >> > > > > > > unexpected
> > > >> > > > > > > > behavior, plus there is the potential to introduce a
> bit
> > > of
> > > >> a
> > > >> > > > > > performance
> > > >> > > > > > > > hit to always loop through everything in cases where
> the
> > > >> > > > > schemas/values
> > > >> > > > > > > > might be quite large.
> > > >> > > > > > > >
> > > >> > > > > > > > One thing I wanted to ask: instead of creating a new
> > "path
> > > >> > > parser"
> > > >> > > > > > > > including some bespoke or "borrowed" syntax, why not
> > just
> > > >> use
> > > >> > > > > something
> > > >> > > > > > > > that already exists? Specifically here I am thinking
> > about
> > > >> > > > JsonPath (
> > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > >> > > > > > > >
> > > >> > > > > > > > There is already quite nice support in JsonPath for
> > > handling
> > > >> > > > special
> > > >> > > > > > > > characters in field names, for handling different
> > > >> non-primitive
> > > >> > > > types
> > > >> > > > > > > > (arrays etc), for handling multiple levels of nesting,
> > etc
> > > >> etc.
> > > >> > > > > Would
> > > >> > > > > > it
> > > >> > > > > > > > be possible to instead to re-think this and maybe have
> > > some
> > > >> > kind
> > > >> > > of
> > > >> > > > > > > > JsonPath-based Schema selector / updater and/or
> > > >> JsonPath-based
> > > >> > > > Value
> > > >> > > > > > > > selector / updater? Conceptually this feels like it
> > makes
> > > >> sense
> > > >> > > to
> > > >> > > > > me,
> > > >> > > > > > as
> > > >> > > > > > > > from the top of my head it would be quite a natural
> fit
> > to
> > > >> map
> > > >> > a
> > > >> > > > Json
> > > >> > > > > > > data
> > > >> > > > > > > > structure to the Connect API data structure (and you
> > could
> > > >> > > > > potentially
> > > >> > > > > > > even
> > > >> > > > > > > > try to leverage the existing Json-to-Connect
> > > >> > > > serializer/deserializer
> > > >> > > > > to
> > > >> > > > > > > > help out with this even in a more "out of the
> > box"-feeling
> > > >> kind
> > > >> > > of
> > > >> > > > > > way).
> > > >> > > > > > > >
> > > >> > > > > > > > Maybe also as Tom mentioned, this part (in my example,
> > > this
> > > >> > > > > > > JsonPath-based
> > > >> > > > > > > > "thing") could even be a generic API that could be
> used
> > by
> > > >> any
> > > >> > > SMT,
> > > >> > > > > > > > including used in custom ones built by the community.
> > > Then
> > > >> I
> > > >> > > think
> > > >> > > > > to
> > > >> > > > > > > use
> > > >> > > > > > > > a completely separate config property somehow related
> to
> > > >> "path"
> > > >> > > (as
> > > >> > > > > Tom
> > > >> > > > > > > > also mentioned) would also make a lot of sense here as
> > > well.
> > > >> > This
> > > >> > > > > way,
> > > >> > > > > > if
> > > >> > > > > > > > you select based on "path" then this JsonPath-based
> API
> > > >> would
> > > >> > be
> > > >> > > > > used,
> > > >> > > > > > > > otherwise it could use something similar to the
> existing
> > > >> > > get-field
> > > >> > > > > > based
> > > >> > > > > > > > approach (which I guess could also be refactored into
> > some
> > > >> kind
> > > >> > > of
> > > >> > > > > > > utility
> > > >> > > > > > > > / API as well if it made sense?)
> > > >> > > > > > > >
> > > >> > > > > > > > And with that in mind, if this was the kind of
> direction
> > > to
> > > >> go,
> > > >> > > > then
> > > >> > > > > a
> > > >> > > > > > > > "recursive" capability like I pitched in KIP-683 would
> > > also
> > > >> > > become
> > > >> > > > > > > > unnecessary because you could easily write a JsonPath
> > > >> > expression
> > > >> > > > like
> > > >> > > > > > > > "$..someRecuriveField" and it would do the same thing
> > (on
> > > >> top
> > > >> > of
> > > >> > > > > > anything
> > > >> > > > > > > > else you would want to do that is already supported by
> > > >> > JsonPath).
> > > >> > > > > Then
> > > >> > > > > > we
> > > >> > > > > > > > could also kill that older KIP and do a bit of
> clean-up
> > :)
> > > >> > > > > > > >
> > > >> > > > > > > > [1] -
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > >> > > > > > > >
> > > >> > > > > > > > Just some extra food for thought. All-in-all I think
> > this
> > > >> is a
> > > >> > > > super
> > > >> > > > > > > great
> > > >> > > > > > > > initiative!
> > > >> > > > > > > >
> > > >> > > > > > > > Best,
> > > >> > > > > > > >
> > > >> > > > > > > > Joshua
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris Egerton <
> > > >> > > > > > > fearthecellos@gmail.com
> > > >> > > > > > > > >:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Tom,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for taking a look at this, and for your
> > > thoughtful
> > > >> > > > comments.
> > > >> > > > > > > I'll
> > > >> > > > > > > > > leave it up to Jorge to address most of your
> comments
> > > but
> > > >> I
> > > >> > > > wanted
> > > >> > > > > to
> > > >> > > > > > > > share
> > > >> > > > > > > > > a couple quick thoughts I had regarding 103 and 104.
> > > >> > > > > > > > >
> > > >> > > > > > > > > 103. Like you, I was envisioning a possible syntax
> for
> > > >> array
> > > >> > > > access
> > > >> > > > > > > that
> > > >> > > > > > > > > used classic C-style brackets; e.g., `arr[index]`.
> > > >> However, I
> > > >> > > > > wonder
> > > >> > > > > > if
> > > >> > > > > > > > we
> > > >> > > > > > > > > could keep things simple and use the same syntax
> that
> > > >> we're
> > > >> > > > > proposing
> > > >> > > > > > > for
> > > >> > > > > > > > > nested field access? In other words, instead of
> > > >> `arr[index]`,
> > > >> > > > you'd
> > > >> > > > > > > write
> > > >> > > > > > > > > `arr.index`. It'd save us and users the headache of
> > > >> reserving
> > > >> > > > > > > characters
> > > >> > > > > > > > > now that would need to be escaped even if their
> > > unescaped
> > > >> > > > brethren
> > > >> > > > > > > aren't
> > > >> > > > > > > > > used for anything, and also avoid the question of
> what
> > > >> > exactly
> > > >> > > we
> > > >> > > > > > > should
> > > >> > > > > > > > do
> > > >> > > > > > > > > when we see a config that uses reserved characters
> > that
> > > >> > aren't
> > > >> > > > yet
> > > >> > > > > > > > > supported (throwing an exception seems pretty
> > unfriendly
> > > >> for
> > > >> > > new
> > > >> > > > > > > users).
> > > >> > > > > > > > >
> > > >> > > > > > > > > 104. This would probably be useful, but it would
> come
> > > with
> > > >> > some
> > > >> > > > > nasty
> > > >> > > > > > > > > compatibility questions that would need to be
> > addressed
> > > if
> > > >> > we'd
> > > >> > > > > want
> > > >> > > > > > > SMTs
> > > >> > > > > > > > > that leverage this new API to be viable for older
> > > >> versions of
> > > >> > > > > > Connect.
> > > >> > > > > > > If
> > > >> > > > > > > > > we package and distribute this feature as a library
> > > >> (either
> > > >> > via
> > > >> > > > an
> > > >> > > > > > > > entirely
> > > >> > > > > > > > > new artifact, or as part of the existing
> > > >> connect-transforms
> > > >> > or
> > > >> > > > > > > > connect-api
> > > >> > > > > > > > > artifacts), then we'd have to either sidestep the
> > > existing
> > > >> > > plugin
> > > >> > > > > > > > isolation
> > > >> > > > > > > > > logic [1] that basically makes it impossible for
> > Connect
> > > >> > > plugins
> > > >> > > > to
> > > >> > > > > > > ship
> > > >> > > > > > > > > their own versions of Connect artifacts, or issue a
> > big
> > > >> > warning
> > > >> > > > to
> > > >> > > > > > > people
> > > >> > > > > > > > > that any SMT that uses this API won't work with any
> > > older
> > > >> > > > versions
> > > >> > > > > of
> > > >> > > > > > > > > Connect. There's also some other features we might
> > want
> > > to
> > > >> > add
> > > >> > > in
> > > >> > > > > an
> > > >> > > > > > > > > SMT-utils library such as the existing, internal,
> > utils
> > > >> that
> > > >> > > > > Connect
> > > >> > > > > > > uses
> > > >> > > > > > > > > right now [2]. It may be worth exploring this in a
> > > >> separate
> > > >> > KIP
> > > >> > > > of
> > > >> > > > > > its
> > > >> > > > > > > > own.
> > > >> > > > > > > > >
> > > >> > > > > > > > > [1] -
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > >> > > > > > > > >
> > > >> > > > > > > > > [2] -
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > >> > > > > > > > >
> > > >> > > > > > > > > Cheers,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Chris
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom Bentley <
> > > >> > > tbentley@redhat.com
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for the KIP, especially for the examples
> > which
> > > >> are
> > > >> > > > > > > super-clear.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 100. The name `field.style` isn't so clear for
> > > something
> > > >> > like
> > > >> > > > > > > > > ReplaceField:
> > > >> > > > > > > > > > it's not so obvious that field.style applies to
> > > >> `include`
> > > >> > and
> > > >> > > > > > > > `exclude`.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 101. The permitted values for `field.style` don't
> > seem
> > > >> > > terribly
> > > >> > > > > > > > intuitive
> > > >> > > > > > > > > > (to me anyway): the meaning of `plain` isn't very
> > > >> > guessable.
> > > >> > > > Why
> > > >> > > > > > not
> > > >> > > > > > > > > > `top-level` or `root` instead? Also `nested` could
> > be
> > > >> > > > > misconstrued
> > > >> > > > > > to
> > > >> > > > > > > > > mean
> > > >> > > > > > > > > > nested-but-not-top-level, so perhaps `recursive`
> or
> > > >> > > `cascading`
> > > >> > > > > > might
> > > >> > > > > > > > be
> > > >> > > > > > > > > > better?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 102. I'm torn on whether making the interpretation
> > of
> > > >> > > existing
> > > >> > > > > > > configs
> > > >> > > > > > > > > like
> > > >> > > > > > > > > > `field` be dependent on `field.style` is a good
> > idea.
> > > I
> > > >> can
> > > >> > > see
> > > >> > > > > > that
> > > >> > > > > > > > it's
> > > >> > > > > > > > > > the simplest thing to do, but it just feels a bit
> > odd
> > > >> that
> > > >> > > > > > sometimes
> > > >> > > > > > > > the
> > > >> > > > > > > > > > `field` would actually be a path and have
> different
> > > >> > escaping
> > > >> > > > > rules.
> > > >> > > > > > > An
> > > >> > > > > > > > > > alternative would be to come up with a parallel
> set
> > of
> > > >> > config
> > > >> > > > > names
> > > >> > > > > > > > (e.g.
> > > >> > > > > > > > > > as well as "field" an SMT might support "path")
> > which
> > > >> were
> > > >> > > > > defined
> > > >> > > > > > to
> > > >> > > > > > > > > > always take paths, thus avoiding the changeable
> > > >> > > interpretation
> > > >> > > > of
> > > >> > > > > > the
> > > >> > > > > > > > > > existing configs. The SMT's #configure() would
> need
> > to
> > > >> > throw
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > > > > case
> > > >> > > > > > > > > > that both configs were given. I can see that that
> > > would
> > > >> be
> > > >> > > more
> > > >> > > > > > work
> > > >> > > > > > > in
> > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 103. I think in order to allow for supporting
> arrays
> > > in
> > > >> a
> > > >> > > later
> > > >> > > > > KIP
> > > >> > > > > > > > > (which
> > > >> > > > > > > > > > certainly seems like it could be useful), we'd
> want
> > to
> > > >> > > specify
> > > >> > > > > the
> > > >> > > > > > > > syntax
> > > >> > > > > > > > > > now, even if it wasn't implemented under this KIP.
> > > >> That's
> > > >> > > > > because I
> > > >> > > > > > > > don't
> > > >> > > > > > > > > > think you can't exclude the possibility that
> > > characters
> > > >> > such
> > > >> > > as
> > > >> > > > > `[`
> > > >> > > > > > > and
> > > >> > > > > > > > > `]`
> > > >> > > > > > > > > > appear in field names. So you'd have a
> compatibility
> > > >> > problem
> > > >> > > if
> > > >> > > > > > > people
> > > >> > > > > > > > > > started using the features of this KIP to access
> > such
> > > >> > fields,
> > > >> > > > > only
> > > >> > > > > > > for
> > > >> > > > > > > > > > those characters to change their meaning under a
> > later
> > > >> KIP.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 104. I also wonder whether making paths into a
> > public
> > > >> Java
> > > >> > > API,
> > > >> > > > > for
> > > >> > > > > > > use
> > > >> > > > > > > > > by
> > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks again,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Tom
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris Egerton <
> > > >> > > > > > fearthecellos@gmail.com
> > > >> > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge Esteban
> Quilcate
> > > >> Otoya
> > > >> > <
> > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Thank you, Chris! Not possible without your
> > > >> feedback.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04, Chris Egerton <
> > > >> > > > > > > > fearthecellos@gmail.com
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thank you for sticking through this. I have
> > one
> > > >> small
> > > >> > > > > remark
> > > >> > > > > > > and
> > > >> > > > > > > > > one
> > > >> > > > > > > > > > > > small
> > > >> > > > > > > > > > > > > clarification; assuming you agree with me on
> > > them
> > > >> > then
> > > >> > > > I'm
> > > >> > > > > > > ready
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > vote
> > > >> > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > the KIP.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > 1. InsertField: The
> "field.on.missing.parent"
> > > and
> > > >> > > > > > > > > > > > "field.on.existing.field"
> > > >> > > > > > > > > > > > > docs both mention a permitted value of
> > "ingore";
> > > >> this
> > > >> > > > > should
> > > >> > > > > > be
> > > >> > > > > > > > > > > "ignore",
> > > >> > > > > > > > > > > > > right?
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Of course, one more typo :)
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > 2. InsertField: The examples are still
> missing
> > > the
> > > >> > > > > > > "field.style"
> > > >> > > > > > > > > > > property
> > > >> > > > > > > > > > > > > from the configurations. They should all
> > include
> > > >> the
> > > >> > > > > property
> > > >> > > > > > > > > > > > > "transforms.smt1.field.style": "nested",
> > > correct?
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Yes, it is there. I think I know what you mean
> > > now,
> > > >> > seems
> > > >> > > > > that
> > > >> > > > > > > > > > Confluence
> > > >> > > > > > > > > > > > is putting everything in one line when it's in
> > > >> separate
> > > >> > > > lines
> > > >> > > > > > in
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > editor.
> > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks again for working through this, and
> > > >> > > > congratulations
> > > >> > > > > > on a
> > > >> > > > > > > > > > > > > well-written KIP!
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Chris
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM Jorge
> Esteban
> > > >> > Quilcate
> > > >> > > > > Otoya
> > > >> > > > > > <
> > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Thank you, Chris! I apply these
> improvements
> > > to
> > > >> the
> > > >> > > > KIP,
> > > >> > > > > > let
> > > >> > > > > > > me
> > > >> > > > > > > > > > know
> > > >> > > > > > > > > > > > how
> > > >> > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > looks now.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43, Chris
> Egerton
> > <
> > > >> > > > > > > > > > fearthecellos@gmail.com
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Wow, those examples are great! A few
> more
> > > >> > remarks,
> > > >> > > > but
> > > >> > > > > I
> > > >> > > > > > > > think
> > > >> > > > > > > > > > > we're
> > > >> > > > > > > > > > > > > > > getting close:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > 1. The examples differ across SMTs with
> > the
> > > >> name
> > > >> > of
> > > >> > > > the
> > > >> > > > > > > > > > > > > newly-introduced
> > > >> > > > > > > > > > > > > > > style property; some of them use
> > > >> "field.style",
> > > >> > and
> > > >> > > > > some
> > > >> > > > > > > use
> > > >> > > > > > > > > > > > > > > "fields.style". I think for
> consistency's
> > > >> sake we
> > > >> > > > > should
> > > >> > > > > > > > stick
> > > >> > > > > > > > > > with
> > > >> > > > > > > > > > > > > just
> > > >> > > > > > > > > > > > > > > "field.style"; otherwise it could be
> > painful
> > > >> for
> > > >> > > > users
> > > >> > > > > to
> > > >> > > > > > > > have
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > > > remember
> > > >> > > > > > > > > > > > > > > which to use.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed the config
> names
> > > to
> > > >> > > > > > > `field.style`.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > 2. Some of the examples are off:
> > > >> > > > > > > > > > > > > > > - TimestampConverter: the input in the
> > > second
> > > >> > > example
> > > >> > > > > > > ("when
> > > >> > > > > > > > > > field
> > > >> > > > > > > > > > > > > names
> > > >> > > > > > > > > > > > > > > include dots") doesn't contain a field
> > with
> > > a
> > > >> > > dotted
> > > >> > > > > name
> > > >> > > > > > > > > > > > > > > - ValueToKey: the config in the third
> > > example
> > > >> > > ("when
> > > >> > > > > > field
> > > >> > > > > > > > > names
> > > >> > > > > > > > > > > > > include
> > > >> > > > > > > > > > > > > > > dots") should probably use
> > > "parent..child.k2"
> > > >> as
> > > >> > > the
> > > >> > > > > > > > > > > > > > > "transforms.smt1.fields" property
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > 3. RE changes to InsertField:
> > > >> > > > > > > > > > > > > > > - The InsertField SMT should also come
> > with
> > > >> the
> > > >> > new
> > > >> > > > > > > > > "field.style"
> > > >> > > > > > > > > > > > > > property
> > > >> > > > > > > > > > > > > > > in order to preserve backwards
> > > compatibility,
> > > >> > > right?
> > > >> > > > I
> > > >> > > > > > > don't
> > > >> > > > > > > > > see
> > > >> > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > included in the example configs for that
> > > one,
> > > >> > just
> > > >> > > > want
> > > >> > > > > > to
> > > >> > > > > > > > make
> > > >> > > > > > > > > > > sure
> > > >> > > > > > > > > > > > > > > - I don't know of any cases where we use
> > > >> > snake_case
> > > >> > > > for
> > > >> > > > > > > > > property
> > > >> > > > > > > > > > > > names
> > > >> > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > Kafka; we should probably use
> > > >> "on.missing.parent"
> > > >> > > and
> > > >> > > > > > > > > > > > > "on.existing.field"
> > > >> > > > > > > > > > > > > > > as the new property names for
> InsertField.
> > > >> > > > > > > > > > > > > > > - Why is the "on_existing_field" (or
> > > >> > > > > "on.existing.field")
> > > >> > > > > > > > > > property
> > > >> > > > > > > > > > > > only
> > > >> > > > > > > > > > > > > > > applied when the field style is nested?
> > > >> Couldn't
> > > >> > it
> > > >> > > > be
> > > >> > > > > > > useful
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Great points! I have applied these
> > suggestions
> > > >> to
> > > >> > the
> > > >> > > > > KIP.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Chris
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40 PM Jorge
> > > Esteban
> > > >> > > > Quilcate
> > > >> > > > > > > Otoya
> > > >> > > > > > > > <
> > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Again, great feedback Chris. Much
> > > >> appreciated.
> > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at 20:22, Chris
> > > Egerton <
> > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > >> > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Looking good! I have a few comments
> > left
> > > >> but
> > > >> > > all
> > > >> > > > > but
> > > >> > > > > > > one
> > > >> > > > > > > > or
> > > >> > > > > > > > > > two
> > > >> > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > minor.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 1. The motivation section states
> "This
> > > >> KIP is
> > > >> > > > aimed
> > > >> > > > > > to
> > > >> > > > > > > > > > include
> > > >> > > > > > > > > > > > > > support
> > > >> > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > nested structures on the existing
> > > SMTs...
> > > >> and
> > > >> > > to
> > > >> > > > > > > include
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > > > > abstractions
> > > >> > > > > > > > > > > > > > > > > to reuse this in future SMTs". A
> good
> > > >> > > > > implementation
> > > >> > > > > > of
> > > >> > > > > > > > > this
> > > >> > > > > > > > > > > KIP
> > > >> > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > > definitely isolate reusable logic
> > into a
> > > >> > > separate
> > > >> > > > > > > > > abstraction
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > easily pulled in to the SMTs we want
> > to
> > > >> add
> > > >> > > > nested
> > > >> > > > > > > field
> > > >> > > > > > > > > > > support
> > > >> > > > > > > > > > > > > to,
> > > >> > > > > > > > > > > > > > > but
> > > >> > > > > > > > > > > > > > > > > unless we plan on making this kind
> of
> > > >> > > abstraction
> > > >> > > > > > > > publicly
> > > >> > > > > > > > > > > > > available
> > > >> > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > some kind of utility method or class
> > > that
> > > >> > > > external
> > > >> > > > > > SMT
> > > >> > > > > > > > > > > developers
> > > >> > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > leverage, we can probably leave this
> > > part
> > > >> out
> > > >> > > as
> > > >> > > > > it's
> > > >> > > > > > > > more
> > > >> > > > > > > > > of
> > > >> > > > > > > > > > > an
> > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Make sense, will leave this out of the
> > > KIP.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a little
> > > >> misleading,
> > > >> > > isn't
> > > >> > > > > it?
> > > >> > > > > > > It
> > > >> > > > > > > > > > > > > demonstrates
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > escape syntax for fields with dot
> > > >> literals in
> > > >> > > > their
> > > >> > > > > > > > names,
> > > >> > > > > > > > > > but
> > > >> > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > doesn't
> > > >> > > > > > > > > > > > > > > > > demonstrate a way to actually use
> the
> > > Cast
> > > >> > (or
> > > >> > > > any
> > > >> > > > > > > other)
> > > >> > > > > > > > > SMT
> > > >> > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > access a
> > > >> > > > > > > > > > > > > > > > > nested field in a record, which is
> the
> > > >> whole
> > > >> > > > point
> > > >> > > > > of
> > > >> > > > > > > the
> > > >> > > > > > > > > > KIP.
> > > >> > > > > > > > > > > I
> > > >> > > > > > > > > > > > > like
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > example of escape syntax but we
> should
> > > >> > probably
> > > >> > > > > also
> > > >> > > > > > > add
> > > >> > > > > > > > > one
> > > >> > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > > > field access.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Agree. I have added examples to each
> SMT
> > > to
> > > >> be
> > > >> > > more
> > > >> > > > > > clear
> > > >> > > > > > > > > about
> > > >> > > > > > > > > > > how
> > > >> > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > affects each function
> > > >> > > > > > > > > > > > > > > > .
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 3. With the InsertField SMT, I'm
> > > wondering
> > > >> > what
> > > >> > > > the
> > > >> > > > > > > > > specific
> > > >> > > > > > > > > > > > > behavior
> > > >> > > > > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > > be when some or all of the "middle
> > > layer"
> > > >> > > nested
> > > >> > > > > > fields
> > > >> > > > > > > > are
> > > >> > > > > > > > > > > > > missing.
> > > >> > > > > > > > > > > > > > > For
> > > >> > > > > > > > > > > > > > > > > example, if I have a record with a
> > value
> > > >> of {
> > > >> > > > "k1":
> > > >> > > > > > > "v1 }
> > > >> > > > > > > > > > and I
> > > >> > > > > > > > > > > > > apply
> > > >> > > > > > > > > > > > > > > > > InsertField with topic.field =
> > > >> > n1.n2.n3.topic,
> > > >> > > > what
> > > >> > > > > > > will
> > > >> > > > > > > > > > > happen?
> > > >> > > > > > > > > > > > > Will
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > updated value become { "k1": "v1",
> > > "n1": {
> > > >> > > "n2":
> > > >> > > > {
> > > >> > > > > > > "n3":
> > > >> > > > > > > > > > > "topic"
> > > >> > > > > > > > > > > > > }}},
> > > >> > > > > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > > an exception be thrown, or something
> > > else?
> > > >> > This
> > > >> > > > > seems
> > > >> > > > > > > > > > analogous
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > command line mkdir command, which
> (at
> > > >> least
> > > >> > on
> > > >> > > > some
> > > >> > > > > > > > > operating
> > > >> > > > > > > > > > > > > > systems)
> > > >> > > > > > > > > > > > > > > > > fails by default if you try to
> create
> > a
> > > >> new
> > > >> > > > nested
> > > >> > > > > > > > > directory
> > > >> > > > > > > > > > > > where
> > > >> > > > > > > > > > > > > > > > anything
> > > >> > > > > > > > > > > > > > > > > but the last element in the path
> > doesn't
> > > >> > exist,
> > > >> > > > but
> > > >> > > > > > can
> > > >> > > > > > > > be
> > > >> > > > > > > > > > > > invoked
> > > >> > > > > > > > > > > > > > > with a
> > > >> > > > > > > > > > > > > > > > > flag that instructs it to go ahead
> and
> > > >> create
> > > >> > > all
> > > >> > > > > > > levels
> > > >> > > > > > > > of
> > > >> > > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > > > directory instead. I'm leaning on
> the
> > > >> side of
> > > >> > > > "just
> > > >> > > > > > > > create
> > > >> > > > > > > > > > > > > > everything"
> > > >> > > > > > > > > > > > > > > > but
> > > >> > > > > > > > > > > > > > > > > would be interested in your
> thoughts,
> > > and
> > > >> > > either
> > > >> > > > > way,
> > > >> > > > > > > we
> > > >> > > > > > > > > > should
> > > >> > > > > > > > > > > > > > > probably
> > > >> > > > > > > > > > > > > > > > > make sure the intended behavior is
> > > >> > well-defined
> > > >> > > > > > before
> > > >> > > > > > > > > > voting.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > This is an interesting case, thanks
> for
> > > >> > catching
> > > >> > > > > this!
> > > >> > > > > > > > > > > > > > > > The default behavior I see is to
> create
> > > >> parents
> > > >> > > if
> > > >> > > > > they
> > > >> > > > > > > are
> > > >> > > > > > > > > > > missing
> > > >> > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > overwrite fields
> > > >> > > > > > > > > > > > > > > > if they already exist.
> > > >> > > > > > > > > > > > > > > > I'm planning to include the following
> > two
> > > >> flags
> > > >> > > if
> > > >> > > > > > there
> > > >> > > > > > > > is a
> > > >> > > > > > > > > > > need
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> (CREATE|IGNORE),
> > > >> > > > > default=CREATE
> > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > (OVERWRITE|IGNORE),
> > > >> > > > > > > > default=OVERWRITE
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will the behavior
> > be
> > > if
> > > >> > any
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > > > field
> > > >> > > > > > > > > > > > > elements
> > > >> > > > > > > > > > > > > > > > > specified with InsertField already
> > exist
> > > >> in
> > > >> > the
> > > >> > > > > > record
> > > >> > > > > > > > > value?
> > > >> > > > > > > > > > > > Will
> > > >> > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > just
> > > >> > > > > > > > > > > > > > > > > overwrite them? What's the behavior
> of
> > > >> > > > InsertField
> > > >> > > > > > > today
> > > >> > > > > > > > > > under
> > > >> > > > > > > > > > > > > > similar
> > > >> > > > > > > > > > > > > > > > > circumstances?
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > The current behavior is to overwrite
> the
> > > >> value.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Chris
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at 4:15 PM
> Jorge
> > > >> Esteban
> > > >> > > > > > Quilcate
> > > >> > > > > > > > > Otoya
> > > >> > > > > > > > > > <
> > > >> > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much appreciated
> all
> > > the
> > > >> > > > feedback
> > > >> > > > > > > here.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > 1. You nailed it setting the
> design
> > > goal
> > > >> > > here:
> > > >> > > > > "it
> > > >> > > > > > > > > > shouldn't
> > > >> > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > impossible
> > > >> > > > > > > > > > > > > > > > > > to use this new feature for any
> > field
> > > >> name,
> > > >> > > no
> > > >> > > > > > matter
> > > >> > > > > > > > how
> > > >> > > > > > > > > > > > > > convoluted.
> > > >> > > > > > > > > > > > > > > > > It's
> > > >> > > > > > > > > > > > > > > > > > fine if edge cases introduce
> > > difficulty
> > > >> > (such
> > > >> > > > as
> > > >> > > > > > > > > > > less-readable
> > > >> > > > > > > > > > > > > > > > > > configurations), but it's not fine
> > if
> > > >> they
> > > >> > > > can't
> > > >> > > > > be
> > > >> > > > > > > > > > addressed
> > > >> > > > > > > > > > > > at
> > > >> > > > > > > > > > > > > > > all."
> > > >> > > > > > > > > > > > > > > > > > Back to the previous proposals
> > (using
> > > >> only
> > > >> > > dots
> > > >> > > > > as
> > > >> > > > > > > > > > > separators)
> > > >> > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > have
> > > >> > > > > > > > > > > > > > > > 2
> > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > >> > > > > > > > > > > > > > > > > > 1. escaping with backslashes
> > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots itself
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > I'll lean for alternative 2, as
> you
> > > >> > proposed
> > > >> > > > > > before.
> > > >> > > > > > > > > Feels
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > me
> > > >> > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > backslashes have more potential to
> > > lead
> > > >> to
> > > >> > > > > > confusion
> > > >> > > > > > > in
> > > >> > > > > > > > > > JSON
> > > >> > > > > > > > > > > > > > configs,
> > > >> > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > CSV seems like a good precedent to
> > use
> > > >> the
> > > >> > > same
> > > >> > > > > > > > character
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > > escape
> > > >> > > > > > > > > > > > > > > > > itself.
> > > >> > > > > > > > > > > > > > > > > > KIP is updated to reflect this.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an example,
> and
> > > >> stick
> > > >> > > with
> > > >> > > > > the
> > > >> > > > > > > > > current
> > > >> > > > > > > > > > > > > approach
> > > >> > > > > > > > > > > > > > > > > > defining the style per individual
> > > >> transform
> > > >> > > > > > > > > configuration.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP updated.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP updated.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at 21:59,
> Chris
> > > >> > Egerton <
> > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Thanks for addressing my
> comments;
> > > the
> > > >> > KIP
> > > >> > > > > looks
> > > >> > > > > > > > > > up-to-date
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > pretty
> > > >> > > > > > > > > > > > > > > > > > > readable now, and the rejected
> > > >> > alternatives
> > > >> > > > > > section
> > > >> > > > > > > > > does
> > > >> > > > > > > > > > a
> > > >> > > > > > > > > > > > > great
> > > >> > > > > > > > > > > > > > > job
> > > >> > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > outlining the discussion so far
> > and
> > > >> > > providing
> > > >> > > > > > > context
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > > > anyone
> > > >> > > > > > > > > > > > > > > else
> > > >> > > > > > > > > > > > > > > > > who
> > > >> > > > > > > > > > > > > > > > > > > might want to join in.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on choice of
> > delimiter:
> > > >> > > > > > > > > > > > > > > > > > > - I like the optimization for
> > simple
> > > >> > cases,
> > > >> > > > > but I
> > > >> > > > > > > > think
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > proposal
> > > >> > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > a little too restrictive. What
> if
> > > >> > there's a
> > > >> > > > > field
> > > >> > > > > > > > whose
> > > >> > > > > > > > > > > name
> > > >> > > > > > > > > > > > > > > contains
> > > >> > > > > > > > > > > > > > > > > all
> > > >> > > > > > > > > > > > > > > > > > > of the permitted options
> > (currently
> > > >> just
> > > >> > > ".",
> > > >> > > > > > ",",
> > > >> > > > > > > > and
> > > >> > > > > > > > > > > "/")?
> > > >> > > > > > > > > > > > > > > > > > > - If we expand the set of
> > permitted
> > > >> > > > delimiters
> > > >> > > > > to
> > > >> > > > > > > > allow
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > > > any
> > > >> > > > > > > > > > > > > > > > > > > single-character string,
> > > configuration
> > > >> > > > > complexity
> > > >> > > > > > > > will
> > > >> > > > > > > > > > > > increase
> > > >> > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > readability may decrease
> > > >> > > > > > > > > > > > > > > > > > > - Also worth pointing out that
> > there
> > > >> is
> > > >> > > some
> > > >> > > > > > > > convention
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > doubling
> > > >> > > > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > > delimiter character as an escape
> > > >> > mechanism
> > > >> > > > with
> > > >> > > > > > > > formats
> > > >> > > > > > > > > > > like
> > > >> > > > > > > > > > > > > CSV
> > > >> > > > > > > > > > > > > > > [1]
> > > >> > > > > > > > > > > > > > > > > > > - Overall I think we may be
> > > >> approaching
> > > >> > the
> > > >> > > > > > > > saturation
> > > >> > > > > > > > > > > point
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > productive
> > > >> > > > > > > > > > > > > > > > > > > discussion on delimiter syntax
> so
> > I
> > > >> don't
> > > >> > > > want
> > > >> > > > > to
> > > >> > > > > > > > spend
> > > >> > > > > > > > > > too
> > > >> > > > > > > > > > > > > much
> > > >> > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > your time on it. I think the one
> > > point
> > > >> > I'd
> > > >> > > > like
> > > >> > > > > > to
> > > >> > > > > > > > > leave
> > > >> > > > > > > > > > > for
> > > >> > > > > > > > > > > > > now
> > > >> > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > shouldn't be impossible to use
> > this
> > > >> new
> > > >> > > > feature
> > > >> > > > > > for
> > > >> > > > > > > > any
> > > >> > > > > > > > > > > field
> > > >> > > > > > > > > > > > > > name,
> > > >> > > > > > > > > > > > > > > > no
> > > >> > > > > > > > > > > > > > > > > > > matter how convoluted. It's fine
> > if
> > > >> edge
> > > >> > > > cases
> > > >> > > > > > > > > introduce
> > > >> > > > > > > > > > > > > > difficulty
> > > >> > > > > > > > > > > > > > > > > (such
> > > >> > > > > > > > > > > > > > > > > > > as less-readable
> configurations),
> > > but
> > > >> > it's
> > > >> > > > not
> > > >> > > > > > fine
> > > >> > > > > > > > if
> > > >> > > > > > > > > > they
> > > >> > > > > > > > > > > > > can't
> > > >> > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > 2.
> > > >> > > > > > > > > > > > > > > > > > > The configuration style where
> you
> > > >> define
> > > >> > > > > > > > > > > > > "transforms.field.style"
> > > >> > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > connector config, and then this
> > > >> applies
> > > >> > to
> > > >> > > > all
> > > >> > > > > > SMTs
> > > >> > > > > > > > for
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > connector,
> > > >> > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > very interesting. However, it
> > > doesn't
> > > >> > > follow
> > > >> > > > > > > > convention
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > existing
> > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > >> > > > > > > > > > > > > > > > > > > Right now, if you want to
> > configure
> > > an
> > > >> > SMT,
> > > >> > > > you
> > > >> > > > > > > > define
> > > >> > > > > > > > > > its
> > > >> > > > > > > > > > > > name
> > > >> > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > connector config (for example,
> > > >> > > "transforms":
> > > >> > > > > > > "smt1"),
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > > then
> > > >> > > > > > > > > > > > > > > define
> > > >> > > > > > > > > > > > > > > > > all
> > > >> > > > > > > > > > > > > > > > > > > of the properties for that SMT
> in
> > > the
> > > >> > > > connector
> > > >> > > > > > > > config
> > > >> > > > > > > > > > > using
> > > >> > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > namespacing
> > > >> > > > > > > > > > > > > > > > > > > mechanism specific to that SMT
> > (for
> > > >> > > example,
> > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT then sees only
> > the
> > > >> > > > properties
> > > >> > > > > > > > defined
> > > >> > > > > > > > > > in
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > namespace,
> > > >> > > > > > > > > > > > > > > > > > > with the prefix stripped (for
> > > example,
> > > >> > > > "prop1":
> > > >> > > > > > > > "val1")
> > > >> > > > > > > > > > in
> > > >> > > > > > > > > > > > its
> > > >> > > > > > > > > > > > > > > > > configure
> > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > >> > > > > > > > > > > > > > > > > > > If we want to continue to follow
> > > this
> > > >> > > > > convention,
> > > >> > > > > > > > then
> > > >> > > > > > > > > > > > instead
> > > >> > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > specifying
> > "transforms.field.style"
> > > >> in a
> > > >> > > > > > connector
> > > >> > > > > > > > > > config,
> > > >> > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > > > expect
> > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > >> > > > > > "transforms.<name>.field.style",
> > > >> > > > > > > > for
> > > >> > > > > > > > > > > each
> > > >> > > > > > > > > > > > > SMT
> > > >> > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > they
> > > >> > > > > > > > > > > > > > > > > > > want to configure a field style
> > for.
> > > >> This
> > > >> > > > would
> > > >> > > > > > > > require
> > > >> > > > > > > > > > > more
> > > >> > > > > > > > > > > > > work
> > > >> > > > > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > part of the user, but would be
> > > >> simpler to
> > > >> > > > > reason
> > > >> > > > > > > > about
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > > > easier
> > > >> > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > implement.
> > > >> > > > > > > > > > > > > > > > > > > If we want to explore an
> > alternative
> > > >> > where
> > > >> > > > > users
> > > >> > > > > > > can
> > > >> > > > > > > > > > > specify
> > > >> > > > > > > > > > > > > > global
> > > >> > > > > > > > > > > > > > > > > > > properties that apply to all
> > > >> transforms
> > > >> > in
> > > >> > > a
> > > >> > > > > > > > connector
> > > >> > > > > > > > > > > > config,
> > > >> > > > > > > > > > > > > > then
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > semantics for this need to be
> > > defined
> > > >> in
> > > >> > > the
> > > >> > > > > KIP.
> > > >> > > > > > > > This
> > > >> > > > > > > > > > > would
> > > >> > > > > > > > > > > > > have
> > > >> > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > include whether this will apply
> > only
> > > >> for
> > > >> > > the
> > > >> > > > > > > special
> > > >> > > > > > > > > case
> > > >> > > > > > > > > > > of
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > "field.style" and possibly
> > > >> > > "field.separator"
> > > >> > > > > > > > properties
> > > >> > > > > > > > > > or
> > > >> > > > > > > > > > > if
> > > >> > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > > available more generally for
> other
> > > >> > > > properties,
> > > >> > > > > > > > whether
> > > >> > > > > > > > > it
> > > >> > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > apply
> > > >> > > > > > > > > > > > > > > > > only
> > > >> > > > > > > > > > > > > > > > > > > for the SMTs outlined in the KIP
> > or
> > > if
> > > >> > the
> > > >> > > > > > > > > "field.style"
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > possibly
> > > >> > > > > > > > > > > > > > > > > > > "field.separator" properties
> would
> > > >> also
> > > >> > be
> > > >> > > > > passed
> > > >> > > > > > > > into
> > > >> > > > > > > > > > > custom
> > > >> > > > > > > > > > > > > > SMTs
> > > >> > > > > > > > > > > > > > > so
> > > >> > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > they could choose to act on them
> > if
> > > >> > > > applicable,
> > > >> > > > > > how
> > > >> > > > > > > > > edge
> > > >> > > > > > > > > > > > cases
> > > >> > > > > > > > > > > > > > like
> > > >> > > > > > > > > > > > > > > > > > having
> > > >> > > > > > > > > > > > > > > > > > > an SMT named "field" in your
> > > connector
> > > >> > > config
> > > >> > > > > > would
> > > >> > > > > > > > be
> > > >> > > > > > > > > > > > handled,
> > > >> > > > > > > > > > > > > > > etc.
> > > >> > > > > > > > > > > > > > > > > > > Either way, it might help to
> have
> > an
> > > >> > > example
> > > >> > > > in
> > > >> > > > > > the
> > > >> > > > > > > > KIP
> > > >> > > > > > > > > > > > > outlining
> > > >> > > > > > > > > > > > > > > how
> > > >> > > > > > > > > > > > > > > > > one
> > > >> > > > > > > > > > > > > > > > > > > of the to-be-augmented SMTs can
> be
> > > >> > > configured
> > > >> > > > > > with
> > > >> > > > > > > > this
> > > >> > > > > > > > > > new
> > > >> > > > > > > > > > > > > > feature
> > > >> > > > > > > > > > > > > > > > > and a
> > > >> > > > > > > > > > > > > > > > > > > before/after of how a record
> value
> > > >> would
> > > >> > be
> > > >> > > > > > > > transformed
> > > >> > > > > > > > > > > with
> > > >> > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for the
> > > >> > > > > "transforms.field.style"
> > > >> > > > > > > > > > property
> > > >> > > > > > > > > > > > > > mentions
> > > >> > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > the permitted values are "plain"
> > and
> > > >> > > > "nested",
> > > >> > > > > > but
> > > >> > > > > > > > then
> > > >> > > > > > > > > > > > > describes
> > > >> > > > > > > > > > > > > > > > > > behavior
> > > >> > > > > > > > > > > > > > > > > > > with a value of "root". Should
> > that
> > > be
> > > >> > > > "plain"
> > > >> > > > > > > > instead?
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for the
> > > >> > > > > > > "transforms.field.separator"
> > > >> > > > > > > > > > > > property
> > > >> > > > > > > > > > > > > > > > > > exclusively
> > > >> > > > > > > > > > > > > > > > > > > mentions structs, but the
> feature
> > is
> > > >> > > intended
> > > >> > > > > to
> > > >> > > > > > > work
> > > >> > > > > > > > > > with
> > > >> > > > > > > > > > > > maps
> > > >> > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > well.
> > > >> > > > > > > > > > > > > > > > > > > Can we update it to reflect
> this?
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > References:
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > >> > https://stackoverflow.com/a/17808731
> > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Chris
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022 at 1:32 PM
> > > Jorge
> > > >> > > Esteban
> > > >> > > > > > > > Quilcate
> > > >> > > > > > > > > > > Otoya
> > > >> > > > > > > > > > > > <
> > > >> > > > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> wrote:
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> "this..field.child"
> > > >> could
> > > >> > be
> > > >> > > > > > harder
> > > >> > > > > > > to
> > > >> > > > > > > > > > grasp
> > > >> > > > > > > > > > > > > than
> > > >> > > > > > > > > > > > > > > > > > > > "this.field/child" +
> separator:
> > > "/".
> > > >> > > > > > > > > > > > > > > > > > > > Even though this represents
> > > >> additional
> > > >> > > > > > > information,
> > > >> > > > > > > > > it
> > > >> > > > > > > > > > > > > follows
> > > >> > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > similar
> > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > "Flatten#delimeter"
> > > >> > > > > > > configuration.
> > > >> > > > > > > > > > > > > > > > > > > > I want to give the separator
> > > >> approach
> > > >> > > > another
> > > >> > > > > > > try,
> > > >> > > > > > > > > so I
> > > >> > > > > > > > > > > > have
> > > >> > > > > > > > > > > > > > > > updated
> > > >> > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > KIP with the separator
> proposal,
> > > >> > sticking
> > > >> > > > to
> > > >> > > > > > > only 2
> > > >> > > > > > > > > > > > > > alternatives
> > > >> > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > should hopefully cover most
> > > >> scenarios.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has been updated
> > > with
> > > >> > this
> > > >> > > > > > > > improvement.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I have
> updated
> > > this
> > > >> > > > section
> > > >> > > > > > > > > > accordingly.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch! I've replaced
> it
> > > with
> > > >> > > > > > > `DropHeaders`.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Looking forward to your
> > feedback.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022 at 21:33,
> > Chris
> > > >> > > Egerton
> > > >> > > > <
> > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > Looking good! Got a few more
> > > >> > thoughts.
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to revisit this,
> but
> > I
> > > >> think
> > > >> > > we
> > > >> > > > > may
> > > >> > > > > > > want
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > adopt
> > > >> > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > slightly
> > > >> > > > > > > > > > > > > > > > > > > > > different escape syntax
> style.
> > > >> > > > Backslashes
> > > >> > > > > > are
> > > >> > > > > > > > > great,
> > > >> > > > > > > > > > > but
> > > >> > > > > > > > > > > > > > since
> > > >> > > > > > > > > > > > > > > > > > they're
> > > >> > > > > > > > > > > > > > > > > > > > > already used by JSON, using
> > them
> > > >> as
> > > >> > an
> > > >> > > > > escape
> > > >> > > > > > > > > > sequence
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > field
> > > >> > > > > > > > > > > > > > > > > > > notation
> > > >> > > > > > > > > > > > > > > > > > > > > would also lead to some
> pretty
> > > >> ugly
> > > >> > > > > connector
> > > >> > > > > > > > > > configs.
> > > >> > > > > > > > > > > > > Anyone
> > > >> > > > > > > > > > > > > > > > who's
> > > >> > > > > > > > > > > > > > > > > > had
> > > >> > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > write regular expressions
> with
> > > >> > > > backslashes
> > > >> > > > > in
> > > >> > > > > > > > Java
> > > >> > > > > > > > > is
> > > >> > > > > > > > > > > > > > probably
> > > >> > > > > > > > > > > > > > > > > > already
> > > >> > > > > > > > > > > > > > > > > > > > > familiar with this:
> > > >> > > > > > > > > > > > > > >
> > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > >> > > > > > > > > > > > > > > > > What
> > > >> > > > > > > > > > > > > > > > > > > do
> > > >> > > > > > > > > > > > > > > > > > > > > you think about using the
> dot
> > > >> > character
> > > >> > > > to
> > > >> > > > > > > escape
> > > >> > > > > > > > > > > itself?
> > > >> > > > > > > > > > > > > In
> > > >> > > > > > > > > > > > > > > > other
> > > >> > > > > > > > > > > > > > > > > > > words,
> > > >> > > > > > > > > > > > > > > > > > > > > to access a single field
> named
> > > >> > > > > "this.field",
> > > >> > > > > > > > > instead
> > > >> > > > > > > > > > of
> > > >> > > > > > > > > > > > > using
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > syntax
> > > >> > > > > > > > > > > > > > > > > > > > > "this\.field" (which in JSON
> > > would
> > > >> > have
> > > >> > > > to
> > > >> > > > > be
> > > >> > > > > > > > > > expressed
> > > >> > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > >> > > > > > > > > > > > > > > > > > > > > we could use "this..field",
> > and
> > > >> for a
> > > >> > > > > single
> > > >> > > > > > > > field
> > > >> > > > > > > > > > > named
> > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > >> > > > > > > > > > > > > > > > > > > > > instead of using the syntax
> > > >> > > "this\\field"
> > > >> > > > > > (or,
> > > >> > > > > > > in
> > > >> > > > > > > > > > JSON,
> > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > >> > > > > > > > > > > > > > > > > > > > > we could use "this\field"
> (or,
> > > in
> > > >> > JSON,
> > > >> > > > > > > > > > "this\\field").
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you flesh out the
> > > >> details on
> > > >> > > the
> > > >> > > > > new
> > > >> > > > > > > > > > > > "field.style"
> > > >> > > > > > > > > > > > > > > > > property,
> > > >> > > > > > > > > > > > > > > > > > > > > including the type, default
> > > value,
> > > >> > > > > > importance,
> > > >> > > > > > > > and
> > > >> > > > > > > > > a
> > > >> > > > > > > > > > > > > > > preliminary
> > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > >> > > > > > > > > > > > > > > > > > > > > See
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the "Compatibility,
> > > >> > Deprecation,
> > > >> > > > and
> > > >> > > > > > > > > Migration
> > > >> > > > > > > > > > > > Plan"
> > > >> > > > > > > > > > > > > > > > section
> > > >> > > > > > > > > > > > > > > > > > > still
> > > >> > > > > > > > > > > > > > > > > > > > > accurate after the latest
> > > update?
> > > >> > Seems
> > > >> > > > > like
> > > >> > > > > > > it's
> > > >> > > > > > > > > > still
> > > >> > > > > > > > > > > > > > written
> > > >> > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > assumption that nested field
> > > >> syntax
> > > >> > > will
> > > >> > > > be
> > > >> > > > > > > > > hardcoded
> > > >> > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > > opt-in,
> > > >> > > > > > > > > > > > > > > > > > which
> > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > >> > > > > > > > > > > > > > > > > > > > > isn't the case anymore.
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The "These SMTs do
> not
> > > >> > require
> > > >> > > > > nested
> > > >> > > > > > > > > > structure
> > > >> > > > > > > > > > > > > > > support"
> > > >> > > > > > > > > > > > > > > > > > > section
> > > >> > > > > > > > > > > > > > > > > > > > > mentions a "Drop" SMT. I
> think
> > > >> this
> > > >> > may
> > > >> > > > be
> > > >> > > > > > > > > referring
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > Confluent
> > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > >> > > > > > > > > > > > > > > > > > > > > SMT, which isn't a part of
> > > Apache
> > > >> > > Kafka.
> > > >> > > > > > Should
> > > >> > > > > > > > we
> > > >> > > > > > > > > > drop
> > > >> > > > > > > > > > > > > (heh)
> > > >> > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > SMT
> > > >> > > > > > > > > > > > > > > > > > > > from
> > > >> > > > > > > > > > > > > > > > > > > > > the list? Or perhaps just
> > > replace
> > > >> it
> > > >> > > with
> > > >> > > > > > > > > > > "DropHeaders",
> > > >> > > > > > > > > > > > > > which
> > > >> > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > currently
> > > >> > > > > > > > > > > > > > > > > > > > > missing from the list and
> > > >> shouldn't
> > > >> > > > require
> > > >> > > > > > any
> > > >> > > > > > > > > > > > > nested-field
> > > >> > > > > > > > > > > > > > > > > related
> > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28, 2022 at 2:12
> > PM
> > > >> Jorge
> > > >> > > > > Esteban
> > > >> > > > > > > > > > Quilcate
> > > >> > > > > > > > > > > > > Otoya
> > > >> > > > > > > > > > > > > > <
> > > >> > > > > > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Thank you, Chris! and
> sorry
> > > for
> > > >> the
> > > >> > > > > delayed
> > > >> > > > > > > > > > response.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Please, find my comments
> > > below:
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb 2022 at
> > 17:34,
> > > >> Chris
> > > >> > > > > Egerton
> > > >> > > > > > > > > > > > > > > > > > > > <chrise@confluent.io.invalid
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP! I'd
> > love
> > > >> to
> > > >> > see
> > > >> > > > > > support
> > > >> > > > > > > > for
> > > >> > > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > fields
> > > >> > > > > > > > > > > > > > > > > > added
> > > >> > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > out-of-the-box SMTs
> > provided
> > > >> with
> > > >> > > > > > Connect.
> > > >> > > > > > > > Here
> > > >> > > > > > > > > > are
> > > >> > > > > > > > > > > > my
> > > >> > > > > > > > > > > > > > > > initial
> > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree that there's
> a
> > > >> case to
> > > >> > > be
> > > >> > > > > made
> > > >> > > > > > > for
> > > >> > > > > > > > > > > > expanding
> > > >> > > > > > > > > > > > > > > > > > HoistField
> > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > >> > > > > > > > > > > > > > > > > > > > > > > new config property for
> > > >> > > identifying a
> > > >> > > > > > > nested,
> > > >> > > > > > > > > > > > > > to-be-hoisted
> > > >> > > > > > > > > > > > > > > > > > field,
> > > >> > > > > > > > > > > > > > > > > > > > but
> > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > example in the KIP
> doesn't
> > > >> really
> > > >> > > > > > > demonstrate
> > > >> > > > > > > > > why
> > > >> > > > > > > > > > > > this
> > > >> > > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd be helpful to
> > > >> expand
> > > >> > the
> > > >> > > > > > example
> > > >> > > > > > > > to
> > > >> > > > > > > > > > add
> > > >> > > > > > > > > > > > > other
> > > >> > > > > > > > > > > > > > > > fields
> > > >> > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > order
> > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > show how adding nested
> > field
> > > >> > > support
> > > >> > > > > > > enables
> > > >> > > > > > > > > > users
> > > >> > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > hoist a
> > > >> > > > > > > > > > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > >> > > > > > > > > > > > > > > > > > > > > > > without dropping other
> > > fields
> > > >> > from
> > > >> > > > the
> > > >> > > > > > > value.
> > > >> > > > > > > > > > Maybe
> > > >> > > > > > > > > > > > > > > something
> > > >> > > > > > > > > > > > > > > > > > like
> > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >     source = nested.val
> > > >> > > > > > > > > > > > > > > > > > > > > > >     field = line
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >     value (before):
> > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > >> > > > > > > > > > > > > > > > > > > > > > >             "nested": {
> > > >> > > > > > > > > > > > > > > > > > > > > > >                 "val":
> 42,
> > > >> > > > > > > > > > > > > > > > > > > > > > >                 "other
> > val":
> > > >> 96
> > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >     value (after):
> > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > >> > > > > > > > > > > > > > > > > > > > > > >             "nested": {
> > > >> > > > > > > > > > > > > > > > > > > > > > >                 "line":
> {
> > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "val":
> > > 42,
> > > >> > > > > > > > > > > > > > > > > > > > > > >                 }
> > > >> > > > > > > > > > > > > > > > > > > > > > >                 "other
> > val":
> > > >> 96
> > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I think "source"
> > is
> > > a
> > > >> > > little
> > > >> > > > > > > strange
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > HoistField
> > > >> > > > > > > > > > > > > > > > > > > > > > > property name. Maybe
> > > >> "hoisted" or
> > > >> > > > > > > > > "hoisted.field"
> > > >> > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and 2.:
> > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The example for
> this
> > > SMT
> > > >> is
> > > >> > > > > updated
> > > >> > > > > > > and
> > > >> > > > > > > > > have
> > > >> > > > > > > > > > > > added
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there a reasonable
> > use
> > > >> case
> > > >> > > for
> > > >> > > > > > > > expanding
> > > >> > > > > > > > > > > > Flatten
> > > >> > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > able
> > > >> > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > flatten specific fields?
> > My
> > > >> > > > > understanding
> > > >> > > > > > > is
> > > >> > > > > > > > > that
> > > >> > > > > > > > > > > > it's
> > > >> > > > > > > > > > > > > > > mostly
> > > >> > > > > > > > > > > > > > > > > > > useful
> > > >> > > > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > writing to systems like
> > > >> databases
> > > >> > > > that
> > > >> > > > > > > don't
> > > >> > > > > > > > > > > support
> > > >> > > > > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > > > > values
> > > >> > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > require everything to
> be a
> > > >> flat
> > > >> > > list
> > > >> > > > of
> > > >> > > > > > > > > key-value
> > > >> > > > > > > > > > > > > pairs.
> > > >> > > > > > > > > > > > > > > > Being
> > > >> > > > > > > > > > > > > > > > > > able
> > > >> > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a nested field
> > > >> wouldn't
> > > >> > > > provide
> > > >> > > > > > any
> > > >> > > > > > > > > > > advantage
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > use
> > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > >> > > > > > > > > > > > > > > > > > > > > > > Are there other cases
> > where
> > > it
> > > >> > > would?
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't think we
> should
> > > >> > > > > > unconditionally
> > > >> > > > > > > > > change
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > default
> > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten. It's a
> > > >> > > > backwards-incompatible,
> > > >> > > > > > > > > breaking
> > > >> > > > > > > > > > > > change
> > > >> > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > could
> > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > >> > > > > > > > > > > > > > > > > > > > > > > headaches for users. It
> > > might
> > > >> be
> > > >> > > > > > reasonable
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > change
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > default
> > > >> > > > > > > > > > > > > > > > > > > > value
> > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically based on
> > whether
> > > >> the
> > > >> > > user
> > > >> > > > > has
> > > >> > > > > > > > > > > specified a
> > > >> > > > > > > > > > > > > > value
> > > >> > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > >> > > > > > > > > > > > > > > > > > > > > > > property, but
> considering
> > > the
> > > >> > > > > motivation
> > > >> > > > > > > for
> > > >> > > > > > > > > > > changing
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > default
> > > >> > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > > > > it creates conflicts
> with
> > > the
> > > >> > > > > > > > to-be-introduced
> > > >> > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > field
> > > >> > > > > > > > > > > > > > > > > > syntax
> > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > >> > > > > > > > > > > > > > > > > > > > > > > could arise with
> > downstream
> > > >> SMTs
> > > >> > > > > > regardless
> > > >> > > > > > > > of
> > > >> > > > > > > > > > > > whether
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > user
> > > >> > > > > > > > > > > > > > > > > > has
> > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly configured
> > > Flatten
> > > >> > with
> > > >> > > > the
> > > >> > > > > > > > "field"
> > > >> > > > > > > > > > > > > > property), I
> > > >> > > > > > > > > > > > > > > > > don't
> > > >> > > > > > > > > > > > > > > > > > > > know
> > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > > > > this would be too useful
> > > >> either.
> > > >> > I
> > > >> > > > have
> > > >> > > > > > > some
> > > >> > > > > > > > > > > thoughts
> > > >> > > > > > > > > > > > > > below
> > > >> > > > > > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > > > > > > how
> > > >> > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > handle possible
> conflicts
> > > >> between
> > > >> > > > names
> > > >> > > > > > > with
> > > >> > > > > > > > > dots
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > their
> > > >> > > > > > > > > > > > > > > > > names
> > > >> > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for nested field
> > > >> > references
> > > >> > > > that
> > > >> > > > > > > > should
> > > >> > > > > > > > > > > > > hopefully
> > > >> > > > > > > > > > > > > > > make
> > > >> > > > > > > > > > > > > > > > > > > either
> > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough. With the
> > support
> > > >> for
> > > >> > > > nested
> > > >> > > > > > > fields
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > > other
> > > >> > > > > > > > > > > > > > > SMTs,
> > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > >> > > > > > > > > > > > > > > > > > > > > > could stay as it is.
> > > >> > > > > > > > > > > > > > > > > > > > > > This removes the need for
> > (4)
> > > >> > > changing
> > > >> > > > > > > Flatten
> > > >> > > > > > > > > > config
> > > >> > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > well.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think it's fine to
> > > expand
> > > >> > > > > > ExtractField
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > > support
> > > >> > > > > > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > >> > > > > > > > > > > > > > > > > > > > > > > it might be worth noting
> > in
> > > >> the
> > > >> > > > > rejected
> > > >> > > > > > > > > > > alternatives
> > > >> > > > > > > > > > > > > > > section
> > > >> > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > this
> > > >> > > > > > > > > > > > > > > > > > > > > > > isn't strictly necessary
> > > since
> > > >> > you
> > > >> > > > can
> > > >> > > > > > > > replace
> > > >> > > > > > > > > > any
> > > >> > > > > > > > > > > > > single
> > > >> > > > > > > > > > > > > > > > > > > invocation
> > > >> > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT that uses
> nested
> > > >> field
> > > >> > > > > notation
> > > >> > > > > > > with
> > > >> > > > > > > > > > > > multiple
> > > >> > > > > > > > > > > > > > > > > > invocations
> > > >> > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > > > > > that use non-nested
> > > notation.
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding it.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit: "RegerRouter"
> > should
> > > >> be
> > > >> > > > > > > "RegexRouter"
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > list
> > > >> > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > SMTs
> > > >> > > > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > > do
> > > >> > > > > > > > > > > > > > > > > > > > > > > not require nested
> > structure
> > > >> > > support.
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing it.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be rare for
> dots
> > > in
> > > >> > field
> > > >> > > > > names
> > > >> > > > > > > to
> > > >> > > > > > > > > > occur
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > wild
> > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be so certain
> of
> > > >> this),
> > > >> > > but
> > > >> > > > > > unless
> > > >> > > > > > > > we
> > > >> > > > > > > > > > want
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > inflict
> > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > > > > > > > > > > > users of Flatten, I
> think
> > > >> we're
> > > >> > > going
> > > >> > > > > to
> > > >> > > > > > > have
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > think
> > > >> > > > > > > > > > > > > > > about
> > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > >> > > > > > > > > > > > > > > > > > > > > > > between dotted notation
> > and
> > > >> > > > non-nested
> > > >> > > > > > > fields
> > > >> > > > > > > > > > whose
> > > >> > > > > > > > > > > > > names
> > > >> > > > > > > > > > > > > > > > > contain
> > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > >> > > > > > > > > > > > > > > > > > > > > > > don't think this is
> > actually
> > > >> > such a
> > > >> > > > bad
> > > >> > > > > > > > thing,
> > > >> > > > > > > > > > > > though.
> > > >> > > > > > > > > > > > > I
> > > >> > > > > > > > > > > > > > > > agree
> > > >> > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > >> > > > > > > > > > > > > > > > > > > > > > > notation is intuitive
> and
> > > >> pretty
> > > >> > > > > > > commonplace
> > > >> > > > > > > > > (in
> > > >> > > > > > > > > > > > tools
> > > >> > > > > > > > > > > > > > like
> > > >> > > > > > > > > > > > > > > > jq,
> > > >> > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > example), so I'd like it
> > if
> > > we
> > > >> > > could
> > > >> > > > > > stick
> > > >> > > > > > > to
> > > >> > > > > > > > > it.
> > > >> > > > > > > > > > > > What
> > > >> > > > > > > > > > > > > > > about
> > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > >> > > > > > > > > > > > > > > > > > > > > > > escape syntax, using a
> > > >> backslash?
> > > >> > > > That
> > > >> > > > > > way,
> > > >> > > > > > > > > users
> > > >> > > > > > > > > > > > could
> > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > >> > > > > > > > > > > > > > > > > > > > > > > between "this.field"
> > (which
> > > >> would
> > > >> > > > refer
> > > >> > > > > > to
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > nested
> > > >> > > > > > > > > > > > > > field
> > > >> > > > > > > > > > > > > > > > > > "field"
> > > >> > > > > > > > > > > > > > > > > > > > > under
> > > >> > > > > > > > > > > > > > > > > > > > > > > the top-level "this"
> > field),
> > > >> and
> > > >> > > > > > > > "this\.field"
> > > >> > > > > > > > > > > (which
> > > >> > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > > refer
> > > >> > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> "this.field").
> > > >> Like
> > > >> > > with
> > > >> > > > > most
> > > >> > > > > > > > > > languages
> > > >> > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > use
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > >> > > > > > > > > > > > > > > > > > > > > > > for escape sequences, it
> > > could
> > > >> > also
> > > >> > > > be
> > > >> > > > > > used
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > escape
> > > >> > > > > > > > > > > > > > > itself,
> > > >> > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > >> > > > > > > > > > > > > > > > > > > > > > > that a field name
> > contains a
> > > >> > > > > backslash. I
> > > >> > > > > > > > think
> > > >> > > > > > > > > > > this
> > > >> > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > simpler than, e.g.,
> > adding a
> > > >> new
> > > >> > > > config
> > > >> > > > > > > > > property
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > > > toggle
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > be used when parsing
> > nested
> > > >> field
> > > >> > > > > > > references.
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > I like this approach.
> Adding
> > > to
> > > >> the
> > > >> > > > KIP.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't think we can
> > > >> > > > unconditionally
> > > >> > > > > > > turn
> > > >> > > > > > > > > this
> > > >> > > > > > > > > > > > > feature
> > > >> > > > > > > > > > > > > > > on.
> > > >> > > > > > > > > > > > > > > > > The
> > > >> > > > > > > > > > > > > > > > > > > > risk
> > > >> > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > breaking existing
> > pipelines
> > > >> > > > (especially
> > > >> > > > > > > ones
> > > >> > > > > > > > > that
> > > >> > > > > > > > > > > > > > involve,
> > > >> > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > >> > > > > > > > > > > > > > > > > > > > > > > combination of the
> Flatten
> > > and
> > > >> > Cast
> > > >> > > > > SMTs)
> > > >> > > > > > > is
> > > >> > > > > > > > > > pretty
> > > >> > > > > > > > > > > > > > high. I
> > > >> > > > > > > > > > > > > > > > > think
> > > >> > > > > > > > > > > > > > > > > > > > this
> > > >> > > > > > > > > > > > > > > > > > > > > > > should be an opt-in
> > feature,
> > > >> at
> > > >> > > least
> > > >> > > > > > until
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > next
> > > >> > > > > > > > > > > > > > major
> > > >> > > > > > > > > > > > > > > > > > release.
> > > >> > > > > > > > > > > > > > > > > > > > One
> > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > >> > > > > > > > > > > > > > > > > > > > > > > we could accomplish this
> > is
> > > by
> > > >> > > > > > introducing
> > > >> > > > > > > a
> > > >> > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > "field.style"
> > > >> > > > > > > > > > > > > > > > > > > (name
> > > >> > > > > > > > > > > > > > > > > > > > > > > obviously subject to
> > change)
> > > >> > > property
> > > >> > > > > > with
> > > >> > > > > > > > > values
> > > >> > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > "plain"
> > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If set to
> > "plain"
> > > >> then
> > > >> > > the
> > > >> > > > > > > current
> > > >> > > > > > > > > > > > non-nested
> > > >> > > > > > > > > > > > > > > > > behavior
> > > >> > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > >> > > > > > > > > > > > > > > > > > > > > > > and if set to "nested",
> > then
> > > >> the
> > > >> > > > > proposed
> > > >> > > > > > > > > nested
> > > >> > > > > > > > > > > > > behavior
> > > >> > > > > > > > > > > > > > > is.
> > > >> > > > > > > > > > > > > > > > > We
> > > >> > > > > > > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > > > > > > > consider updating the
> > > default
> > > >> > value
> > > >> > > > to
> > > >> > > > > > > > "nested"
> > > >> > > > > > > > > > in
> > > >> > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > future
> > > >> > > > > > > > > > > > > > > > > major
> > > >> > > > > > > > > > > > > > > > > > > > > release
> > > >> > > > > > > > > > > > > > > > > > > > > > > (or even codify that
> plan
> > in
> > > >> this
> > > >> > > > KIP,
> > > >> > > > > if
> > > >> > > > > > > > > there's
> > > >> > > > > > > > > > > > > enough
> > > >> > > > > > > > > > > > > > > > > support
> > > >> > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > >> > > > > > > > > > > > > > > > > > > > > > > This would also leave
> the
> > > door
> > > >> > open
> > > >> > > > for
> > > >> > > > > > > > adding
> > > >> > > > > > > > > > > > > recursive
> > > >> > > > > > > > > > > > > > > > > support
> > > >> > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > the future by adding a
> > > >> permitted
> > > >> > > > value
> > > >> > > > > of
> > > >> > > > > > > > > > > > "recursive".
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of the linked
> > tickets
> > > >> in
> > > >> > the
> > > >> > > > > > > > > "Motivation"
> > > >> > > > > > > > > > > > > section,
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-10640
> ,
> > > >> > > > > > > > > > > > has
> > > >> > > > > > > > > > > > > > an
> > > >> > > > > > > > > > > > > > > > open
> > > >> > > > > > > > > > > > > > > > > > PR
> > > >> > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > >> > > > > > > > > > > > > > > > > > > > > > > that propose adding
> > > recursive
> > > >> > > support
> > > >> > > > > to
> > > >> > > > > > > some
> > > >> > > > > > > > > > SMTs.
> > > >> > > > > > > > > > > > > Have
> > > >> > > > > > > > > > > > > > > you
> > > >> > > > > > > > > > > > > > > > > > > > considered
> > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> functionality
> > > for
> > > >> > your
> > > >> > > > > KIP?
> > > >> > > > > > Or
> > > >> > > > > > > > is
> > > >> > > > > > > > > > your
> > > >> > > > > > > > > > > > aim
> > > >> > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > stick
> > > >> > > > > > > > > > > > > > > > > > > > solely
> > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > nested field support?
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > I like the `field.style`
> > > >> > > configuration
> > > >> > > > > flag
> > > >> > > > > > > > > > approach.
> > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for pointing out
> the
> > > >> > > `recursive`
> > > >> > > > > > > > approach.
> > > >> > > > > > > > > > > Will
> > > >> > > > > > > > > > > > > add
> > > >> > > > > > > > > > > > > > > > > `nested`
> > > >> > > > > > > > > > > > > > > > > > > at
> > > >> > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's check the
> > demand
> > > >> for
> > > >> > > > > > > `recursive`
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > > > consider
> > > >> > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > part
> > > >> > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > this or another KIP.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > I have added the following
> > on
> > > >> the
> > > >> > > KIP:
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs could extend
> > this
> > > >> > support
> > > >> > > > > for:
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive notation:
> name a
> > > >> field
> > > >> > > and
> > > >> > > > > > apply
> > > >> > > > > > > it
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > all
> > > >> > > > > > > > > > > > > > fields
> > > >> > > > > > > > > > > > > > > > > > across
> > > >> > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > schema matching that name.
> > > >> > > > > > > > > > > > > > > > > > > > > > - Access to arrays: Adding
> > > `[]`
> > > >> > > > notation
> > > >> > > > > to
> > > >> > > > > > > > > > represent
> > > >> > > > > > > > > > > > > > access
> > > >> > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > arrays
> > > >> > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs to fields
> > within
> > > >> an
> > > >> > > > array.
> > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 8, 2022 at
> > 1:23
> > > PM
> > > >> > > Jorge
> > > >> > > > > > > Esteban
> > > >> > > > > > > > > > > Quilcate
> > > >> > > > > > > > > > > > > > > Otoya <
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> quilcate.jorge@gmail.com>
> > > >> wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev team,
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> new
> > > >> > > discussion
> > > >> > > > > > thread
> > > >> > > > > > > > on
> > > >> > > > > > > > > > > Kafka
> > > >> > > > > > > > > > > > > > > Connect
> > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP is aimed to
> > > include
> > > >> > > > support
> > > >> > > > > > for
> > > >> > > > > > > > > nested
> > > >> > > > > > > > > > > > > > > structures
> > > >> > > > > > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs — where this make
> > > >> sense.
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to
> your
> > > >> > feedback.
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Robert Yokota <ra...@gmail.com>]

Thanks Jorge.

+1 (non-binding)

Robert

On Mon, Aug 15, 2022 at 5:50 AM Jorge Esteban Quilcate Otoya <
quilcate.jorge@gmail.com> wrote:

> Thanks Robert.
>
> KIP is now updated with the latest backtick proposal:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
>
> Looking forward to your feedback. Also, a reminder that the vote thread is
> still open.
>
> Thanks,
> Jorge.
>
> On Fri, 12 Aug 2022 at 19:11, Robert Yokota <ra...@gmail.com> wrote:
>
> > Also, a double backslash can be used to escape backslash.
> >
> > Robert
> >
> > On Fri, Aug 12, 2022 at 10:56 AM Jorge Esteban Quilcate Otoya <
> > quilcate.jorge@gmail.com> wrote:
> >
> > > Awesome, thanks Robert!
> > > Will work on updated the proposal around this. Also, @Chris Egerton
> > > <fe...@gmail.com> if you have any feedback about this, would
> be
> > > much appreciated.
> > >
> > > Cheers,
> > > Jorge.
> > >
> > > On Fri, 12 Aug 2022 at 18:45, Robert Yokota <ra...@gmail.com>
> wrote:
> > >
> > > > Hi Jorge,
> > > >
> > > > Yes, to escape a backtick I would recommend a backslash.
> > > >
> > > > For example, MySQL allows identifiers to be surrounded with single
> > > quotes,
> > > > double quotes, or backticks, and they use backslash to escape.
> > > >
> > > > Jsonata went with primarily backticks as they are less common in
> > strings
> > > > that appear in JSON, and it seems to have worked nicely for them.
> > > >
> > > > Regards,
> > > > Robert
> > > >
> > > > On Fri, Aug 12, 2022 at 10:01 AM Jorge Esteban Quilcate Otoya <
> > > > quilcate.jorge@gmail.com> wrote:
> > > >
> > > > > Thanks Robert! And sorry for the late reply.
> > > > >
> > > > > That's a great catch and will require the current proposal to be
> > > updated.
> > > > >
> > > > > Using back-ticks is a good proposal. Though, the challenge we got
> is
> > > that
> > > > > JSON allows really any character into attribute names, including
> > > > > back-ticks.
> > > > > I wonder what's the best approach to escape back-ticks then. Given
> > that
> > > > > those use-cases should be minimal we could fallback to backslash to
> > > > escape
> > > > > back-ticks?
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Many thanks,
> > > > > Jorge.
> > > > >
> > > > >
> > > > > On Wed, 20 Jul 2022 at 23:03, Robert Yokota <ra...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm late to this thread, but would like to comment on the double
> > > > > dot/double
> > > > > > asterisk syntax.
> > > > > >
> > > > > > Unfortunately double dot is often used in JSON Path as a
> descendant
> > > > > > selector, see
> > > https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html
> > > > > >
> > > > > > I think a better notation would be to use backticks to quote any
> > > names
> > > > > with
> > > > > > a dot, such as  root.`child.with.dot`.  See Jsonata syntax for
> > > example:
> > > > > > http://docs.jsonata.org/simple
> > > > > >
> > > > > > Robert
> > > > > >
> > > > > > On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
> > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks Chris! I have updated the KIP to include this fix.
> > > > > > >
> > > > > > > I will keep the array as a potential improvement at the moment,
> > and
> > > > out
> > > > > > of
> > > > > > > the scope of this KIP.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jorge.
> > > > > > >
> > > > > > > On Tue, 28 Jun 2022 at 23:19, Chris Egerton <
> > > fearthecellos@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jorge,
> > > > > > > >
> > > > > > > > Apologies for the long delay, had my own KIP-related work to
> > > focus
> > > > > on.
> > > > > > > >
> > > > > > > > I think it's fine to include array accesses but it's not a
> > > blocker.
> > > > > I'm
> > > > > > > +1
> > > > > > > > either way. On that front though, I think the MaskField
> section
> > > > might
> > > > > > > need
> > > > > > > > to be updated as it still mentions arrays and deep-scan?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya
> <
> > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hi there,
> > > > > > > > >
> > > > > > > > > I have update the KIP to the previous state voted,
> including
> > > the
> > > > > > > > > configuration change from `field.style` to
> > > > `field.syntax.version`.
> > > > > > > > >
> > > > > > > > > I'll bump the vote thread as well to see if there's
> agreement
> > > on
> > > > > > adding
> > > > > > > > > this feature to Connect.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Jorge.
> > > > > > > > >
> > > > > > > > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya
> <
> > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks, Chris. Your feedback is much appreciated!
> > > > > > > > > >
> > > > > > > > > > I see how the current proposal might be underestimating
> > some
> > > > edge
> > > > > > > > cases.
> > > > > > > > > > I'm happy to move the design for deep-scan and
> multi-values
> > > to
> > > > > > future
> > > > > > > > > > developments related with this KIP and reduce its scope,
> > > though
> > > > > > open
> > > > > > > > for
> > > > > > > > > > more feedback.
> > > > > > > > > >
> > > > > > > > > > Also, just to be sure, are you proposing also to not
> > include
> > > > > array
> > > > > > > > access
> > > > > > > > > > at this stage?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Jorge.
> > > > > > > > > >
> > > > > > > > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <
> > > > > > fearthecellos@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Jorge,
> > > > > > > > > >>
> > > > > > > > > >> I've done some more thinking and I hate to say it, but I
> > > think
> > > > > the
> > > > > > > > > syntax
> > > > > > > > > >> does need to be expanded. Right now it's clear what
> "a.b"
> > > > refers
> > > > > > to
> > > > > > > > and
> > > > > > > > > >> what "a..b" refers to, but what about "a...b"? Is that
> > > > referring
> > > > > > to
> > > > > > > > > >> subfield ".b" of field "a", or subfield "b" of field
> "a."?
> > > > This
> > > > > > gets
> > > > > > > > > even
> > > > > > > > > >> more complicated when thinking about fields whose names
> > are
> > > > > > > > exclusively
> > > > > > > > > >> made up of dots.
> > > > > > > > > >>
> > > > > > > > > >> I'm also a little hesitant to mix the cases of
> multi-value
> > > > paths
> > > > > > and
> > > > > > > > > deep
> > > > > > > > > >> scans. What if you only want to access one subfield deep
> > for
> > > > an
> > > > > > SMT,
> > > > > > > > > >> instead of recursing through all the children of a given
> > > > field?
> > > > > > It's
> > > > > > > > > akin
> > > > > > > > > >> to the distinction between * and ** with file globbing
> > > > patterns,
> > > > > > and
> > > > > > > > > there
> > > > > > > > > >> could be a substantial performance difference if you
> have
> > > > > > > > heavily-nested
> > > > > > > > > >> fields.
> > > > > > > > > >>
> > > > > > > > > >> Ultimately, I think that if the proposed
> > > > "field.syntax.version"
> > > > > > > > property
> > > > > > > > > >> sits well with people, it might be better to reduce the
> > > scope
> > > > of
> > > > > > the
> > > > > > > > KIP
> > > > > > > > > >> back to the original proposal and just focus on adding
> > > support
> > > > > for
> > > > > > > > > >> explicitly-specified nested values, with no multi-value
> > > paths
> > > > > > > > > whatsoever,
> > > > > > > > > >> knowing that we have an easy way to introduce new syntax
> > and
> > > > > > > features
> > > > > > > > in
> > > > > > > > > >> the future. (We could probably leave the "a...b" case
> for
> > > that
> > > > > > next
> > > > > > > > > >> version
> > > > > > > > > >> too.)
> > > > > > > > > >>
> > > > > > > > > >> I was a huge fan of this KIP before we started trying to
> > > > address
> > > > > > > more
> > > > > > > > > >> complex use cases, and although I don't want to write
> > those
> > > > > off, I
> > > > > > > > think
> > > > > > > > > >> we
> > > > > > > > > >> may have bitten off more than we can chew in time for
> the
> > > > 3.3.0
> > > > > > > > release
> > > > > > > > > >> and
> > > > > > > > > >> would hate to see this KIP get delayed as a result.
> > > > > > > > > >>
> > > > > > > > > >> I'd be really curious to hear from Joshua and Tom on
> this
> > > > front,
> > > > > > > > though.
> > > > > > > > > >> Is
> > > > > > > > > >> it acceptable to move more incrementally here and settle
> > on
> > > > the
> > > > > > > syntax
> > > > > > > > > >> version property as our means of introducing new
> features,
> > > or
> > > > is
> > > > > > it
> > > > > > > > > >> preferable to implement things monolithically and try to
> > get
> > > > > > > > everything
> > > > > > > > > >> (or
> > > > > > > > > >> at least, as much as possible) right the first time?
> > > > > > > > > >>
> > > > > > > > > >> Thanks again for your continued effort on this KIP!
> > > > > > > > > >>
> > > > > > > > > >> Cheers,
> > > > > > > > > >>
> > > > > > > > > >> Chris
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate
> > Otoya
> > > <
> > > > > > > > > >> quilcate.jorge@gmail.com> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Thanks, Chris!
> > > > > > > > > >> >
> > > > > > > > > >> > Please, find my comments below:
> > > > > > > > > >> >
> > > > > > > > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> > > > > > > fearthecellos@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi Jorge,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > > > > > > > >> > >
> > > > > > > > > >> > > 1. Under the "Accessing multiple values by
> deep-scan"
> > > > header
> > > > > > > it's
> > > > > > > > > >> stated
> > > > > > > > > >> > > that "If deep-scan is used, it must have only one
> > field
> > > > > after
> > > > > > > the
> > > > > > > > > >> > asterisk
> > > > > > > > > >> > > level.". However, in example 3 for the Cast SMT and
> > > other
> > > > > > > examples
> > > > > > > > > for
> > > > > > > > > >> > > other SMTs, the spec contains a field of
> "*.child.k2",
> > > > which
> > > > > > > > appears
> > > > > > > > > >> to
> > > > > > > > > >> > > have two fields after the asterisk level. I may be
> > > > > > > > misunderstanding
> > > > > > > > > >> the
> > > > > > > > > >> > > proposal, but it seems like the two contradict each
> > > other.
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks for catching this. I have clarified it by
> > removing
> > > > this
> > > > > > > > > >> restriction.
> > > > > > > > > >> > Also, have extended the deep-scan scenarios.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > >
> > > > > > > > > >> > > 2. I'm a little unclear on why we need the special
> > > > handling
> > > > > > for
> > > > > > > > > arrays
> > > > > > > > > >> > > where, for an array field "a", the field name "a"
> can
> > be
> > > > > > treated
> > > > > > > > as
> > > > > > > > > >> > either
> > > > > > > > > >> > > the array itself, or every element in the array. Is
> > > there
> > > > a
> > > > > > > reason
> > > > > > > > > we
> > > > > > > > > >> > can't
> > > > > > > > > >> > > use the field name "a.*" to handle the latter case,
> > and
> > > > "a"
> > > > > to
> > > > > > > > > handle
> > > > > > > > > >> the
> > > > > > > > > >> > > former?
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> > Agree, this is confusing. I like the `a.*` approach to
> > > > access
> > > > > > > array
> > > > > > > > > >> items.
> > > > > > > > > >> > I have added this to the proposal.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > >
> > > > > > > > > >> > > 3. How would a user specify that they'd like to
> > access a
> > > > > field
> > > > > > > > with
> > > > > > > > > >> the
> > > > > > > > > >> > > literal name "*"?
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> > Good one. I'm proposing an approach similar to how
> it's
> > > > > proposed
> > > > > > > to
> > > > > > > > > >> escape
> > > > > > > > > >> > dots, with a double-asterisk. Curious on your thoughts
> > > > around
> > > > > > > this.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > >
> > > > > > > > > >> > > 4. For the Cast SMT, do you think it might bite some
> > > > people
> > > > > if
> > > > > > > > > fields
> > > > > > > > > >> > that
> > > > > > > > > >> > > can't be cast correctly are silently ignored? I'm
> > > > imagining
> > > > > > the
> > > > > > > > case
> > > > > > > > > >> > where
> > > > > > > > > >> > > none of the fields in a multi-path expression can be
> > > cast
> > > > > > > > correctly
> > > > > > > > > >> and
> > > > > > > > > >> > it
> > > > > > > > > >> > > ends up eating half of someone's day to track down
> why
> > > > their
> > > > > > SMT
> > > > > > > > > isn't
> > > > > > > > > >> > > doing anything.
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> > If I understand correctly, this challenge could be
> > > relevant
> > > > > > across
> > > > > > > > > SMTs.
> > > > > > > > > >> > At the moment, most/all? SMTs just silently ignore.
> > > > > > > > > >> > Was thinking about adding a flag
> > `field.on.path.not.found`
> > > > to
> > > > > > > either
> > > > > > > > > >> ignore
> > > > > > > > > >> > or fail when no paths are found. What do your think?
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > >
> > > > > > > > > >> > > 5. For the ExtractField and ValueToKey SMTs, what
> > > happens
> > > > > if a
> > > > > > > > > >> deep-scan
> > > > > > > > > >> > > field name is used, but only one field is found? Is
> > the
> > > > > > > resulting
> > > > > > > > > >> field
> > > > > > > > > >> > > still an array, or is it just the single field that
> > was
> > > > > found?
> > > > > > > > (FWIW
> > > > > > > > > >> I'm
> > > > > > > > > >> > > leaning towards keeping it an array just to keep
> > schemas
> > > > > > > > consistent
> > > > > > > > > >> in a
> > > > > > > > > >> > > pipeline in case the number of fields found
> fluctuates
> > > > > across
> > > > > > > > > >> records.)
> > > > > > > > > >> > >
> > > > > > > > > >> > > Agree. Will clarify that an array is always produced
> > > even
> > > > > for
> > > > > > 1
> > > > > > > > or 0
> > > > > > > > > >> > fields are found.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that
> "As
> > > this
> > > > > SMT
> > > > > > > > > affects
> > > > > > > > > >> > only
> > > > > > > > > >> > > existing fields, additional configurations will not
> be
> > > > > > > required.".
> > > > > > > > > >> Given
> > > > > > > > > >> > > the new "field.syntax.version" property, this part
> > > should
> > > > > > > probably
> > > > > > > > > be
> > > > > > > > > >> > > removed.
> > > > > > > > > >> > >
> > > > > > > > > >> > Agree.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > >
> > > > > > > > > >> > > 7. Is recursive descent intentionally excluded? That
> > was
> > > > an
> > > > > > > > > important
> > > > > > > > > >> > part
> > > > > > > > > >> > > of Joshua's KIP and his feedback on this KIP; I
> think
> > > it's
> > > > > > worth
> > > > > > > > > >> pursuing
> > > > > > > > > >> > > if we can.
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> > My understanding from Joshua's feedback is that by
> > > including
> > > > > > > support
> > > > > > > > > for
> > > > > > > > > >> > deep-scan, we are already covering the recursive
> > > > > functionality.
> > > > > > > > > Though,
> > > > > > > > > >> I
> > > > > > > > > >> > may be missing something.
> > > > > > > > > >> >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > Cheers,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Chris
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban
> Quilcate
> > > > > Otoya <
> > > > > > > > > >> > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Thanks, Chris!
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > I have updated the KIP with the rejected
> > alternatives
> > > > > > updated.
> > > > > > > > > >> Also, I
> > > > > > > > > >> > > have
> > > > > > > > > >> > > > drafted the support for arrays and deep scans as
> > part
> > > of
> > > > > the
> > > > > > > > > >> proposed
> > > > > > > > > >> > > > notation to make it more complete, even though
> these
> > > can
> > > > > be
> > > > > > > > > >> implemented
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > multiple PRs.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Looking forward to your feedback.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Cheers,
> > > > > > > > > >> > > > Jorge.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > > > > > > > >> fearthecellos@gmail.com>
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hi Jorge,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > I really appreciate the effort you've made to
> > > simplify
> > > > > the
> > > > > > > > > syntax
> > > > > > > > > >> and
> > > > > > > > > >> > > > > feature set of a JSONPath-based approach as much
> > as
> > > > > > > possible.
> > > > > > > > > I'm
> > > > > > > > > >> > still
> > > > > > > > > >> > > > > hesitant to continue with it, though.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > 1. The syntax is much less friendly. Just
> compare
> > > > > > > > > >> "top.mid.bottom" to
> > > > > > > > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses
> > > > > JSONPath
> > > > > > or
> > > > > > > > > even
> > > > > > > > > >> > JSON,
> > > > > > > > > >> > > > and
> > > > > > > > > >> > > > > the learning curve for the former is going to be
> > > > > steeper.
> > > > > > > The
> > > > > > > > > >> > examples
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > > the new KIP draft you published demonstrate this
> > > > pretty
> > > > > > > well,
> > > > > > > > > and
> > > > > > > > > >> > this
> > > > > > > > > >> > > is
> > > > > > > > > >> > > > > without diving into the details of what escape
> > > syntax
> > > > > > would
> > > > > > > > look
> > > > > > > > > >> > like.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > 2. The three new major features that this syntax
> > > adds
> > > > > > (array
> > > > > > > > > >> > accesses,
> > > > > > > > > >> > > > deep
> > > > > > > > > >> > > > > scans, and multi-value paths) could all be added
> > > > pretty
> > > > > > > easily
> > > > > > > > > to
> > > > > > > > > >> the
> > > > > > > > > >> > > dot
> > > > > > > > > >> > > > > notation syntax without introducing brackets and
> > > > dollar
> > > > > > > signs.
> > > > > > > > > >> Array
> > > > > > > > > >> > > > > accesses can be described using the same syntax
> as
> > > > > > > struct/map
> > > > > > > > > >> field
> > > > > > > > > >> > > > access,
> > > > > > > > > >> > > > > deep scans can be described using '*', and
> > > multi-value
> > > > > > paths
> > > > > > > > can
> > > > > > > > > >> be
> > > > > > > > > >> > > > > described by referencing the name of a field
> > that's
> > > > > > expected
> > > > > > > > to
> > > > > > > > > >> have
> > > > > > > > > >> > > > > children. These are all top-of-the-head ideas
> and
> > > can
> > > > > > > probably
> > > > > > > > > be
> > > > > > > > > >> > > > refined,
> > > > > > > > > >> > > > > but hopefully they demonstrate that we can keep
> > the
> > > > > syntax
> > > > > > > > > simple
> > > > > > > > > >> > > without
> > > > > > > > > >> > > > > sacrificing features. Of course, the question of
> > > > leaving
> > > > > > > room
> > > > > > > > > for
> > > > > > > > > >> > > future
> > > > > > > > > >> > > > > features might arise... given that these are the
> > > > > > > > out-of-the-box
> > > > > > > > > >> SMTs
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > > are likely to be the first that many people
> > > encounter,
> > > > > I'd
> > > > > > > err
> > > > > > > > > on
> > > > > > > > > >> the
> > > > > > > > > >> > > > side
> > > > > > > > > >> > > > > of simplicity and a gentle learning curve; if
> > people
> > > > > need
> > > > > > to
> > > > > > > > get
> > > > > > > > > >> more
> > > > > > > > > >> > > out
> > > > > > > > > >> > > > > of transforms, the option to implement their own
> > is
> > > > > still
> > > > > > > > there.
> > > > > > > > > >> If
> > > > > > > > > >> > we
> > > > > > > > > >> > > > can
> > > > > > > > > >> > > > > address 95% of use cases with something easy to
> > use,
> > > > > it's
> > > > > > > not
> > > > > > > > > >> worth
> > > > > > > > > >> > > > making
> > > > > > > > > >> > > > > the feature harder for everyone to use just to
> > > > > accommodate
> > > > > > > the
> > > > > > > > > >> > > remaining
> > > > > > > > > >> > > > > 5%.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > 3. The advantage of leveraging an existing
> syntax
> > is
> > > > > > > twofold:
> > > > > > > > > >> users
> > > > > > > > > >> > who
> > > > > > > > > >> > > > are
> > > > > > > > > >> > > > > already familiar with that syntax don't need to
> > > learn
> > > > a
> > > > > > new
> > > > > > > > > >> syntax,
> > > > > > > > > >> > and
> > > > > > > > > >> > > > > maintainers of the syntax get to leverage
> existing
> > > > > > libraries
> > > > > > > > and
> > > > > > > > > >> > > > > documentation for the syntax. With the current
> > > > proposal,
> > > > > > > > reusing
> > > > > > > > > >> > > > libraries
> > > > > > > > > >> > > > > is off the table, which means that we're going
> to
> > > have
> > > > > to
> > > > > > > > parse
> > > > > > > > > >> this
> > > > > > > > > >> > > > > ourselves (including all the escape syntax edge
> > > > cases),
> > > > > > and
> > > > > > > we
> > > > > > > > > >> won't
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > able to automatically leverage new features
> added
> > to
> > > > > that
> > > > > > > > > syntax.
> > > > > > > > > >> And
> > > > > > > > > >> > > > given
> > > > > > > > > >> > > > > how stripped-down the syntax is in comparison to
> > > full
> > > > > > > > JSONPath,
> > > > > > > > > >> > there's
> > > > > > > > > >> > > > > still going to be a learning curve for users who
> > are
> > > > > > already
> > > > > > > > > >> familiar
> > > > > > > > > >> > > > with
> > > > > > > > > >> > > > > it, and we'll still have to document how
> Connect's
> > > > > variant
> > > > > > > of
> > > > > > > > > >> > JSONPath
> > > > > > > > > >> > > > > works either instead of or in addition to just
> > > linking
> > > > > to
> > > > > > a
> > > > > > > > > >> > > > well-maintained
> > > > > > > > > >> > > > > third-party docs site.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On the topic of "field.path" and "include.path"
> > vs.
> > > > > > > > > >> "field.style", I
> > > > > > > > > >> > > > > actually think that a single property per SMT is
> > > > cleaner
> > > > > > and
> > > > > > > > > >> simpler.
> > > > > > > > > >> > > > > Allowing users to mix and match styles within
> the
> > > same
> > > > > SMT
> > > > > > > > > config
> > > > > > > > > >> > seems
> > > > > > > > > >> > > > > like a recipe for confusion, and with a single
> > > > property
> > > > > > that
> > > > > > > > > >> dictates
> > > > > > > > > >> > > > field
> > > > > > > > > >> > > > > syntax behavior, we leave the door open to
> change
> > > the
> > > > > > > default
> > > > > > > > > at a
> > > > > > > > > >> > > later
> > > > > > > > > >> > > > > date. We could even fully remove support for
> plain
> > > > field
> > > > > > > > > notation
> > > > > > > > > >> at
> > > > > > > > > >> > > some
> > > > > > > > > >> > > > > point and still be able to retain the simple
> > > property
> > > > > > names
> > > > > > > of
> > > > > > > > > >> > "field",
> > > > > > > > > >> > > > > "include", etc. instead of forcing people to use
> > > > > > > "field.path"
> > > > > > > > > and
> > > > > > > > > >> the
> > > > > > > > > >> > > > like.
> > > > > > > > > >> > > > > That said, the term "field.style" and the
> > permitted
> > > > > values
> > > > > > > > might
> > > > > > > > > >> be a
> > > > > > > > > >> > > > > little ambiguous. One alternative, though a
> little
> > > > > > > > heavy-handed,
> > > > > > > > > >> is
> > > > > > > > > >> > to
> > > > > > > > > >> > > > > change it to "field.syntax.version" with
> permitted
> > > > > values
> > > > > > of
> > > > > > > > > "V1"
> > > > > > > > > >> > > > (default,
> > > > > > > > > >> > > > > equivalent to "field.style = plain") and "V2"
> > > > > (equivalent
> > > > > > to
> > > > > > > > > >> > > > "field.style =
> > > > > > > > > >> > > > > nested"). This would leave us room in the future
> > to
> > > > make
> > > > > > > > further
> > > > > > > > > >> > > changes
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > the syntax without having to come up with new
> > names,
> > > > > > > although
> > > > > > > > it
> > > > > > > > > >> does
> > > > > > > > > >> > > > > sacrifice a little bit in that the permitted
> > values
> > > > are
> > > > > no
> > > > > > > > > longer
> > > > > > > > > >> > > > > self-describing.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Cheers,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Chris
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban
> > > Quilcate
> > > > > > > Otoya <
> > > > > > > > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Thank you all for your feedback, and sorry for
> > the
> > > > > long
> > > > > > > wait
> > > > > > > > > >> for a
> > > > > > > > > >> > > > reply.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I would like to explore the idea of
> > > > > > > JSONPath-inspired/subset
> > > > > > > > > >> > > notation a
> > > > > > > > > >> > > > > bit
> > > > > > > > > >> > > > > > further:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > It will need to be a much-reduced version of
> > > > JSONPath:
> > > > > > > > > >> > > > > > - No full support for JsonPath therefore an
> > > > additional
> > > > > > > > > >> dependency.
> > > > > > > > > >> > > > > > - All paths must start with '$'
> > > > > > > > > >> > > > > > - No functions support or other operators
> > allowed.
> > > > > > > > > >> > > > > > - JsonPath dotted and square-bracket notations
> > can
> > > > be
> > > > > > > > > supported
> > > > > > > > > >> to
> > > > > > > > > >> > > > avoid
> > > > > > > > > >> > > > > > escaping dots or other characters: `$a.b.c`
> and
> > > > > > > > > `$['a.b']['c']`
> > > > > > > > > >> -
> > > > > > > > > >> > or
> > > > > > > > > >> > > we
> > > > > > > > > >> > > > > > could only support the second one as it's more
> > > > > complete.
> > > > > > > > > >> > > > > > - Add support for arrays with `[<integer>]`,
> > e.g.
> > > > > > > `$a.[1].b`
> > > > > > > > > >> > > > > > - Add support for multiple-value paths using
> > array
> > > > > > access
> > > > > > > > > >> `$a.*.b`
> > > > > > > > > >> > or
> > > > > > > > > >> > > > > deep
> > > > > > > > > >> > > > > > scan `$a..b`.
> > > > > > > > > >> > > > > >     - Some SMTs that will benefit from this:
> > > > > > `MaskField`,
> > > > > > > > > >> `Cast`,
> > > > > > > > > >> > > > > > `ReplaceField`.
> > > > > > > > > >> > > > > > - We could introduce a `path[s]` config under
> > the
> > > > > > current
> > > > > > > > > >> > > > configurations
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > apply this feature so no compatibility issues
> > are
> > > > > > > > introduced:
> > > > > > > > > >> e.g.
> > > > > > > > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > With these,  100, 101, 102, and 103 will be
> > > > > effectively
> > > > > > > > > solved.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > The main challenge that I see at the moment is
> > > that
> > > > by
> > > > > > > being
> > > > > > > > > >> JSON
> > > > > > > > > >> > > > > > path-like, there may be some edge cases that I
> > > can't
> > > > > > > foresee
> > > > > > > > > at
> > > > > > > > > >> the
> > > > > > > > > >> > > > > moment,
> > > > > > > > > >> > > > > > that could make this hard to implement, test,
> > and
> > > > > > maintain
> > > > > > > > in
> > > > > > > > > >> the
> > > > > > > > > >> > > long
> > > > > > > > > >> > > > > run.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I'll appreciate your feedback on how this
> > > > > JSONPath-based
> > > > > > > > > >> > alternative
> > > > > > > > > >> > > > > > compares to the dotted notation initially
> > > proposed,
> > > > > and
> > > > > > if
> > > > > > > > it
> > > > > > > > > >> > matches
> > > > > > > > > >> > > > > your
> > > > > > > > > >> > > > > > feedback.
> > > > > > > > > >> > > > > > I have drafted a copy of the KIP to change the
> > > > > examples
> > > > > > to
> > > > > > > > > >> JSONPath
> > > > > > > > > >> > > > > > approach and validate some differences:
> > > > > > > > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > About 104. I agreed with Chris. We can handle
> > this
> > > > as
> > > > > > part
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > >> new
> > > > > > > > > >> > > > KIP.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > >> > > > > > Jorge.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > > > > > > > >> > fearthecellos@gmail.com
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Hi Joshua,
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > I have a few reservations about using
> JsonPath
> > > > > > notation
> > > > > > > > > here.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 1. There's likely to be a substantial
> > > performance
> > > > > > > penalty
> > > > > > > > > for
> > > > > > > > > >> > > > > converting
> > > > > > > > > >> > > > > > > between the Kafka Connect format and
> something
> > > > that
> > > > > a
> > > > > > > > > JsonPath
> > > > > > > > > >> > > > library
> > > > > > > > > >> > > > > > > would understand.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 2. The complexity of the feature will be
> > > > > significantly
> > > > > > > > > >> higher. It
> > > > > > > > > >> > > > will
> > > > > > > > > >> > > > > be
> > > > > > > > > >> > > > > > > harder to test, document, and implement.
> There
> > > > will
> > > > > be
> > > > > > > > many
> > > > > > > > > >> more
> > > > > > > > > >> > > edge
> > > > > > > > > >> > > > > > cases
> > > > > > > > > >> > > > > > > to consider and support, and we'll be on the
> > > hook
> > > > to
> > > > > > > > handle
> > > > > > > > > >> any
> > > > > > > > > >> > > bugs
> > > > > > > > > >> > > > or
> > > > > > > > > >> > > > > > > inconsistencies that arise, either as a
> result
> > > of
> > > > > our
> > > > > > > use
> > > > > > > > of
> > > > > > > > > >> the
> > > > > > > > > >> > > > > JsonPath
> > > > > > > > > >> > > > > > > library we choose, or as a result of bugs in
> > > that
> > > > > > > library.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 3. It's not clear that JsonPath is superior
> or
> > > > even
> > > > > > > > suitable
> > > > > > > > > >> for
> > > > > > > > > >> > > some
> > > > > > > > > >> > > > > of
> > > > > > > > > >> > > > > > > the SMTs proposed here. What would be the
> > > > advantages
> > > > > > of
> > > > > > > > > >> JsonPath
> > > > > > > > > >> > > with
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > InsertField or HoistField SMTs?
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > I also don't think that adding dot notation
> is
> > > > > > > unfriendly
> > > > > > > > to
> > > > > > > > > >> > users;
> > > > > > > > > >> > > > > many
> > > > > > > > > >> > > > > > > have proposed this type of syntax in the
> past,
> > > and
> > > > > > it's
> > > > > > > > > >> > frequently
> > > > > > > > > >> > > > used
> > > > > > > > > >> > > > > > in
> > > > > > > > > >> > > > > > > informal discussions to refer to nested
> > fields.
> > > If
> > > > > the
> > > > > > > > > >> proposed
> > > > > > > > > >> > > > syntax
> > > > > > > > > >> > > > > > was
> > > > > > > > > >> > > > > > > not already intuitive then a case against
> > > deciding
> > > > > on
> > > > > > > our
> > > > > > > > > own
> > > > > > > > > >> > might
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > more
> > > > > > > > > >> > > > > > > convincing, but as things stand, simple dot
> > > > notation
> > > > > > is
> > > > > > > > > likely
> > > > > > > > > >> > > going
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > be
> > > > > > > > > >> > > > > > > easier for users to understand than JsonPath
> > > > syntax.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Cheers,
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Chris
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua
> > Grisham <
> > > > > > > > > >> > > > grishamjosh@gmail.com>
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > Hello all!
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Sorry that I come a bit later to the party
> > > here,
> > > > > > but I
> > > > > > > > am
> > > > > > > > > >> the
> > > > > > > > > >> > one
> > > > > > > > > >> > > > who
> > > > > > > > > >> > > > > > > wrote
> > > > > > > > > >> > > > > > > > KIP-683 [1] for recursive support (just
> > simply
> > > > > > looping
> > > > > > > > > >> through
> > > > > > > > > >> > > all
> > > > > > > > > >> > > > > > child
> > > > > > > > > >> > > > > > > > non-primitive structures for the same
> > matching
> > > > > > > name(s))
> > > > > > > > > >> which
> > > > > > > > > >> > is
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > > > > slightly
> > > > > > > > > >> > > > > > > > different way to try and solve a similar
> > > > > requirement
> > > > > > > --
> > > > > > > > > >> > > > unfortunately
> > > > > > > > > >> > > > > > at
> > > > > > > > > >> > > > > > > > the time the dev community was not quite
> as
> > > > active
> > > > > > and
> > > > > > > > > then
> > > > > > > > > >> I
> > > > > > > > > >> > > also
> > > > > > > > > >> > > > > got
> > > > > > > > > >> > > > > > > busy
> > > > > > > > > >> > > > > > > > with work and just life in general so
> wasn't
> > > > able
> > > > > to
> > > > > > > > > follow
> > > > > > > > > >> up
> > > > > > > > > >> > or
> > > > > > > > > >> > > > > push
> > > > > > > > > >> > > > > > > it.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > I do think it is a very good idea to have
> > some
> > > > > kind
> > > > > > of
> > > > > > > > > >> > path-like
> > > > > > > > > >> > > > > > > expression
> > > > > > > > > >> > > > > > > > to be able to specifically address a
> nested
> > > > field,
> > > > > > as
> > > > > > > I
> > > > > > > > > can
> > > > > > > > > >> see
> > > > > > > > > >> > > > that
> > > > > > > > > >> > > > > > the
> > > > > > > > > >> > > > > > > > simple "recursive" case could potentially
> > > result
> > > > > in
> > > > > > > > > >> unwanted or
> > > > > > > > > >> > > > > > > unexpected
> > > > > > > > > >> > > > > > > > behavior, plus there is the potential to
> > > > > introduce a
> > > > > > > bit
> > > > > > > > > of
> > > > > > > > > >> a
> > > > > > > > > >> > > > > > performance
> > > > > > > > > >> > > > > > > > hit to always loop through everything in
> > cases
> > > > > where
> > > > > > > the
> > > > > > > > > >> > > > > schemas/values
> > > > > > > > > >> > > > > > > > might be quite large.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > One thing I wanted to ask: instead of
> > > creating a
> > > > > new
> > > > > > > > "path
> > > > > > > > > >> > > parser"
> > > > > > > > > >> > > > > > > > including some bespoke or "borrowed"
> syntax,
> > > why
> > > > > not
> > > > > > > > just
> > > > > > > > > >> use
> > > > > > > > > >> > > > > something
> > > > > > > > > >> > > > > > > > that already exists? Specifically here I
> am
> > > > > thinking
> > > > > > > > about
> > > > > > > > > >> > > > JsonPath (
> > > > > > > > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > There is already quite nice support in
> > > JsonPath
> > > > > for
> > > > > > > > > handling
> > > > > > > > > >> > > > special
> > > > > > > > > >> > > > > > > > characters in field names, for handling
> > > > different
> > > > > > > > > >> non-primitive
> > > > > > > > > >> > > > types
> > > > > > > > > >> > > > > > > > (arrays etc), for handling multiple levels
> > of
> > > > > > nesting,
> > > > > > > > etc
> > > > > > > > > >> etc.
> > > > > > > > > >> > > > > Would
> > > > > > > > > >> > > > > > it
> > > > > > > > > >> > > > > > > > be possible to instead to re-think this
> and
> > > > maybe
> > > > > > have
> > > > > > > > > some
> > > > > > > > > >> > kind
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > > > > JsonPath-based Schema selector / updater
> > > and/or
> > > > > > > > > >> JsonPath-based
> > > > > > > > > >> > > > Value
> > > > > > > > > >> > > > > > > > selector / updater? Conceptually this
> feels
> > > like
> > > > > it
> > > > > > > > makes
> > > > > > > > > >> sense
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > me,
> > > > > > > > > >> > > > > > as
> > > > > > > > > >> > > > > > > > from the top of my head it would be quite
> a
> > > > > natural
> > > > > > > fit
> > > > > > > > to
> > > > > > > > > >> map
> > > > > > > > > >> > a
> > > > > > > > > >> > > > Json
> > > > > > > > > >> > > > > > > data
> > > > > > > > > >> > > > > > > > structure to the Connect API data
> structure
> > > (and
> > > > > you
> > > > > > > > could
> > > > > > > > > >> > > > > potentially
> > > > > > > > > >> > > > > > > even
> > > > > > > > > >> > > > > > > > try to leverage the existing
> Json-to-Connect
> > > > > > > > > >> > > > serializer/deserializer
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > help out with this even in a more "out of
> > the
> > > > > > > > box"-feeling
> > > > > > > > > >> kind
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > > way).
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Maybe also as Tom mentioned, this part (in
> > my
> > > > > > example,
> > > > > > > > > this
> > > > > > > > > >> > > > > > > JsonPath-based
> > > > > > > > > >> > > > > > > > "thing") could even be a generic API that
> > > could
> > > > be
> > > > > > > used
> > > > > > > > by
> > > > > > > > > >> any
> > > > > > > > > >> > > SMT,
> > > > > > > > > >> > > > > > > > including used in custom ones built by the
> > > > > > community.
> > > > > > > > > Then
> > > > > > > > > >> I
> > > > > > > > > >> > > think
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > use
> > > > > > > > > >> > > > > > > > a completely separate config property
> > somehow
> > > > > > related
> > > > > > > to
> > > > > > > > > >> "path"
> > > > > > > > > >> > > (as
> > > > > > > > > >> > > > > Tom
> > > > > > > > > >> > > > > > > > also mentioned) would also make a lot of
> > sense
> > > > > here
> > > > > > as
> > > > > > > > > well.
> > > > > > > > > >> > This
> > > > > > > > > >> > > > > way,
> > > > > > > > > >> > > > > > if
> > > > > > > > > >> > > > > > > > you select based on "path" then this
> > > > > JsonPath-based
> > > > > > > API
> > > > > > > > > >> would
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > used,
> > > > > > > > > >> > > > > > > > otherwise it could use something similar
> to
> > > the
> > > > > > > existing
> > > > > > > > > >> > > get-field
> > > > > > > > > >> > > > > > based
> > > > > > > > > >> > > > > > > > approach (which I guess could also be
> > > refactored
> > > > > > into
> > > > > > > > some
> > > > > > > > > >> kind
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > > > utility
> > > > > > > > > >> > > > > > > > / API as well if it made sense?)
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > And with that in mind, if this was the
> kind
> > of
> > > > > > > direction
> > > > > > > > > to
> > > > > > > > > >> go,
> > > > > > > > > >> > > > then
> > > > > > > > > >> > > > > a
> > > > > > > > > >> > > > > > > > "recursive" capability like I pitched in
> > > KIP-683
> > > > > > would
> > > > > > > > > also
> > > > > > > > > >> > > become
> > > > > > > > > >> > > > > > > > unnecessary because you could easily
> write a
> > > > > > JsonPath
> > > > > > > > > >> > expression
> > > > > > > > > >> > > > like
> > > > > > > > > >> > > > > > > > "$..someRecuriveField" and it would do the
> > > same
> > > > > > thing
> > > > > > > > (on
> > > > > > > > > >> top
> > > > > > > > > >> > of
> > > > > > > > > >> > > > > > anything
> > > > > > > > > >> > > > > > > > else you would want to do that is already
> > > > > supported
> > > > > > by
> > > > > > > > > >> > JsonPath).
> > > > > > > > > >> > > > > Then
> > > > > > > > > >> > > > > > we
> > > > > > > > > >> > > > > > > > could also kill that older KIP and do a
> bit
> > of
> > > > > > > clean-up
> > > > > > > > :)
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > [1] -
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Just some extra food for thought.
> > All-in-all I
> > > > > think
> > > > > > > > this
> > > > > > > > > >> is a
> > > > > > > > > >> > > > super
> > > > > > > > > >> > > > > > > great
> > > > > > > > > >> > > > > > > > initiative!
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Best,
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Joshua
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris
> > > > Egerton
> > > > > <
> > > > > > > > > >> > > > > > > fearthecellos@gmail.com
> > > > > > > > > >> > > > > > > > >:
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > Hi Tom,
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Thanks for taking a look at this, and
> for
> > > your
> > > > > > > > > thoughtful
> > > > > > > > > >> > > > comments.
> > > > > > > > > >> > > > > > > I'll
> > > > > > > > > >> > > > > > > > > leave it up to Jorge to address most of
> > your
> > > > > > > comments
> > > > > > > > > but
> > > > > > > > > >> I
> > > > > > > > > >> > > > wanted
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > share
> > > > > > > > > >> > > > > > > > > a couple quick thoughts I had regarding
> > 103
> > > > and
> > > > > > 104.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > 103. Like you, I was envisioning a
> > possible
> > > > > syntax
> > > > > > > for
> > > > > > > > > >> array
> > > > > > > > > >> > > > access
> > > > > > > > > >> > > > > > > that
> > > > > > > > > >> > > > > > > > > used classic C-style brackets; e.g.,
> > > > > `arr[index]`.
> > > > > > > > > >> However, I
> > > > > > > > > >> > > > > wonder
> > > > > > > > > >> > > > > > if
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > could keep things simple and use the
> same
> > > > syntax
> > > > > > > that
> > > > > > > > > >> we're
> > > > > > > > > >> > > > > proposing
> > > > > > > > > >> > > > > > > for
> > > > > > > > > >> > > > > > > > > nested field access? In other words,
> > instead
> > > > of
> > > > > > > > > >> `arr[index]`,
> > > > > > > > > >> > > > you'd
> > > > > > > > > >> > > > > > > write
> > > > > > > > > >> > > > > > > > > `arr.index`. It'd save us and users the
> > > > headache
> > > > > > of
> > > > > > > > > >> reserving
> > > > > > > > > >> > > > > > > characters
> > > > > > > > > >> > > > > > > > > now that would need to be escaped even
> if
> > > > their
> > > > > > > > > unescaped
> > > > > > > > > >> > > > brethren
> > > > > > > > > >> > > > > > > aren't
> > > > > > > > > >> > > > > > > > > used for anything, and also avoid the
> > > question
> > > > > of
> > > > > > > what
> > > > > > > > > >> > exactly
> > > > > > > > > >> > > we
> > > > > > > > > >> > > > > > > should
> > > > > > > > > >> > > > > > > > do
> > > > > > > > > >> > > > > > > > > when we see a config that uses reserved
> > > > > characters
> > > > > > > > that
> > > > > > > > > >> > aren't
> > > > > > > > > >> > > > yet
> > > > > > > > > >> > > > > > > > > supported (throwing an exception seems
> > > pretty
> > > > > > > > unfriendly
> > > > > > > > > >> for
> > > > > > > > > >> > > new
> > > > > > > > > >> > > > > > > users).
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > 104. This would probably be useful, but
> it
> > > > would
> > > > > > > come
> > > > > > > > > with
> > > > > > > > > >> > some
> > > > > > > > > >> > > > > nasty
> > > > > > > > > >> > > > > > > > > compatibility questions that would need
> to
> > > be
> > > > > > > > addressed
> > > > > > > > > if
> > > > > > > > > >> > we'd
> > > > > > > > > >> > > > > want
> > > > > > > > > >> > > > > > > SMTs
> > > > > > > > > >> > > > > > > > > that leverage this new API to be viable
> > for
> > > > > older
> > > > > > > > > >> versions of
> > > > > > > > > >> > > > > > Connect.
> > > > > > > > > >> > > > > > > If
> > > > > > > > > >> > > > > > > > > we package and distribute this feature
> as
> > a
> > > > > > library
> > > > > > > > > >> (either
> > > > > > > > > >> > via
> > > > > > > > > >> > > > an
> > > > > > > > > >> > > > > > > > entirely
> > > > > > > > > >> > > > > > > > > new artifact, or as part of the existing
> > > > > > > > > >> connect-transforms
> > > > > > > > > >> > or
> > > > > > > > > >> > > > > > > > connect-api
> > > > > > > > > >> > > > > > > > > artifacts), then we'd have to either
> > > sidestep
> > > > > the
> > > > > > > > > existing
> > > > > > > > > >> > > plugin
> > > > > > > > > >> > > > > > > > isolation
> > > > > > > > > >> > > > > > > > > logic [1] that basically makes it
> > impossible
> > > > for
> > > > > > > > Connect
> > > > > > > > > >> > > plugins
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > ship
> > > > > > > > > >> > > > > > > > > their own versions of Connect artifacts,
> > or
> > > > > issue
> > > > > > a
> > > > > > > > big
> > > > > > > > > >> > warning
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > people
> > > > > > > > > >> > > > > > > > > that any SMT that uses this API won't
> work
> > > > with
> > > > > > any
> > > > > > > > > older
> > > > > > > > > >> > > > versions
> > > > > > > > > >> > > > > of
> > > > > > > > > >> > > > > > > > > Connect. There's also some other
> features
> > we
> > > > > might
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > >> > add
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > > an
> > > > > > > > > >> > > > > > > > > SMT-utils library such as the existing,
> > > > > internal,
> > > > > > > > utils
> > > > > > > > > >> that
> > > > > > > > > >> > > > > Connect
> > > > > > > > > >> > > > > > > uses
> > > > > > > > > >> > > > > > > > > right now [2]. It may be worth exploring
> > > this
> > > > > in a
> > > > > > > > > >> separate
> > > > > > > > > >> > KIP
> > > > > > > > > >> > > > of
> > > > > > > > > >> > > > > > its
> > > > > > > > > >> > > > > > > > own.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > [1] -
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > [2] -
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom
> > Bentley
> > > <
> > > > > > > > > >> > > tbentley@redhat.com
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Thanks for the KIP, especially for the
> > > > > examples
> > > > > > > > which
> > > > > > > > > >> are
> > > > > > > > > >> > > > > > > super-clear.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > 100. The name `field.style` isn't so
> > clear
> > > > for
> > > > > > > > > something
> > > > > > > > > >> > like
> > > > > > > > > >> > > > > > > > > ReplaceField:
> > > > > > > > > >> > > > > > > > > > it's not so obvious that field.style
> > > applies
> > > > > to
> > > > > > > > > >> `include`
> > > > > > > > > >> > and
> > > > > > > > > >> > > > > > > > `exclude`.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > 101. The permitted values for
> > > `field.style`
> > > > > > don't
> > > > > > > > seem
> > > > > > > > > >> > > terribly
> > > > > > > > > >> > > > > > > > intuitive
> > > > > > > > > >> > > > > > > > > > (to me anyway): the meaning of `plain`
> > > isn't
> > > > > > very
> > > > > > > > > >> > guessable.
> > > > > > > > > >> > > > Why
> > > > > > > > > >> > > > > > not
> > > > > > > > > >> > > > > > > > > > `top-level` or `root` instead? Also
> > > `nested`
> > > > > > could
> > > > > > > > be
> > > > > > > > > >> > > > > misconstrued
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > > mean
> > > > > > > > > >> > > > > > > > > > nested-but-not-top-level, so perhaps
> > > > > `recursive`
> > > > > > > or
> > > > > > > > > >> > > `cascading`
> > > > > > > > > >> > > > > > might
> > > > > > > > > >> > > > > > > > be
> > > > > > > > > >> > > > > > > > > > better?
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > 102. I'm torn on whether making the
> > > > > > interpretation
> > > > > > > > of
> > > > > > > > > >> > > existing
> > > > > > > > > >> > > > > > > configs
> > > > > > > > > >> > > > > > > > > like
> > > > > > > > > >> > > > > > > > > > `field` be dependent on `field.style`
> > is a
> > > > > good
> > > > > > > > idea.
> > > > > > > > > I
> > > > > > > > > >> can
> > > > > > > > > >> > > see
> > > > > > > > > >> > > > > > that
> > > > > > > > > >> > > > > > > > it's
> > > > > > > > > >> > > > > > > > > > the simplest thing to do, but it just
> > > feels
> > > > a
> > > > > > bit
> > > > > > > > odd
> > > > > > > > > >> that
> > > > > > > > > >> > > > > > sometimes
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > `field` would actually be a path and
> > have
> > > > > > > different
> > > > > > > > > >> > escaping
> > > > > > > > > >> > > > > rules.
> > > > > > > > > >> > > > > > > An
> > > > > > > > > >> > > > > > > > > > alternative would be to come up with a
> > > > > parallel
> > > > > > > set
> > > > > > > > of
> > > > > > > > > >> > config
> > > > > > > > > >> > > > > names
> > > > > > > > > >> > > > > > > > (e.g.
> > > > > > > > > >> > > > > > > > > > as well as "field" an SMT might
> support
> > > > > "path")
> > > > > > > > which
> > > > > > > > > >> were
> > > > > > > > > >> > > > > defined
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > > > always take paths, thus avoiding the
> > > > > changeable
> > > > > > > > > >> > > interpretation
> > > > > > > > > >> > > > of
> > > > > > > > > >> > > > > > the
> > > > > > > > > >> > > > > > > > > > existing configs. The SMT's
> #configure()
> > > > would
> > > > > > > need
> > > > > > > > to
> > > > > > > > > >> > throw
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > case
> > > > > > > > > >> > > > > > > > > > that both configs were given. I can
> see
> > > that
> > > > > > that
> > > > > > > > > would
> > > > > > > > > >> be
> > > > > > > > > >> > > more
> > > > > > > > > >> > > > > > work
> > > > > > > > > >> > > > > > > in
> > > > > > > > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > 103. I think in order to allow for
> > > > supporting
> > > > > > > arrays
> > > > > > > > > in
> > > > > > > > > >> a
> > > > > > > > > >> > > later
> > > > > > > > > >> > > > > KIP
> > > > > > > > > >> > > > > > > > > (which
> > > > > > > > > >> > > > > > > > > > certainly seems like it could be
> > useful),
> > > > we'd
> > > > > > > want
> > > > > > > > to
> > > > > > > > > >> > > specify
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > syntax
> > > > > > > > > >> > > > > > > > > > now, even if it wasn't implemented
> under
> > > > this
> > > > > > KIP.
> > > > > > > > > >> That's
> > > > > > > > > >> > > > > because I
> > > > > > > > > >> > > > > > > > don't
> > > > > > > > > >> > > > > > > > > > think you can't exclude the
> possibility
> > > that
> > > > > > > > > characters
> > > > > > > > > >> > such
> > > > > > > > > >> > > as
> > > > > > > > > >> > > > > `[`
> > > > > > > > > >> > > > > > > and
> > > > > > > > > >> > > > > > > > > `]`
> > > > > > > > > >> > > > > > > > > > appear in field names. So you'd have a
> > > > > > > compatibility
> > > > > > > > > >> > problem
> > > > > > > > > >> > > if
> > > > > > > > > >> > > > > > > people
> > > > > > > > > >> > > > > > > > > > started using the features of this KIP
> > to
> > > > > access
> > > > > > > > such
> > > > > > > > > >> > fields,
> > > > > > > > > >> > > > > only
> > > > > > > > > >> > > > > > > for
> > > > > > > > > >> > > > > > > > > > those characters to change their
> meaning
> > > > > under a
> > > > > > > > later
> > > > > > > > > >> KIP.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > 104. I also wonder whether making
> paths
> > > > into a
> > > > > > > > public
> > > > > > > > > >> Java
> > > > > > > > > >> > > API,
> > > > > > > > > >> > > > > for
> > > > > > > > > >> > > > > > > use
> > > > > > > > > >> > > > > > > > > by
> > > > > > > > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Thanks again,
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Tom
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris
> > > Egerton
> > > > <
> > > > > > > > > >> > > > > > fearthecellos@gmail.com
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge
> > > Esteban
> > > > > > > Quilcate
> > > > > > > > > >> Otoya
> > > > > > > > > >> > <
> > > > > > > > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Thank you, Chris! Not possible
> > without
> > > > > your
> > > > > > > > > >> feedback.
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04,
> Chris
> > > > > Egerton
> > > > > > <
> > > > > > > > > >> > > > > > > > fearthecellos@gmail.com
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Thank you for sticking through
> > > this. I
> > > > > > have
> > > > > > > > one
> > > > > > > > > >> small
> > > > > > > > > >> > > > > remark
> > > > > > > > > >> > > > > > > and
> > > > > > > > > >> > > > > > > > > one
> > > > > > > > > >> > > > > > > > > > > > small
> > > > > > > > > >> > > > > > > > > > > > > clarification; assuming you
> agree
> > > with
> > > > > me
> > > > > > on
> > > > > > > > > them
> > > > > > > > > >> > then
> > > > > > > > > >> > > > I'm
> > > > > > > > > >> > > > > > > ready
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > vote
> > > > > > > > > >> > > > > > > > > > > > on
> > > > > > > > > >> > > > > > > > > > > > > the KIP.
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > 1. InsertField: The
> > > > > > > "field.on.missing.parent"
> > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > "field.on.existing.field"
> > > > > > > > > >> > > > > > > > > > > > > docs both mention a permitted
> > value
> > > of
> > > > > > > > "ingore";
> > > > > > > > > >> this
> > > > > > > > > >> > > > > should
> > > > > > > > > >> > > > > > be
> > > > > > > > > >> > > > > > > > > > > "ignore",
> > > > > > > > > >> > > > > > > > > > > > > right?
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Of course, one more typo :)
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > 2. InsertField: The examples are
> > > still
> > > > > > > missing
> > > > > > > > > the
> > > > > > > > > >> > > > > > > "field.style"
> > > > > > > > > >> > > > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > > > > from the configurations. They
> > should
> > > > all
> > > > > > > > include
> > > > > > > > > >> the
> > > > > > > > > >> > > > > property
> > > > > > > > > >> > > > > > > > > > > > > "transforms.smt1.field.style":
> > > > "nested",
> > > > > > > > > correct?
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Yes, it is there. I think I know
> > what
> > > > you
> > > > > > mean
> > > > > > > > > now,
> > > > > > > > > >> > seems
> > > > > > > > > >> > > > > that
> > > > > > > > > >> > > > > > > > > > Confluence
> > > > > > > > > >> > > > > > > > > > > > is putting everything in one line
> > when
> > > > > it's
> > > > > > in
> > > > > > > > > >> separate
> > > > > > > > > >> > > > lines
> > > > > > > > > >> > > > > > in
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > editor.
> > > > > > > > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Thanks again for working through
> > > this,
> > > > > and
> > > > > > > > > >> > > > congratulations
> > > > > > > > > >> > > > > > on a
> > > > > > > > > >> > > > > > > > > > > > > well-written KIP!
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM
> > > Jorge
> > > > > > > Esteban
> > > > > > > > > >> > Quilcate
> > > > > > > > > >> > > > > Otoya
> > > > > > > > > >> > > > > > <
> > > > > > > > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com>
> wrote:
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > Thank you, Chris! I apply
> these
> > > > > > > improvements
> > > > > > > > > to
> > > > > > > > > >> the
> > > > > > > > > >> > > > KIP,
> > > > > > > > > >> > > > > > let
> > > > > > > > > >> > > > > > > me
> > > > > > > > > >> > > > > > > > > > know
> > > > > > > > > >> > > > > > > > > > > > how
> > > > > > > > > >> > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > looks now.
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43,
> > > Chris
> > > > > > > Egerton
> > > > > > > > <
> > > > > > > > > >> > > > > > > > > > fearthecellos@gmail.com
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > Wow, those examples are
> > great! A
> > > > few
> > > > > > > more
> > > > > > > > > >> > remarks,
> > > > > > > > > >> > > > but
> > > > > > > > > >> > > > > I
> > > > > > > > > >> > > > > > > > think
> > > > > > > > > >> > > > > > > > > > > we're
> > > > > > > > > >> > > > > > > > > > > > > > > getting close:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > 1. The examples differ
> across
> > > SMTs
> > > > > > with
> > > > > > > > the
> > > > > > > > > >> name
> > > > > > > > > >> > of
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > > > > > > newly-introduced
> > > > > > > > > >> > > > > > > > > > > > > > > style property; some of them
> > use
> > > > > > > > > >> "field.style",
> > > > > > > > > >> > and
> > > > > > > > > >> > > > > some
> > > > > > > > > >> > > > > > > use
> > > > > > > > > >> > > > > > > > > > > > > > > "fields.style". I think for
> > > > > > > consistency's
> > > > > > > > > >> sake we
> > > > > > > > > >> > > > > should
> > > > > > > > > >> > > > > > > > stick
> > > > > > > > > >> > > > > > > > > > with
> > > > > > > > > >> > > > > > > > > > > > > just
> > > > > > > > > >> > > > > > > > > > > > > > > "field.style"; otherwise it
> > > could
> > > > be
> > > > > > > > painful
> > > > > > > > > >> for
> > > > > > > > > >> > > > users
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > have
> > > > > > > > > >> > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > remember
> > > > > > > > > >> > > > > > > > > > > > > > > which to use.
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed
> the
> > > > config
> > > > > > > names
> > > > > > > > > to
> > > > > > > > > >> > > > > > > `field.style`.
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > 2. Some of the examples are
> > off:
> > > > > > > > > >> > > > > > > > > > > > > > > - TimestampConverter: the
> > input
> > > in
> > > > > the
> > > > > > > > > second
> > > > > > > > > >> > > example
> > > > > > > > > >> > > > > > > ("when
> > > > > > > > > >> > > > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > names
> > > > > > > > > >> > > > > > > > > > > > > > > include dots") doesn't
> > contain a
> > > > > field
> > > > > > > > with
> > > > > > > > > a
> > > > > > > > > >> > > dotted
> > > > > > > > > >> > > > > name
> > > > > > > > > >> > > > > > > > > > > > > > > - ValueToKey: the config in
> > the
> > > > > third
> > > > > > > > > example
> > > > > > > > > >> > > ("when
> > > > > > > > > >> > > > > > field
> > > > > > > > > >> > > > > > > > > names
> > > > > > > > > >> > > > > > > > > > > > > include
> > > > > > > > > >> > > > > > > > > > > > > > > dots") should probably use
> > > > > > > > > "parent..child.k2"
> > > > > > > > > >> as
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.fields"
> > > property
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > 3. RE changes to
> InsertField:
> > > > > > > > > >> > > > > > > > > > > > > > > - The InsertField SMT should
> > > also
> > > > > come
> > > > > > > > with
> > > > > > > > > >> the
> > > > > > > > > >> > new
> > > > > > > > > >> > > > > > > > > "field.style"
> > > > > > > > > >> > > > > > > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > > > > > > in order to preserve
> backwards
> > > > > > > > > compatibility,
> > > > > > > > > >> > > right?
> > > > > > > > > >> > > > I
> > > > > > > > > >> > > > > > > don't
> > > > > > > > > >> > > > > > > > > see
> > > > > > > > > >> > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > included in the example
> > configs
> > > > for
> > > > > > that
> > > > > > > > > one,
> > > > > > > > > >> > just
> > > > > > > > > >> > > > want
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > make
> > > > > > > > > >> > > > > > > > > > > sure
> > > > > > > > > >> > > > > > > > > > > > > > > - I don't know of any cases
> > > where
> > > > we
> > > > > > use
> > > > > > > > > >> > snake_case
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > > > names
> > > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > > Kafka; we should probably
> use
> > > > > > > > > >> "on.missing.parent"
> > > > > > > > > >> > > and
> > > > > > > > > >> > > > > > > > > > > > > "on.existing.field"
> > > > > > > > > >> > > > > > > > > > > > > > > as the new property names
> for
> > > > > > > InsertField.
> > > > > > > > > >> > > > > > > > > > > > > > > - Why is the
> > "on_existing_field"
> > > > (or
> > > > > > > > > >> > > > > "on.existing.field")
> > > > > > > > > >> > > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > > > only
> > > > > > > > > >> > > > > > > > > > > > > > > applied when the field style
> > is
> > > > > > nested?
> > > > > > > > > >> Couldn't
> > > > > > > > > >> > it
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > > useful
> > > > > > > > > >> > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > Great points! I have applied
> > these
> > > > > > > > suggestions
> > > > > > > > > >> to
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > KIP.
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40
> > PM
> > > > > Jorge
> > > > > > > > > Esteban
> > > > > > > > > >> > > > Quilcate
> > > > > > > > > >> > > > > > > Otoya
> > > > > > > > > >> > > > > > > > <
> > > > > > > > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Again, great feedback
> Chris.
> > > > Much
> > > > > > > > > >> appreciated.
> > > > > > > > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at
> 20:22,
> > > > Chris
> > > > > > > > > Egerton <
> > > > > > > > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > Looking good! I have a
> few
> > > > > > comments
> > > > > > > > left
> > > > > > > > > >> but
> > > > > > > > > >> > > all
> > > > > > > > > >> > > > > but
> > > > > > > > > >> > > > > > > one
> > > > > > > > > >> > > > > > > > or
> > > > > > > > > >> > > > > > > > > > two
> > > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > >> > > > > > > > > > > > > > > > minor.
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > 1. The motivation
> section
> > > > states
> > > > > > > "This
> > > > > > > > > >> KIP is
> > > > > > > > > >> > > > aimed
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > > > include
> > > > > > > > > >> > > > > > > > > > > > > > support
> > > > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > nested structures on the
> > > > > existing
> > > > > > > > > SMTs...
> > > > > > > > > >> and
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > include
> > > > > > > > > >> > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > abstractions
> > > > > > > > > >> > > > > > > > > > > > > > > > > to reuse this in future
> > > > SMTs". A
> > > > > > > good
> > > > > > > > > >> > > > > implementation
> > > > > > > > > >> > > > > > of
> > > > > > > > > >> > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > KIP
> > > > > > > > > >> > > > > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > > > > > > definitely isolate
> > reusable
> > > > > logic
> > > > > > > > into a
> > > > > > > > > >> > > separate
> > > > > > > > > >> > > > > > > > > abstraction
> > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > can
> > > > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > > easily pulled in to the
> > SMTs
> > > > we
> > > > > > want
> > > > > > > > to
> > > > > > > > > >> add
> > > > > > > > > >> > > > nested
> > > > > > > > > >> > > > > > > field
> > > > > > > > > >> > > > > > > > > > > support
> > > > > > > > > >> > > > > > > > > > > > > to,
> > > > > > > > > >> > > > > > > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > > > > > > unless we plan on making
> > > this
> > > > > kind
> > > > > > > of
> > > > > > > > > >> > > abstraction
> > > > > > > > > >> > > > > > > > publicly
> > > > > > > > > >> > > > > > > > > > > > > available
> > > > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > > > >> > > > > > > > > > > > > > > > > some kind of utility
> > method
> > > or
> > > > > > class
> > > > > > > > > that
> > > > > > > > > >> > > > external
> > > > > > > > > >> > > > > > SMT
> > > > > > > > > >> > > > > > > > > > > developers
> > > > > > > > > >> > > > > > > > > > > > > can
> > > > > > > > > >> > > > > > > > > > > > > > > > > leverage, we can
> probably
> > > > leave
> > > > > > this
> > > > > > > > > part
> > > > > > > > > >> out
> > > > > > > > > >> > > as
> > > > > > > > > >> > > > > it's
> > > > > > > > > >> > > > > > > > more
> > > > > > > > > >> > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > an
> > > > > > > > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Make sense, will leave
> this
> > > out
> > > > of
> > > > > > the
> > > > > > > > > KIP.
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a
> > > > little
> > > > > > > > > >> misleading,
> > > > > > > > > >> > > isn't
> > > > > > > > > >> > > > > it?
> > > > > > > > > >> > > > > > > It
> > > > > > > > > >> > > > > > > > > > > > > demonstrates
> > > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > escape syntax for fields
> > > with
> > > > > dot
> > > > > > > > > >> literals in
> > > > > > > > > >> > > > their
> > > > > > > > > >> > > > > > > > names,
> > > > > > > > > >> > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > doesn't
> > > > > > > > > >> > > > > > > > > > > > > > > > > demonstrate a way to
> > > actually
> > > > > use
> > > > > > > the
> > > > > > > > > Cast
> > > > > > > > > >> > (or
> > > > > > > > > >> > > > any
> > > > > > > > > >> > > > > > > other)
> > > > > > > > > >> > > > > > > > > SMT
> > > > > > > > > >> > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > access a
> > > > > > > > > >> > > > > > > > > > > > > > > > > nested field in a
> record,
> > > > which
> > > > > is
> > > > > > > the
> > > > > > > > > >> whole
> > > > > > > > > >> > > > point
> > > > > > > > > >> > > > > of
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > > > KIP.
> > > > > > > > > >> > > > > > > > > > > I
> > > > > > > > > >> > > > > > > > > > > > > like
> > > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > example of escape syntax
> > but
> > > > we
> > > > > > > should
> > > > > > > > > >> > probably
> > > > > > > > > >> > > > > also
> > > > > > > > > >> > > > > > > add
> > > > > > > > > >> > > > > > > > > one
> > > > > > > > > >> > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > > > field access.
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Agree. I have added
> examples
> > > to
> > > > > each
> > > > > > > SMT
> > > > > > > > > to
> > > > > > > > > >> be
> > > > > > > > > >> > > more
> > > > > > > > > >> > > > > > clear
> > > > > > > > > >> > > > > > > > > about
> > > > > > > > > >> > > > > > > > > > > how
> > > > > > > > > >> > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > > affects each function
> > > > > > > > > >> > > > > > > > > > > > > > > > .
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > 3. With the InsertField
> > SMT,
> > > > I'm
> > > > > > > > > wondering
> > > > > > > > > >> > what
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > > specific
> > > > > > > > > >> > > > > > > > > > > > > behavior
> > > > > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > > > > > > be when some or all of
> the
> > > > > "middle
> > > > > > > > > layer"
> > > > > > > > > >> > > nested
> > > > > > > > > >> > > > > > fields
> > > > > > > > > >> > > > > > > > are
> > > > > > > > > >> > > > > > > > > > > > > missing.
> > > > > > > > > >> > > > > > > > > > > > > > > For
> > > > > > > > > >> > > > > > > > > > > > > > > > > example, if I have a
> > record
> > > > > with a
> > > > > > > > value
> > > > > > > > > >> of {
> > > > > > > > > >> > > > "k1":
> > > > > > > > > >> > > > > > > "v1 }
> > > > > > > > > >> > > > > > > > > > and I
> > > > > > > > > >> > > > > > > > > > > > > apply
> > > > > > > > > >> > > > > > > > > > > > > > > > > InsertField with
> > > topic.field =
> > > > > > > > > >> > n1.n2.n3.topic,
> > > > > > > > > >> > > > what
> > > > > > > > > >> > > > > > > will
> > > > > > > > > >> > > > > > > > > > > happen?
> > > > > > > > > >> > > > > > > > > > > > > Will
> > > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > updated value become {
> > "k1":
> > > > > "v1",
> > > > > > > > > "n1": {
> > > > > > > > > >> > > "n2":
> > > > > > > > > >> > > > {
> > > > > > > > > >> > > > > > > "n3":
> > > > > > > > > >> > > > > > > > > > > "topic"
> > > > > > > > > >> > > > > > > > > > > > > }}},
> > > > > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > > > > > > an exception be thrown,
> or
> > > > > > something
> > > > > > > > > else?
> > > > > > > > > >> > This
> > > > > > > > > >> > > > > seems
> > > > > > > > > >> > > > > > > > > > analogous
> > > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > command line mkdir
> > command,
> > > > > which
> > > > > > > (at
> > > > > > > > > >> least
> > > > > > > > > >> > on
> > > > > > > > > >> > > > some
> > > > > > > > > >> > > > > > > > > operating
> > > > > > > > > >> > > > > > > > > > > > > > systems)
> > > > > > > > > >> > > > > > > > > > > > > > > > > fails by default if you
> > try
> > > to
> > > > > > > create
> > > > > > > > a
> > > > > > > > > >> new
> > > > > > > > > >> > > > nested
> > > > > > > > > >> > > > > > > > > directory
> > > > > > > > > >> > > > > > > > > > > > where
> > > > > > > > > >> > > > > > > > > > > > > > > > anything
> > > > > > > > > >> > > > > > > > > > > > > > > > > but the last element in
> > the
> > > > path
> > > > > > > > doesn't
> > > > > > > > > >> > exist,
> > > > > > > > > >> > > > but
> > > > > > > > > >> > > > > > can
> > > > > > > > > >> > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > invoked
> > > > > > > > > >> > > > > > > > > > > > > > > with a
> > > > > > > > > >> > > > > > > > > > > > > > > > > flag that instructs it
> to
> > go
> > > > > ahead
> > > > > > > and
> > > > > > > > > >> create
> > > > > > > > > >> > > all
> > > > > > > > > >> > > > > > > levels
> > > > > > > > > >> > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > > > directory instead. I'm
> > > leaning
> > > > > on
> > > > > > > the
> > > > > > > > > >> side of
> > > > > > > > > >> > > > "just
> > > > > > > > > >> > > > > > > > create
> > > > > > > > > >> > > > > > > > > > > > > > everything"
> > > > > > > > > >> > > > > > > > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > > > > > > would be interested in
> > your
> > > > > > > thoughts,
> > > > > > > > > and
> > > > > > > > > >> > > either
> > > > > > > > > >> > > > > way,
> > > > > > > > > >> > > > > > > we
> > > > > > > > > >> > > > > > > > > > should
> > > > > > > > > >> > > > > > > > > > > > > > > probably
> > > > > > > > > >> > > > > > > > > > > > > > > > > make sure the intended
> > > > behavior
> > > > > is
> > > > > > > > > >> > well-defined
> > > > > > > > > >> > > > > > before
> > > > > > > > > >> > > > > > > > > > voting.
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > This is an interesting
> case,
> > > > > thanks
> > > > > > > for
> > > > > > > > > >> > catching
> > > > > > > > > >> > > > > this!
> > > > > > > > > >> > > > > > > > > > > > > > > > The default behavior I see
> > is
> > > to
> > > > > > > create
> > > > > > > > > >> parents
> > > > > > > > > >> > > if
> > > > > > > > > >> > > > > they
> > > > > > > > > >> > > > > > > are
> > > > > > > > > >> > > > > > > > > > > missing
> > > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > overwrite fields
> > > > > > > > > >> > > > > > > > > > > > > > > > if they already exist.
> > > > > > > > > >> > > > > > > > > > > > > > > > I'm planning to include
> the
> > > > > > following
> > > > > > > > two
> > > > > > > > > >> flags
> > > > > > > > > >> > > if
> > > > > > > > > >> > > > > > there
> > > > > > > > > >> > > > > > > > is a
> > > > > > > > > >> > > > > > > > > > > need
> > > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > > > > > > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> > > > > > > (CREATE|IGNORE),
> > > > > > > > > >> > > > > default=CREATE
> > > > > > > > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > > > > > > > (OVERWRITE|IGNORE),
> > > > > > > > > >> > > > > > > > default=OVERWRITE
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will
> > the
> > > > > > behavior
> > > > > > > > be
> > > > > > > > > if
> > > > > > > > > >> > any
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > elements
> > > > > > > > > >> > > > > > > > > > > > > > > > > specified with
> InsertField
> > > > > already
> > > > > > > > exist
> > > > > > > > > >> in
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > record
> > > > > > > > > >> > > > > > > > > value?
> > > > > > > > > >> > > > > > > > > > > > Will
> > > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > >> > > > > > > > > > > > > > > > just
> > > > > > > > > >> > > > > > > > > > > > > > > > > overwrite them? What's
> the
> > > > > > behavior
> > > > > > > of
> > > > > > > > > >> > > > InsertField
> > > > > > > > > >> > > > > > > today
> > > > > > > > > >> > > > > > > > > > under
> > > > > > > > > >> > > > > > > > > > > > > > similar
> > > > > > > > > >> > > > > > > > > > > > > > > > > circumstances?
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > The current behavior is to
> > > > > overwrite
> > > > > > > the
> > > > > > > > > >> value.
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at
> > 4:15
> > > > PM
> > > > > > > Jorge
> > > > > > > > > >> Esteban
> > > > > > > > > >> > > > > > Quilcate
> > > > > > > > > >> > > > > > > > > Otoya
> > > > > > > > > >> > > > > > > > > > <
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> quilcate.jorge@gmail.com>
> > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much
> > > > > appreciated
> > > > > > > all
> > > > > > > > > the
> > > > > > > > > >> > > > feedback
> > > > > > > > > >> > > > > > > here.
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > 1. You nailed it
> setting
> > > the
> > > > > > > design
> > > > > > > > > goal
> > > > > > > > > >> > > here:
> > > > > > > > > >> > > > > "it
> > > > > > > > > >> > > > > > > > > > shouldn't
> > > > > > > > > >> > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > > impossible
> > > > > > > > > >> > > > > > > > > > > > > > > > > > to use this new
> feature
> > > for
> > > > > any
> > > > > > > > field
> > > > > > > > > >> name,
> > > > > > > > > >> > > no
> > > > > > > > > >> > > > > > matter
> > > > > > > > > >> > > > > > > > how
> > > > > > > > > >> > > > > > > > > > > > > > convoluted.
> > > > > > > > > >> > > > > > > > > > > > > > > > > It's
> > > > > > > > > >> > > > > > > > > > > > > > > > > > fine if edge cases
> > > introduce
> > > > > > > > > difficulty
> > > > > > > > > >> > (such
> > > > > > > > > >> > > > as
> > > > > > > > > >> > > > > > > > > > > less-readable
> > > > > > > > > >> > > > > > > > > > > > > > > > > > configurations), but
> > it's
> > > > not
> > > > > > fine
> > > > > > > > if
> > > > > > > > > >> they
> > > > > > > > > >> > > > can't
> > > > > > > > > >> > > > > be
> > > > > > > > > >> > > > > > > > > > addressed
> > > > > > > > > >> > > > > > > > > > > > at
> > > > > > > > > >> > > > > > > > > > > > > > > all."
> > > > > > > > > >> > > > > > > > > > > > > > > > > > Back to the previous
> > > > proposals
> > > > > > > > (using
> > > > > > > > > >> only
> > > > > > > > > >> > > dots
> > > > > > > > > >> > > > > as
> > > > > > > > > >> > > > > > > > > > > separators)
> > > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > >> > > > > > > > > > > > > > > have
> > > > > > > > > >> > > > > > > > > > > > > > > > 2
> > > > > > > > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > 1. escaping with
> > > backslashes
> > > > > > > > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots
> > > itself
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > I'll lean for
> > alternative
> > > 2,
> > > > > as
> > > > > > > you
> > > > > > > > > >> > proposed
> > > > > > > > > >> > > > > > before.
> > > > > > > > > >> > > > > > > > > Feels
> > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > me
> > > > > > > > > >> > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > backslashes have more
> > > > > potential
> > > > > > to
> > > > > > > > > lead
> > > > > > > > > >> to
> > > > > > > > > >> > > > > > confusion
> > > > > > > > > >> > > > > > > in
> > > > > > > > > >> > > > > > > > > > JSON
> > > > > > > > > >> > > > > > > > > > > > > > configs,
> > > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > CSV seems like a good
> > > > > precedent
> > > > > > to
> > > > > > > > use
> > > > > > > > > >> the
> > > > > > > > > >> > > same
> > > > > > > > > >> > > > > > > > character
> > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > escape
> > > > > > > > > >> > > > > > > > > > > > > > > > > itself.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > KIP is updated to
> > reflect
> > > > > this.
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an
> > > > > example,
> > > > > > > and
> > > > > > > > > >> stick
> > > > > > > > > >> > > with
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > > current
> > > > > > > > > >> > > > > > > > > > > > > approach
> > > > > > > > > >> > > > > > > > > > > > > > > > > > defining the style per
> > > > > > individual
> > > > > > > > > >> transform
> > > > > > > > > >> > > > > > > > > configuration.
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP
> > > updated.
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP
> > updated.
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at
> > > > 21:59,
> > > > > > > Chris
> > > > > > > > > >> > Egerton <
> > > > > > > > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for
> addressing
> > my
> > > > > > > comments;
> > > > > > > > > the
> > > > > > > > > >> > KIP
> > > > > > > > > >> > > > > looks
> > > > > > > > > >> > > > > > > > > > up-to-date
> > > > > > > > > >> > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > pretty
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > readable now, and
> the
> > > > > rejected
> > > > > > > > > >> > alternatives
> > > > > > > > > >> > > > > > section
> > > > > > > > > >> > > > > > > > > does
> > > > > > > > > >> > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > great
> > > > > > > > > >> > > > > > > > > > > > > > > job
> > > > > > > > > >> > > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > outlining the
> > discussion
> > > > so
> > > > > > far
> > > > > > > > and
> > > > > > > > > >> > > providing
> > > > > > > > > >> > > > > > > context
> > > > > > > > > >> > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > anyone
> > > > > > > > > >> > > > > > > > > > > > > > > else
> > > > > > > > > >> > > > > > > > > > > > > > > > > who
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > might want to join
> in.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on
> choice
> > of
> > > > > > > > delimiter:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > - I like the
> > > optimization
> > > > > for
> > > > > > > > simple
> > > > > > > > > >> > cases,
> > > > > > > > > >> > > > > but I
> > > > > > > > > >> > > > > > > > think
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > >> > > > > > > > > > > > > > > > > proposal
> > > > > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > a little too
> > > restrictive.
> > > > > What
> > > > > > > if
> > > > > > > > > >> > there's a
> > > > > > > > > >> > > > > field
> > > > > > > > > >> > > > > > > > whose
> > > > > > > > > >> > > > > > > > > > > name
> > > > > > > > > >> > > > > > > > > > > > > > > contains
> > > > > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > of the permitted
> > options
> > > > > > > > (currently
> > > > > > > > > >> just
> > > > > > > > > >> > > ".",
> > > > > > > > > >> > > > > > ",",
> > > > > > > > > >> > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > "/")?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > - If we expand the
> set
> > > of
> > > > > > > > permitted
> > > > > > > > > >> > > > delimiters
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > allow
> > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > any
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > single-character
> > string,
> > > > > > > > > configuration
> > > > > > > > > >> > > > > complexity
> > > > > > > > > >> > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > increase
> > > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > readability may
> > decrease
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > - Also worth
> pointing
> > > out
> > > > > that
> > > > > > > > there
> > > > > > > > > >> is
> > > > > > > > > >> > > some
> > > > > > > > > >> > > > > > > > convention
> > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > doubling
> > > > > > > > > >> > > > > > > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > delimiter character
> as
> > > an
> > > > > > escape
> > > > > > > > > >> > mechanism
> > > > > > > > > >> > > > with
> > > > > > > > > >> > > > > > > > formats
> > > > > > > > > >> > > > > > > > > > > like
> > > > > > > > > >> > > > > > > > > > > > > CSV
> > > > > > > > > >> > > > > > > > > > > > > > > [1]
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > - Overall I think we
> > may
> > > > be
> > > > > > > > > >> approaching
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > > saturation
> > > > > > > > > >> > > > > > > > > > > point
> > > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > productive
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > discussion on
> > delimiter
> > > > > syntax
> > > > > > > so
> > > > > > > > I
> > > > > > > > > >> don't
> > > > > > > > > >> > > > want
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > spend
> > > > > > > > > >> > > > > > > > > > too
> > > > > > > > > >> > > > > > > > > > > > > much
> > > > > > > > > >> > > > > > > > > > > > > > > more
> > > > > > > > > >> > > > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > your time on it. I
> > think
> > > > the
> > > > > > one
> > > > > > > > > point
> > > > > > > > > >> > I'd
> > > > > > > > > >> > > > like
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > > leave
> > > > > > > > > >> > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > now
> > > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > shouldn't be
> > impossible
> > > to
> > > > > use
> > > > > > > > this
> > > > > > > > > >> new
> > > > > > > > > >> > > > feature
> > > > > > > > > >> > > > > > for
> > > > > > > > > >> > > > > > > > any
> > > > > > > > > >> > > > > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > > name,
> > > > > > > > > >> > > > > > > > > > > > > > > > no
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > matter how
> convoluted.
> > > > It's
> > > > > > fine
> > > > > > > > if
> > > > > > > > > >> edge
> > > > > > > > > >> > > > cases
> > > > > > > > > >> > > > > > > > > introduce
> > > > > > > > > >> > > > > > > > > > > > > > difficulty
> > > > > > > > > >> > > > > > > > > > > > > > > > > (such
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > as less-readable
> > > > > > > configurations),
> > > > > > > > > but
> > > > > > > > > >> > it's
> > > > > > > > > >> > > > not
> > > > > > > > > >> > > > > > fine
> > > > > > > > > >> > > > > > > > if
> > > > > > > > > >> > > > > > > > > > they
> > > > > > > > > >> > > > > > > > > > > > > can't
> > > > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > 2.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > The configuration
> > style
> > > > > where
> > > > > > > you
> > > > > > > > > >> define
> > > > > > > > > >> > > > > > > > > > > > > "transforms.field.style"
> > > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > connector config,
> and
> > > then
> > > > > > this
> > > > > > > > > >> applies
> > > > > > > > > >> > to
> > > > > > > > > >> > > > all
> > > > > > > > > >> > > > > > SMTs
> > > > > > > > > >> > > > > > > > for
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > connector,
> > > > > > > > > >> > > > > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > very interesting.
> > > However,
> > > > > it
> > > > > > > > > doesn't
> > > > > > > > > >> > > follow
> > > > > > > > > >> > > > > > > > convention
> > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > existing
> > > > > > > > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Right now, if you
> want
> > > to
> > > > > > > > configure
> > > > > > > > > an
> > > > > > > > > >> > SMT,
> > > > > > > > > >> > > > you
> > > > > > > > > >> > > > > > > > define
> > > > > > > > > >> > > > > > > > > > its
> > > > > > > > > >> > > > > > > > > > > > name
> > > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > connector config
> (for
> > > > > example,
> > > > > > > > > >> > > "transforms":
> > > > > > > > > >> > > > > > > "smt1"),
> > > > > > > > > >> > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > then
> > > > > > > > > >> > > > > > > > > > > > > > > define
> > > > > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > of the properties
> for
> > > that
> > > > > SMT
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >> > > > connector
> > > > > > > > > >> > > > > > > > config
> > > > > > > > > >> > > > > > > > > > > using
> > > > > > > > > >> > > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > namespacing
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > mechanism specific
> to
> > > that
> > > > > SMT
> > > > > > > > (for
> > > > > > > > > >> > > example,
> > > > > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT
> then
> > > > sees
> > > > > > only
> > > > > > > > the
> > > > > > > > > >> > > > properties
> > > > > > > > > >> > > > > > > > defined
> > > > > > > > > >> > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > namespace,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > with the prefix
> > stripped
> > > > > (for
> > > > > > > > > example,
> > > > > > > > > >> > > > "prop1":
> > > > > > > > > >> > > > > > > > "val1")
> > > > > > > > > >> > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > its
> > > > > > > > > >> > > > > > > > > > > > > > > > > configure
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > If we want to
> continue
> > > to
> > > > > > follow
> > > > > > > > > this
> > > > > > > > > >> > > > > convention,
> > > > > > > > > >> > > > > > > > then
> > > > > > > > > >> > > > > > > > > > > > instead
> > > > > > > > > >> > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > specifying
> > > > > > > > "transforms.field.style"
> > > > > > > > > >> in a
> > > > > > > > > >> > > > > > connector
> > > > > > > > > >> > > > > > > > > > config,
> > > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > > > > > expect
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > > > > > > > >> > > > > > "transforms.<name>.field.style",
> > > > > > > > > >> > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > each
> > > > > > > > > >> > > > > > > > > > > > > SMT
> > > > > > > > > >> > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > they
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > want to configure a
> > > field
> > > > > > style
> > > > > > > > for.
> > > > > > > > > >> This
> > > > > > > > > >> > > > would
> > > > > > > > > >> > > > > > > > require
> > > > > > > > > >> > > > > > > > > > > more
> > > > > > > > > >> > > > > > > > > > > > > work
> > > > > > > > > >> > > > > > > > > > > > > > > on
> > > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > part of the user,
> but
> > > > would
> > > > > be
> > > > > > > > > >> simpler to
> > > > > > > > > >> > > > > reason
> > > > > > > > > >> > > > > > > > about
> > > > > > > > > >> > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > easier
> > > > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > implement.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > If we want to
> explore
> > an
> > > > > > > > alternative
> > > > > > > > > >> > where
> > > > > > > > > >> > > > > users
> > > > > > > > > >> > > > > > > can
> > > > > > > > > >> > > > > > > > > > > specify
> > > > > > > > > >> > > > > > > > > > > > > > global
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > properties that
> apply
> > to
> > > > all
> > > > > > > > > >> transforms
> > > > > > > > > >> > in
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > > > > > connector
> > > > > > > > > >> > > > > > > > > > > > config,
> > > > > > > > > >> > > > > > > > > > > > > > then
> > > > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > semantics for this
> > need
> > > to
> > > > > be
> > > > > > > > > defined
> > > > > > > > > >> in
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > KIP.
> > > > > > > > > >> > > > > > > > This
> > > > > > > > > >> > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > have
> > > > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > include whether this
> > > will
> > > > > > apply
> > > > > > > > only
> > > > > > > > > >> for
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > special
> > > > > > > > > >> > > > > > > > > case
> > > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > "field.style" and
> > > possibly
> > > > > > > > > >> > > "field.separator"
> > > > > > > > > >> > > > > > > > properties
> > > > > > > > > >> > > > > > > > > > or
> > > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > >> > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > available more
> > generally
> > > > for
> > > > > > > other
> > > > > > > > > >> > > > properties,
> > > > > > > > > >> > > > > > > > whether
> > > > > > > > > >> > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > > > > apply
> > > > > > > > > >> > > > > > > > > > > > > > > > > only
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > for the SMTs
> outlined
> > in
> > > > the
> > > > > > KIP
> > > > > > > > or
> > > > > > > > > if
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > > > "field.style"
> > > > > > > > > >> > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > possibly
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > "field.separator"
> > > > properties
> > > > > > > would
> > > > > > > > > >> also
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > passed
> > > > > > > > > >> > > > > > > > into
> > > > > > > > > >> > > > > > > > > > > custom
> > > > > > > > > >> > > > > > > > > > > > > > SMTs
> > > > > > > > > >> > > > > > > > > > > > > > > so
> > > > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > they could choose to
> > act
> > > > on
> > > > > > them
> > > > > > > > if
> > > > > > > > > >> > > > applicable,
> > > > > > > > > >> > > > > > how
> > > > > > > > > >> > > > > > > > > edge
> > > > > > > > > >> > > > > > > > > > > > cases
> > > > > > > > > >> > > > > > > > > > > > > > like
> > > > > > > > > >> > > > > > > > > > > > > > > > > > having
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > an SMT named "field"
> > in
> > > > your
> > > > > > > > > connector
> > > > > > > > > >> > > config
> > > > > > > > > >> > > > > > would
> > > > > > > > > >> > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > handled,
> > > > > > > > > >> > > > > > > > > > > > > > > etc.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Either way, it might
> > > help
> > > > to
> > > > > > > have
> > > > > > > > an
> > > > > > > > > >> > > example
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > > > the
> > > > > > > > > >> > > > > > > > KIP
> > > > > > > > > >> > > > > > > > > > > > > outlining
> > > > > > > > > >> > > > > > > > > > > > > > > how
> > > > > > > > > >> > > > > > > > > > > > > > > > > one
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > of the
> to-be-augmented
> > > > SMTs
> > > > > > can
> > > > > > > be
> > > > > > > > > >> > > configured
> > > > > > > > > >> > > > > > with
> > > > > > > > > >> > > > > > > > this
> > > > > > > > > >> > > > > > > > > > new
> > > > > > > > > >> > > > > > > > > > > > > > feature
> > > > > > > > > >> > > > > > > > > > > > > > > > > and a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > before/after of how
> a
> > > > record
> > > > > > > value
> > > > > > > > > >> would
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > > > > transformed
> > > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > >> > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for
> > the
> > > > > > > > > >> > > > > "transforms.field.style"
> > > > > > > > > >> > > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > > > > > mentions
> > > > > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > the permitted values
> > are
> > > > > > "plain"
> > > > > > > > and
> > > > > > > > > >> > > > "nested",
> > > > > > > > > >> > > > > > but
> > > > > > > > > >> > > > > > > > then
> > > > > > > > > >> > > > > > > > > > > > > describes
> > > > > > > > > >> > > > > > > > > > > > > > > > > > behavior
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > with a value of
> > "root".
> > > > > Should
> > > > > > > > that
> > > > > > > > > be
> > > > > > > > > >> > > > "plain"
> > > > > > > > > >> > > > > > > > instead?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for
> > the
> > > > > > > > > >> > > > > > > "transforms.field.separator"
> > > > > > > > > >> > > > > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > > > > > > > > > exclusively
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > mentions structs,
> but
> > > the
> > > > > > > feature
> > > > > > > > is
> > > > > > > > > >> > > intended
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > work
> > > > > > > > > >> > > > > > > > > > with
> > > > > > > > > >> > > > > > > > > > > > maps
> > > > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > > > >> > > > > > > > > > > > > > > > > well.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Can we update it to
> > > > reflect
> > > > > > > this?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > References:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > > > > > > > >> > https://stackoverflow.com/a/17808731
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022
> > at
> > > > 1:32
> > > > > > PM
> > > > > > > > > Jorge
> > > > > > > > > >> > > Esteban
> > > > > > > > > >> > > > > > > > Quilcate
> > > > > > > > > >> > > > > > > > > > > Otoya
> > > > > > > > > >> > > > > > > > > > > > <
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > quilcate.jorge@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> > > > > > > "this..field.child"
> > > > > > > > > >> could
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > > harder
> > > > > > > > > >> > > > > > > to
> > > > > > > > > >> > > > > > > > > > grasp
> > > > > > > > > >> > > > > > > > > > > > > than
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> "this.field/child" +
> > > > > > > separator:
> > > > > > > > > "/".
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Even though this
> > > > > represents
> > > > > > > > > >> additional
> > > > > > > > > >> > > > > > > information,
> > > > > > > > > >> > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > follows
> > > > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > similar
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > > > > > > > "Flatten#delimeter"
> > > > > > > > > >> > > > > > > configuration.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I want to give the
> > > > > separator
> > > > > > > > > >> approach
> > > > > > > > > >> > > > another
> > > > > > > > > >> > > > > > > try,
> > > > > > > > > >> > > > > > > > > so I
> > > > > > > > > >> > > > > > > > > > > > have
> > > > > > > > > >> > > > > > > > > > > > > > > > updated
> > > > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > KIP with the
> > separator
> > > > > > > proposal,
> > > > > > > > > >> > sticking
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > only 2
> > > > > > > > > >> > > > > > > > > > > > > > alternatives
> > > > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > should hopefully
> > cover
> > > > > most
> > > > > > > > > >> scenarios.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has
> > been
> > > > > > updated
> > > > > > > > > with
> > > > > > > > > >> > this
> > > > > > > > > >> > > > > > > > improvement.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I
> > > have
> > > > > > > updated
> > > > > > > > > this
> > > > > > > > > >> > > > section
> > > > > > > > > >> > > > > > > > > > accordingly.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch!
> I've
> > > > > replaced
> > > > > > > it
> > > > > > > > > with
> > > > > > > > > >> > > > > > > `DropHeaders`.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Looking forward to
> > > your
> > > > > > > > feedback.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022
> > at
> > > > > 21:33,
> > > > > > > > Chris
> > > > > > > > > >> > > Egerton
> > > > > > > > > >> > > > <
> > > > > > > > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Looking good!
> Got
> > a
> > > > few
> > > > > > more
> > > > > > > > > >> > thoughts.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to
> > revisit
> > > > > this,
> > > > > > > but
> > > > > > > > I
> > > > > > > > > >> think
> > > > > > > > > >> > > we
> > > > > > > > > >> > > > > may
> > > > > > > > > >> > > > > > > want
> > > > > > > > > >> > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > adopt
> > > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > > > slightly
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > different escape
> > > > syntax
> > > > > > > style.
> > > > > > > > > >> > > > Backslashes
> > > > > > > > > >> > > > > > are
> > > > > > > > > >> > > > > > > > > great,
> > > > > > > > > >> > > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > > > since
> > > > > > > > > >> > > > > > > > > > > > > > > > > > they're
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > already used by
> > > JSON,
> > > > > > using
> > > > > > > > them
> > > > > > > > > >> as
> > > > > > > > > >> > an
> > > > > > > > > >> > > > > escape
> > > > > > > > > >> > > > > > > > > > sequence
> > > > > > > > > >> > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > notation
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > would also lead
> to
> > > > some
> > > > > > > pretty
> > > > > > > > > >> ugly
> > > > > > > > > >> > > > > connector
> > > > > > > > > >> > > > > > > > > > configs.
> > > > > > > > > >> > > > > > > > > > > > > Anyone
> > > > > > > > > >> > > > > > > > > > > > > > > > who's
> > > > > > > > > >> > > > > > > > > > > > > > > > > > had
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > write regular
> > > > > expressions
> > > > > > > with
> > > > > > > > > >> > > > backslashes
> > > > > > > > > >> > > > > in
> > > > > > > > > >> > > > > > > > Java
> > > > > > > > > >> > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > probably
> > > > > > > > > >> > > > > > > > > > > > > > > > > > already
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > familiar with
> > this:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > > > > > > > >> > > > > > > > > > > > > > > > > What
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > do
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > you think about
> > > using
> > > > > the
> > > > > > > dot
> > > > > > > > > >> > character
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > escape
> > > > > > > > > >> > > > > > > > > > > itself?
> > > > > > > > > >> > > > > > > > > > > > > In
> > > > > > > > > >> > > > > > > > > > > > > > > > other
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > words,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > to access a
> single
> > > > field
> > > > > > > named
> > > > > > > > > >> > > > > "this.field",
> > > > > > > > > >> > > > > > > > > instead
> > > > > > > > > >> > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > using
> > > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > syntax
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > "this\.field"
> > (which
> > > > in
> > > > > > JSON
> > > > > > > > > would
> > > > > > > > > >> > have
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > be
> > > > > > > > > >> > > > > > > > > > expressed
> > > > > > > > > >> > > > > > > > > > > > as
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > > > > "this..field",
> > > > > > > > and
> > > > > > > > > >> for a
> > > > > > > > > >> > > > > single
> > > > > > > > > >> > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > named
> > > > > > > > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > instead of using
> > the
> > > > > > syntax
> > > > > > > > > >> > > "this\\field"
> > > > > > > > > >> > > > > > (or,
> > > > > > > > > >> > > > > > > in
> > > > > > > > > >> > > > > > > > > > JSON,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > > > "this\field"
> > > > > > > (or,
> > > > > > > > > in
> > > > > > > > > >> > JSON,
> > > > > > > > > >> > > > > > > > > > "this\\field").
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you
> flesh
> > > out
> > > > > the
> > > > > > > > > >> details on
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > new
> > > > > > > > > >> > > > > > > > > > > > "field.style"
> > > > > > > > > >> > > > > > > > > > > > > > > > > property,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > including the
> > type,
> > > > > > default
> > > > > > > > > value,
> > > > > > > > > >> > > > > > importance,
> > > > > > > > > >> > > > > > > > and
> > > > > > > > > >> > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > preliminary
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > See
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the
> > > > > "Compatibility,
> > > > > > > > > >> > Deprecation,
> > > > > > > > > >> > > > and
> > > > > > > > > >> > > > > > > > > Migration
> > > > > > > > > >> > > > > > > > > > > > Plan"
> > > > > > > > > >> > > > > > > > > > > > > > > > section
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > still
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > accurate after
> the
> > > > > latest
> > > > > > > > > update?
> > > > > > > > > >> > Seems
> > > > > > > > > >> > > > > like
> > > > > > > > > >> > > > > > > it's
> > > > > > > > > >> > > > > > > > > > still
> > > > > > > > > >> > > > > > > > > > > > > > written
> > > > > > > > > >> > > > > > > > > > > > > > > > > with
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > assumption that
> > > nested
> > > > > > field
> > > > > > > > > >> syntax
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > > > > hardcoded
> > > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > >> > > > > > > > > > > > > > > opt-in,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > which
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > isn't the case
> > > > anymore.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The
> "These
> > > > SMTs
> > > > > do
> > > > > > > not
> > > > > > > > > >> > require
> > > > > > > > > >> > > > > nested
> > > > > > > > > >> > > > > > > > > > structure
> > > > > > > > > >> > > > > > > > > > > > > > > support"
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > section
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > mentions a
> "Drop"
> > > > SMT. I
> > > > > > > think
> > > > > > > > > >> this
> > > > > > > > > >> > may
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > > > > referring
> > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > Confluent
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > SMT, which
> isn't a
> > > > part
> > > > > of
> > > > > > > > > Apache
> > > > > > > > > >> > > Kafka.
> > > > > > > > > >> > > > > > Should
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > > drop
> > > > > > > > > >> > > > > > > > > > > > > (heh)
> > > > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > SMT
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > the list? Or
> > perhaps
> > > > > just
> > > > > > > > > replace
> > > > > > > > > >> it
> > > > > > > > > >> > > with
> > > > > > > > > >> > > > > > > > > > > "DropHeaders",
> > > > > > > > > >> > > > > > > > > > > > > > which
> > > > > > > > > >> > > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > currently
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > missing from the
> > > list
> > > > > and
> > > > > > > > > >> shouldn't
> > > > > > > > > >> > > > require
> > > > > > > > > >> > > > > > any
> > > > > > > > > >> > > > > > > > > > > > > nested-field
> > > > > > > > > >> > > > > > > > > > > > > > > > > related
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28,
> > 2022
> > > > at
> > > > > > 2:12
> > > > > > > > PM
> > > > > > > > > >> Jorge
> > > > > > > > > >> > > > > Esteban
> > > > > > > > > >> > > > > > > > > > Quilcate
> > > > > > > > > >> > > > > > > > > > > > > Otoya
> > > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > quilcate.jorge@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thank you,
> > Chris!
> > > > and
> > > > > > > sorry
> > > > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> > > > > delayed
> > > > > > > > > >> > > > > > > > > > response.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Please, find
> my
> > > > > comments
> > > > > > > > > below:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb
> > > 2022
> > > > at
> > > > > > > > 17:34,
> > > > > > > > > >> Chris
> > > > > > > > > >> > > > > Egerton
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > <chrise@confluent.io.invalid
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for
> the
> > > > KIP!
> > > > > > I'd
> > > > > > > > love
> > > > > > > > > >> to
> > > > > > > > > >> > see
> > > > > > > > > >> > > > > > support
> > > > > > > > > >> > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > fields
> > > > > > > > > >> > > > > > > > > > > > > > > > > > added
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> out-of-the-box
> > > > SMTs
> > > > > > > > provided
> > > > > > > > > >> with
> > > > > > > > > >> > > > > > Connect.
> > > > > > > > > >> > > > > > > > Here
> > > > > > > > > >> > > > > > > > > > are
> > > > > > > > > >> > > > > > > > > > > > my
> > > > > > > > > >> > > > > > > > > > > > > > > > initial
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree
> > that
> > > > > > there's
> > > > > > > a
> > > > > > > > > >> case to
> > > > > > > > > >> > > be
> > > > > > > > > >> > > > > made
> > > > > > > > > >> > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > expanding
> > > > > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > new config
> > > > property
> > > > > > for
> > > > > > > > > >> > > identifying a
> > > > > > > > > >> > > > > > > nested,
> > > > > > > > > >> > > > > > > > > > > > > > to-be-hoisted
> > > > > > > > > >> > > > > > > > > > > > > > > > > > field,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example in
> the
> > > KIP
> > > > > > > doesn't
> > > > > > > > > >> really
> > > > > > > > > >> > > > > > > demonstrate
> > > > > > > > > >> > > > > > > > > why
> > > > > > > > > >> > > > > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd
> be
> > > > > helpful
> > > > > > to
> > > > > > > > > >> expand
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > example
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > add
> > > > > > > > > >> > > > > > > > > > > > > other
> > > > > > > > > >> > > > > > > > > > > > > > > > fields
> > > > > > > > > >> > > > > > > > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > order
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > show how
> > adding
> > > > > nested
> > > > > > > > field
> > > > > > > > > >> > > support
> > > > > > > > > >> > > > > > > enables
> > > > > > > > > >> > > > > > > > > > users
> > > > > > > > > >> > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > hoist a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > without
> > dropping
> > > > > other
> > > > > > > > > fields
> > > > > > > > > >> > from
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > value.
> > > > > > > > > >> > > > > > > > > > Maybe
> > > > > > > > > >> > > > > > > > > > > > > > > something
> > > > > > > > > >> > > > > > > > > > > > > > > > > > like
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     source =
> > > > > > nested.val
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     field =
> > line
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> > > > (before):
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "nested":
> > > > > > {
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "val":
> > > > > > > 42,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "other
> > > > > > > > val":
> > > > > > > > > >> 96
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  }
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> > > (after):
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "nested":
> > > > > > {
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >  "line":
> > > > > > > {
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >  "val":
> > > > > > > > > 42,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  }
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "other
> > > > > > > > val":
> > > > > > > > > >> 96
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  }
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I
> > think
> > > > > > "source"
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > >> > > little
> > > > > > > > > >> > > > > > > strange
> > > > > > > > > >> > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > new
> > > > > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property
> name.
> > > > Maybe
> > > > > > > > > >> "hoisted" or
> > > > > > > > > >> > > > > > > > > "hoisted.field"
> > > > > > > > > >> > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and
> 2.:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The
> > example
> > > > for
> > > > > > > this
> > > > > > > > > SMT
> > > > > > > > > >> is
> > > > > > > > > >> > > > > updated
> > > > > > > > > >> > > > > > > and
> > > > > > > > > >> > > > > > > > > have
> > > > > > > > > >> > > > > > > > > > > > added
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there
> a
> > > > > > reasonable
> > > > > > > > use
> > > > > > > > > >> case
> > > > > > > > > >> > > for
> > > > > > > > > >> > > > > > > > expanding
> > > > > > > > > >> > > > > > > > > > > > Flatten
> > > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten
> > specific
> > > > > > fields?
> > > > > > > > My
> > > > > > > > > >> > > > > understanding
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > it's
> > > > > > > > > >> > > > > > > > > > > > > > > mostly
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > useful
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > writing to
> > > systems
> > > > > > like
> > > > > > > > > >> databases
> > > > > > > > > >> > > > that
> > > > > > > > > >> > > > > > > don't
> > > > > > > > > >> > > > > > > > > > > support
> > > > > > > > > >> > > > > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > > > > values
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > require
> > > everything
> > > > > to
> > > > > > > be a
> > > > > > > > > >> flat
> > > > > > > > > >> > > list
> > > > > > > > > >> > > > of
> > > > > > > > > >> > > > > > > > > key-value
> > > > > > > > > >> > > > > > > > > > > > > pairs.
> > > > > > > > > >> > > > > > > > > > > > > > > > Being
> > > > > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a
> > nested
> > > > > field
> > > > > > > > > >> wouldn't
> > > > > > > > > >> > > > provide
> > > > > > > > > >> > > > > > any
> > > > > > > > > >> > > > > > > > > > > advantage
> > > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > use
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Are there
> > other
> > > > > cases
> > > > > > > > where
> > > > > > > > > it
> > > > > > > > > >> > > would?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't
> > think
> > > > we
> > > > > > > should
> > > > > > > > > >> > > > > > unconditionally
> > > > > > > > > >> > > > > > > > > change
> > > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > default
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten.
> It's
> > a
> > > > > > > > > >> > > > backwards-incompatible,
> > > > > > > > > >> > > > > > > > > breaking
> > > > > > > > > >> > > > > > > > > > > > change
> > > > > > > > > >> > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > could
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > headaches
> for
> > > > users.
> > > > > > It
> > > > > > > > > might
> > > > > > > > > >> be
> > > > > > > > > >> > > > > > reasonable
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > change
> > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically
> > > based
> > > > on
> > > > > > > > whether
> > > > > > > > > >> the
> > > > > > > > > >> > > user
> > > > > > > > > >> > > > > has
> > > > > > > > > >> > > > > > > > > > > specified a
> > > > > > > > > >> > > > > > > > > > > > > > value
> > > > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property,
> but
> > > > > > > considering
> > > > > > > > > the
> > > > > > > > > >> > > > > motivation
> > > > > > > > > >> > > > > > > for
> > > > > > > > > >> > > > > > > > > > > changing
> > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it creates
> > > > conflicts
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > >> > > > > > > > to-be-introduced
> > > > > > > > > >> > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > > > > > > syntax
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > could arise
> > with
> > > > > > > > downstream
> > > > > > > > > >> SMTs
> > > > > > > > > >> > > > > > regardless
> > > > > > > > > >> > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > whether
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > user
> > > > > > > > > >> > > > > > > > > > > > > > > > > > has
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly
> > > > > configured
> > > > > > > > > Flatten
> > > > > > > > > >> > with
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > "field"
> > > > > > > > > >> > > > > > > > > > > > > > property), I
> > > > > > > > > >> > > > > > > > > > > > > > > > > don't
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > know
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this would
> be
> > > too
> > > > > > useful
> > > > > > > > > >> either.
> > > > > > > > > >> > I
> > > > > > > > > >> > > > have
> > > > > > > > > >> > > > > > > some
> > > > > > > > > >> > > > > > > > > > > thoughts
> > > > > > > > > >> > > > > > > > > > > > > > below
> > > > > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > > > > >> > > > > > > > > > > > > > > > > > how
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > handle
> > possible
> > > > > > > conflicts
> > > > > > > > > >> between
> > > > > > > > > >> > > > names
> > > > > > > > > >> > > > > > > with
> > > > > > > > > >> > > > > > > > > dots
> > > > > > > > > >> > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > their
> > > > > > > > > >> > > > > > > > > > > > > > > > > names
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for
> > > nested
> > > > > > field
> > > > > > > > > >> > references
> > > > > > > > > >> > > > that
> > > > > > > > > >> > > > > > > > should
> > > > > > > > > >> > > > > > > > > > > > > hopefully
> > > > > > > > > >> > > > > > > > > > > > > > > make
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > either
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough.
> > With
> > > > the
> > > > > > > > support
> > > > > > > > > >> for
> > > > > > > > > >> > > > nested
> > > > > > > > > >> > > > > > > fields
> > > > > > > > > >> > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > other
> > > > > > > > > >> > > > > > > > > > > > > > > SMTs,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > could stay as
> it
> > > is.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > This removes
> the
> > > > need
> > > > > > for
> > > > > > > > (4)
> > > > > > > > > >> > > changing
> > > > > > > > > >> > > > > > > Flatten
> > > > > > > > > >> > > > > > > > > > config
> > > > > > > > > >> > > > > > > > > > > > as
> > > > > > > > > >> > > > > > > > > > > > > > > well.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think
> > it's
> > > > fine
> > > > > > to
> > > > > > > > > expand
> > > > > > > > > >> > > > > > ExtractField
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > support
> > > > > > > > > >> > > > > > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it might be
> > > worth
> > > > > > noting
> > > > > > > > in
> > > > > > > > > >> the
> > > > > > > > > >> > > > > rejected
> > > > > > > > > >> > > > > > > > > > > alternatives
> > > > > > > > > >> > > > > > > > > > > > > > > section
> > > > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > isn't
> strictly
> > > > > > necessary
> > > > > > > > > since
> > > > > > > > > >> > you
> > > > > > > > > >> > > > can
> > > > > > > > > >> > > > > > > > replace
> > > > > > > > > >> > > > > > > > > > any
> > > > > > > > > >> > > > > > > > > > > > > single
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > invocation
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT
> that
> > > uses
> > > > > > > nested
> > > > > > > > > >> field
> > > > > > > > > >> > > > > notation
> > > > > > > > > >> > > > > > > with
> > > > > > > > > >> > > > > > > > > > > > multiple
> > > > > > > > > >> > > > > > > > > > > > > > > > > > invocations
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that use
> > > > non-nested
> > > > > > > > > notation.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding
> > it.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit:
> > > > > "RegerRouter"
> > > > > > > > should
> > > > > > > > > >> be
> > > > > > > > > >> > > > > > > "RegexRouter"
> > > > > > > > > >> > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > list
> > > > > > > > > >> > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > SMTs
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > not require
> > > nested
> > > > > > > > structure
> > > > > > > > > >> > > support.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing
> it.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be
> > > rare
> > > > > for
> > > > > > > dots
> > > > > > > > > in
> > > > > > > > > >> > field
> > > > > > > > > >> > > > > names
> > > > > > > > > >> > > > > > > to
> > > > > > > > > >> > > > > > > > > > occur
> > > > > > > > > >> > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > wild
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be
> so
> > > > > certain
> > > > > > > of
> > > > > > > > > >> this),
> > > > > > > > > >> > > but
> > > > > > > > > >> > > > > > unless
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > > want
> > > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > inflict
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > users of
> > > Flatten,
> > > > I
> > > > > > > think
> > > > > > > > > >> we're
> > > > > > > > > >> > > going
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > have
> > > > > > > > > >> > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > think
> > > > > > > > > >> > > > > > > > > > > > > > > about
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between
> dotted
> > > > > > notation
> > > > > > > > and
> > > > > > > > > >> > > > non-nested
> > > > > > > > > >> > > > > > > fields
> > > > > > > > > >> > > > > > > > > > whose
> > > > > > > > > >> > > > > > > > > > > > > names
> > > > > > > > > >> > > > > > > > > > > > > > > > > contain
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > don't think
> > this
> > > > is
> > > > > > > > actually
> > > > > > > > > >> > such a
> > > > > > > > > >> > > > bad
> > > > > > > > > >> > > > > > > > thing,
> > > > > > > > > >> > > > > > > > > > > > though.
> > > > > > > > > >> > > > > > > > > > > > > I
> > > > > > > > > >> > > > > > > > > > > > > > > > agree
> > > > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > notation is
> > > > > intuitive
> > > > > > > and
> > > > > > > > > >> pretty
> > > > > > > > > >> > > > > > > commonplace
> > > > > > > > > >> > > > > > > > > (in
> > > > > > > > > >> > > > > > > > > > > > tools
> > > > > > > > > >> > > > > > > > > > > > > > like
> > > > > > > > > >> > > > > > > > > > > > > > > > jq,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example), so
> > I'd
> > > > > like
> > > > > > it
> > > > > > > > if
> > > > > > > > > we
> > > > > > > > > >> > > could
> > > > > > > > > >> > > > > > stick
> > > > > > > > > >> > > > > > > to
> > > > > > > > > >> > > > > > > > > it.
> > > > > > > > > >> > > > > > > > > > > > What
> > > > > > > > > >> > > > > > > > > > > > > > > about
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > escape
> syntax,
> > > > > using a
> > > > > > > > > >> backslash?
> > > > > > > > > >> > > > That
> > > > > > > > > >> > > > > > way,
> > > > > > > > > >> > > > > > > > > users
> > > > > > > > > >> > > > > > > > > > > > could
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between
> > > > "this.field"
> > > > > > > > (which
> > > > > > > > > >> would
> > > > > > > > > >> > > > refer
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > > > >> > > > > > > > > > > > > > > > > > "field"
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > under
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the
> top-level
> > > > "this"
> > > > > > > > field),
> > > > > > > > > >> and
> > > > > > > > > >> > > > > > > > "this\.field"
> > > > > > > > > >> > > > > > > > > > > (which
> > > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > > > > refer
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> > > > > > > "this.field").
> > > > > > > > > >> Like
> > > > > > > > > >> > > with
> > > > > > > > > >> > > > > most
> > > > > > > > > >> > > > > > > > > > languages
> > > > > > > > > >> > > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > > use
> > > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > for escape
> > > > > sequences,
> > > > > > it
> > > > > > > > > could
> > > > > > > > > >> > also
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > used
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > escape
> > > > > > > > > >> > > > > > > > > > > > > > > itself,
> > > > > > > > > >> > > > > > > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that a field
> > > name
> > > > > > > > contains a
> > > > > > > > > >> > > > > backslash. I
> > > > > > > > > >> > > > > > > > think
> > > > > > > > > >> > > > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > more
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > simpler
> than,
> > > > e.g.,
> > > > > > > > adding a
> > > > > > > > > >> new
> > > > > > > > > >> > > > config
> > > > > > > > > >> > > > > > > > > property
> > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > toggle
> > > > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > be used when
> > > > parsing
> > > > > > > > nested
> > > > > > > > > >> field
> > > > > > > > > >> > > > > > > references.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like this
> > > > approach.
> > > > > > > Adding
> > > > > > > > > to
> > > > > > > > > >> the
> > > > > > > > > >> > > > KIP.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't
> > think
> > > > we
> > > > > > can
> > > > > > > > > >> > > > unconditionally
> > > > > > > > > >> > > > > > > turn
> > > > > > > > > >> > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > > feature
> > > > > > > > > >> > > > > > > > > > > > > > > on.
> > > > > > > > > >> > > > > > > > > > > > > > > > > The
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > risk
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > breaking
> > > existing
> > > > > > > > pipelines
> > > > > > > > > >> > > > (especially
> > > > > > > > > >> > > > > > > ones
> > > > > > > > > >> > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > involve,
> > > > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > combination
> of
> > > the
> > > > > > > Flatten
> > > > > > > > > and
> > > > > > > > > >> > Cast
> > > > > > > > > >> > > > > SMTs)
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > > > pretty
> > > > > > > > > >> > > > > > > > > > > > > > high. I
> > > > > > > > > >> > > > > > > > > > > > > > > > > think
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > should be an
> > > > opt-in
> > > > > > > > feature,
> > > > > > > > > >> at
> > > > > > > > > >> > > least
> > > > > > > > > >> > > > > > until
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > next
> > > > > > > > > >> > > > > > > > > > > > > > major
> > > > > > > > > >> > > > > > > > > > > > > > > > > > release.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > One
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > we could
> > > > accomplish
> > > > > > this
> > > > > > > > is
> > > > > > > > > by
> > > > > > > > > >> > > > > > introducing
> > > > > > > > > >> > > > > > > a
> > > > > > > > > >> > > > > > > > > new
> > > > > > > > > >> > > > > > > > > > > > > > > > "field.style"
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > (name
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > obviously
> > > subject
> > > > to
> > > > > > > > change)
> > > > > > > > > >> > > property
> > > > > > > > > >> > > > > > with
> > > > > > > > > >> > > > > > > > > values
> > > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > "plain"
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If
> > set
> > > > to
> > > > > > > > "plain"
> > > > > > > > > >> then
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > current
> > > > > > > > > >> > > > > > > > > > > > non-nested
> > > > > > > > > >> > > > > > > > > > > > > > > > > behavior
> > > > > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > and if set
> to
> > > > > > "nested",
> > > > > > > > then
> > > > > > > > > >> the
> > > > > > > > > >> > > > > proposed
> > > > > > > > > >> > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > behavior
> > > > > > > > > >> > > > > > > > > > > > > > > is.
> > > > > > > > > >> > > > > > > > > > > > > > > > > We
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > consider
> > > updating
> > > > > the
> > > > > > > > > default
> > > > > > > > > >> > value
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > > "nested"
> > > > > > > > > >> > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > future
> > > > > > > > > >> > > > > > > > > > > > > > > > > major
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > release
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > (or even
> > codify
> > > > that
> > > > > > > plan
> > > > > > > > in
> > > > > > > > > >> this
> > > > > > > > > >> > > > KIP,
> > > > > > > > > >> > > > > if
> > > > > > > > > >> > > > > > > > > there's
> > > > > > > > > >> > > > > > > > > > > > > enough
> > > > > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > This would
> > also
> > > > > leave
> > > > > > > the
> > > > > > > > > door
> > > > > > > > > >> > open
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > > > adding
> > > > > > > > > >> > > > > > > > > > > > > recursive
> > > > > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > > > > >> > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the future
> by
> > > > > adding a
> > > > > > > > > >> permitted
> > > > > > > > > >> > > > value
> > > > > > > > > >> > > > > of
> > > > > > > > > >> > > > > > > > > > > > "recursive".
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of
> the
> > > > linked
> > > > > > > > tickets
> > > > > > > > > >> in
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > > > "Motivation"
> > > > > > > > > >> > > > > > > > > > > > > section,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/KAFKA-10640
> > > > > > > ,
> > > > > > > > > >> > > > > > > > > > > > has
> > > > > > > > > >> > > > > > > > > > > > > > an
> > > > > > > > > >> > > > > > > > > > > > > > > > open
> > > > > > > > > >> > > > > > > > > > > > > > > > > > PR
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that propose
> > > > adding
> > > > > > > > > recursive
> > > > > > > > > >> > > support
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > some
> > > > > > > > > >> > > > > > > > > > SMTs.
> > > > > > > > > >> > > > > > > > > > > > > Have
> > > > > > > > > >> > > > > > > > > > > > > > > you
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > considered
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> > > > > > > functionality
> > > > > > > > > for
> > > > > > > > > >> > your
> > > > > > > > > >> > > > > KIP?
> > > > > > > > > >> > > > > > Or
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > > your
> > > > > > > > > >> > > > > > > > > > > > aim
> > > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > stick
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > solely
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > nested field
> > > > > support?
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like the
> > > > > `field.style`
> > > > > > > > > >> > > configuration
> > > > > > > > > >> > > > > flag
> > > > > > > > > >> > > > > > > > > > approach.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for
> > > pointing
> > > > > out
> > > > > > > the
> > > > > > > > > >> > > `recursive`
> > > > > > > > > >> > > > > > > > approach.
> > > > > > > > > >> > > > > > > > > > > Will
> > > > > > > > > >> > > > > > > > > > > > > add
> > > > > > > > > >> > > > > > > > > > > > > > > > > `nested`
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > at
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's
> > > check
> > > > > the
> > > > > > > > demand
> > > > > > > > > >> for
> > > > > > > > > >> > > > > > > `recursive`
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > consider
> > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > > as
> > > > > > > > > >> > > > > > > > > > > > > > > > > > part
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > this or
> another
> > > KIP.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I have added
> the
> > > > > > following
> > > > > > > > on
> > > > > > > > > >> the
> > > > > > > > > >> > > KIP:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs
> > could
> > > > > extend
> > > > > > > > this
> > > > > > > > > >> > support
> > > > > > > > > >> > > > > for:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive
> > > > notation:
> > > > > > > name a
> > > > > > > > > >> field
> > > > > > > > > >> > > and
> > > > > > > > > >> > > > > > apply
> > > > > > > > > >> > > > > > > it
> > > > > > > > > >> > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > all
> > > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > >> > > > > > > > > > > > > > > > > > across
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > schema
> matching
> > > that
> > > > > > name.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Access to
> > > arrays:
> > > > > > Adding
> > > > > > > > > `[]`
> > > > > > > > > >> > > > notation
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > > > represent
> > > > > > > > > >> > > > > > > > > > > > > > access
> > > > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > arrays
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs
> to
> > > > > fields
> > > > > > > > within
> > > > > > > > > >> an
> > > > > > > > > >> > > > array.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb
> 8,
> > > > 2022
> > > > > at
> > > > > > > > 1:23
> > > > > > > > > PM
> > > > > > > > > >> > > Jorge
> > > > > > > > > >> > > > > > > Esteban
> > > > > > > > > >> > > > > > > > > > > Quilcate
> > > > > > > > > >> > > > > > > > > > > > > > > Otoya <
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > quilcate.jorge@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev
> team,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like
> to
> > > > start
> > > > > a
> > > > > > > new
> > > > > > > > > >> > > discussion
> > > > > > > > > >> > > > > > thread
> > > > > > > > > >> > > > > > > > on
> > > > > > > > > >> > > > > > > > > > > Kafka
> > > > > > > > > >> > > > > > > > > > > > > > > Connect
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP
> is
> > > > aimed
> > > > > to
> > > > > > > > > include
> > > > > > > > > >> > > > support
> > > > > > > > > >> > > > > > for
> > > > > > > > > >> > > > > > > > > nested
> > > > > > > > > >> > > > > > > > > > > > > > > structures
> > > > > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs —
> where
> > > > this
> > > > > > make
> > > > > > > > > >> sense.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking
> > > forward
> > > > to
> > > > > > > your
> > > > > > > > > >> > feedback.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Jorge Esteban Quilcate Otoya <qu...@gmail.com>]

Thanks Robert.

KIP is now updated with the latest backtick proposal:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures

Looking forward to your feedback. Also, a reminder that the vote thread is
still open.

Thanks,
Jorge.

On Fri, 12 Aug 2022 at 19:11, Robert Yokota <ra...@gmail.com> wrote:

> Also, a double backslash can be used to escape backslash.
>
> Robert
>
> On Fri, Aug 12, 2022 at 10:56 AM Jorge Esteban Quilcate Otoya <
> quilcate.jorge@gmail.com> wrote:
>
> > Awesome, thanks Robert!
> > Will work on updated the proposal around this. Also, @Chris Egerton
> > <fe...@gmail.com> if you have any feedback about this, would be
> > much appreciated.
> >
> > Cheers,
> > Jorge.
> >
> > On Fri, 12 Aug 2022 at 18:45, Robert Yokota <ra...@gmail.com> wrote:
> >
> > > Hi Jorge,
> > >
> > > Yes, to escape a backtick I would recommend a backslash.
> > >
> > > For example, MySQL allows identifiers to be surrounded with single
> > quotes,
> > > double quotes, or backticks, and they use backslash to escape.
> > >
> > > Jsonata went with primarily backticks as they are less common in
> strings
> > > that appear in JSON, and it seems to have worked nicely for them.
> > >
> > > Regards,
> > > Robert
> > >
> > > On Fri, Aug 12, 2022 at 10:01 AM Jorge Esteban Quilcate Otoya <
> > > quilcate.jorge@gmail.com> wrote:
> > >
> > > > Thanks Robert! And sorry for the late reply.
> > > >
> > > > That's a great catch and will require the current proposal to be
> > updated.
> > > >
> > > > Using back-ticks is a good proposal. Though, the challenge we got is
> > that
> > > > JSON allows really any character into attribute names, including
> > > > back-ticks.
> > > > I wonder what's the best approach to escape back-ticks then. Given
> that
> > > > those use-cases should be minimal we could fallback to backslash to
> > > escape
> > > > back-ticks?
> > > >
> > > > What do you think?
> > > >
> > > > Many thanks,
> > > > Jorge.
> > > >
> > > >
> > > > On Wed, 20 Jul 2022 at 23:03, Robert Yokota <ra...@gmail.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm late to this thread, but would like to comment on the double
> > > > dot/double
> > > > > asterisk syntax.
> > > > >
> > > > > Unfortunately double dot is often used in JSON Path as a descendant
> > > > > selector, see
> > https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html
> > > > >
> > > > > I think a better notation would be to use backticks to quote any
> > names
> > > > with
> > > > > a dot, such as  root.`child.with.dot`.  See Jsonata syntax for
> > example:
> > > > > http://docs.jsonata.org/simple
> > > > >
> > > > > Robert
> > > > >
> > > > > On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jorge@gmail.com> wrote:
> > > > >
> > > > > > Thanks Chris! I have updated the KIP to include this fix.
> > > > > >
> > > > > > I will keep the array as a potential improvement at the moment,
> and
> > > out
> > > > > of
> > > > > > the scope of this KIP.
> > > > > >
> > > > > > Thanks,
> > > > > > Jorge.
> > > > > >
> > > > > > On Tue, 28 Jun 2022 at 23:19, Chris Egerton <
> > fearthecellos@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jorge,
> > > > > > >
> > > > > > > Apologies for the long delay, had my own KIP-related work to
> > focus
> > > > on.
> > > > > > >
> > > > > > > I think it's fine to include array accesses but it's not a
> > blocker.
> > > > I'm
> > > > > > +1
> > > > > > > either way. On that front though, I think the MaskField section
> > > might
> > > > > > need
> > > > > > > to be updated as it still mentions arrays and deep-scan?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi there,
> > > > > > > >
> > > > > > > > I have update the KIP to the previous state voted, including
> > the
> > > > > > > > configuration change from `field.style` to
> > > `field.syntax.version`.
> > > > > > > >
> > > > > > > > I'll bump the vote thread as well to see if there's agreement
> > on
> > > > > adding
> > > > > > > > this feature to Connect.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jorge.
> > > > > > > >
> > > > > > > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Thanks, Chris. Your feedback is much appreciated!
> > > > > > > > >
> > > > > > > > > I see how the current proposal might be underestimating
> some
> > > edge
> > > > > > > cases.
> > > > > > > > > I'm happy to move the design for deep-scan and multi-values
> > to
> > > > > future
> > > > > > > > > developments related with this KIP and reduce its scope,
> > though
> > > > > open
> > > > > > > for
> > > > > > > > > more feedback.
> > > > > > > > >
> > > > > > > > > Also, just to be sure, are you proposing also to not
> include
> > > > array
> > > > > > > access
> > > > > > > > > at this stage?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Jorge.
> > > > > > > > >
> > > > > > > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <
> > > > > fearthecellos@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Jorge,
> > > > > > > > >>
> > > > > > > > >> I've done some more thinking and I hate to say it, but I
> > think
> > > > the
> > > > > > > > syntax
> > > > > > > > >> does need to be expanded. Right now it's clear what "a.b"
> > > refers
> > > > > to
> > > > > > > and
> > > > > > > > >> what "a..b" refers to, but what about "a...b"? Is that
> > > referring
> > > > > to
> > > > > > > > >> subfield ".b" of field "a", or subfield "b" of field "a."?
> > > This
> > > > > gets
> > > > > > > > even
> > > > > > > > >> more complicated when thinking about fields whose names
> are
> > > > > > > exclusively
> > > > > > > > >> made up of dots.
> > > > > > > > >>
> > > > > > > > >> I'm also a little hesitant to mix the cases of multi-value
> > > paths
> > > > > and
> > > > > > > > deep
> > > > > > > > >> scans. What if you only want to access one subfield deep
> for
> > > an
> > > > > SMT,
> > > > > > > > >> instead of recursing through all the children of a given
> > > field?
> > > > > It's
> > > > > > > > akin
> > > > > > > > >> to the distinction between * and ** with file globbing
> > > patterns,
> > > > > and
> > > > > > > > there
> > > > > > > > >> could be a substantial performance difference if you have
> > > > > > > heavily-nested
> > > > > > > > >> fields.
> > > > > > > > >>
> > > > > > > > >> Ultimately, I think that if the proposed
> > > "field.syntax.version"
> > > > > > > property
> > > > > > > > >> sits well with people, it might be better to reduce the
> > scope
> > > of
> > > > > the
> > > > > > > KIP
> > > > > > > > >> back to the original proposal and just focus on adding
> > support
> > > > for
> > > > > > > > >> explicitly-specified nested values, with no multi-value
> > paths
> > > > > > > > whatsoever,
> > > > > > > > >> knowing that we have an easy way to introduce new syntax
> and
> > > > > > features
> > > > > > > in
> > > > > > > > >> the future. (We could probably leave the "a...b" case for
> > that
> > > > > next
> > > > > > > > >> version
> > > > > > > > >> too.)
> > > > > > > > >>
> > > > > > > > >> I was a huge fan of this KIP before we started trying to
> > > address
> > > > > > more
> > > > > > > > >> complex use cases, and although I don't want to write
> those
> > > > off, I
> > > > > > > think
> > > > > > > > >> we
> > > > > > > > >> may have bitten off more than we can chew in time for the
> > > 3.3.0
> > > > > > > release
> > > > > > > > >> and
> > > > > > > > >> would hate to see this KIP get delayed as a result.
> > > > > > > > >>
> > > > > > > > >> I'd be really curious to hear from Joshua and Tom on this
> > > front,
> > > > > > > though.
> > > > > > > > >> Is
> > > > > > > > >> it acceptable to move more incrementally here and settle
> on
> > > the
> > > > > > syntax
> > > > > > > > >> version property as our means of introducing new features,
> > or
> > > is
> > > > > it
> > > > > > > > >> preferable to implement things monolithically and try to
> get
> > > > > > > everything
> > > > > > > > >> (or
> > > > > > > > >> at least, as much as possible) right the first time?
> > > > > > > > >>
> > > > > > > > >> Thanks again for your continued effort on this KIP!
> > > > > > > > >>
> > > > > > > > >> Cheers,
> > > > > > > > >>
> > > > > > > > >> Chris
> > > > > > > > >>
> > > > > > > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate
> Otoya
> > <
> > > > > > > > >> quilcate.jorge@gmail.com> wrote:
> > > > > > > > >>
> > > > > > > > >> > Thanks, Chris!
> > > > > > > > >> >
> > > > > > > > >> > Please, find my comments below:
> > > > > > > > >> >
> > > > > > > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> > > > > > fearthecellos@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Jorge,
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > > > > > > >> > >
> > > > > > > > >> > > 1. Under the "Accessing multiple values by deep-scan"
> > > header
> > > > > > it's
> > > > > > > > >> stated
> > > > > > > > >> > > that "If deep-scan is used, it must have only one
> field
> > > > after
> > > > > > the
> > > > > > > > >> > asterisk
> > > > > > > > >> > > level.". However, in example 3 for the Cast SMT and
> > other
> > > > > > examples
> > > > > > > > for
> > > > > > > > >> > > other SMTs, the spec contains a field of "*.child.k2",
> > > which
> > > > > > > appears
> > > > > > > > >> to
> > > > > > > > >> > > have two fields after the asterisk level. I may be
> > > > > > > misunderstanding
> > > > > > > > >> the
> > > > > > > > >> > > proposal, but it seems like the two contradict each
> > other.
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> > Thanks for catching this. I have clarified it by
> removing
> > > this
> > > > > > > > >> restriction.
> > > > > > > > >> > Also, have extended the deep-scan scenarios.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > > 2. I'm a little unclear on why we need the special
> > > handling
> > > > > for
> > > > > > > > arrays
> > > > > > > > >> > > where, for an array field "a", the field name "a" can
> be
> > > > > treated
> > > > > > > as
> > > > > > > > >> > either
> > > > > > > > >> > > the array itself, or every element in the array. Is
> > there
> > > a
> > > > > > reason
> > > > > > > > we
> > > > > > > > >> > can't
> > > > > > > > >> > > use the field name "a.*" to handle the latter case,
> and
> > > "a"
> > > > to
> > > > > > > > handle
> > > > > > > > >> the
> > > > > > > > >> > > former?
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> > Agree, this is confusing. I like the `a.*` approach to
> > > access
> > > > > > array
> > > > > > > > >> items.
> > > > > > > > >> > I have added this to the proposal.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > > 3. How would a user specify that they'd like to
> access a
> > > > field
> > > > > > > with
> > > > > > > > >> the
> > > > > > > > >> > > literal name "*"?
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> > Good one. I'm proposing an approach similar to how it's
> > > > proposed
> > > > > > to
> > > > > > > > >> escape
> > > > > > > > >> > dots, with a double-asterisk. Curious on your thoughts
> > > around
> > > > > > this.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > > 4. For the Cast SMT, do you think it might bite some
> > > people
> > > > if
> > > > > > > > fields
> > > > > > > > >> > that
> > > > > > > > >> > > can't be cast correctly are silently ignored? I'm
> > > imagining
> > > > > the
> > > > > > > case
> > > > > > > > >> > where
> > > > > > > > >> > > none of the fields in a multi-path expression can be
> > cast
> > > > > > > correctly
> > > > > > > > >> and
> > > > > > > > >> > it
> > > > > > > > >> > > ends up eating half of someone's day to track down why
> > > their
> > > > > SMT
> > > > > > > > isn't
> > > > > > > > >> > > doing anything.
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> > If I understand correctly, this challenge could be
> > relevant
> > > > > across
> > > > > > > > SMTs.
> > > > > > > > >> > At the moment, most/all? SMTs just silently ignore.
> > > > > > > > >> > Was thinking about adding a flag
> `field.on.path.not.found`
> > > to
> > > > > > either
> > > > > > > > >> ignore
> > > > > > > > >> > or fail when no paths are found. What do your think?
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > > 5. For the ExtractField and ValueToKey SMTs, what
> > happens
> > > > if a
> > > > > > > > >> deep-scan
> > > > > > > > >> > > field name is used, but only one field is found? Is
> the
> > > > > > resulting
> > > > > > > > >> field
> > > > > > > > >> > > still an array, or is it just the single field that
> was
> > > > found?
> > > > > > > (FWIW
> > > > > > > > >> I'm
> > > > > > > > >> > > leaning towards keeping it an array just to keep
> schemas
> > > > > > > consistent
> > > > > > > > >> in a
> > > > > > > > >> > > pipeline in case the number of fields found fluctuates
> > > > across
> > > > > > > > >> records.)
> > > > > > > > >> > >
> > > > > > > > >> > > Agree. Will clarify that an array is always produced
> > even
> > > > for
> > > > > 1
> > > > > > > or 0
> > > > > > > > >> > fields are found.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that "As
> > this
> > > > SMT
> > > > > > > > affects
> > > > > > > > >> > only
> > > > > > > > >> > > existing fields, additional configurations will not be
> > > > > > required.".
> > > > > > > > >> Given
> > > > > > > > >> > > the new "field.syntax.version" property, this part
> > should
> > > > > > probably
> > > > > > > > be
> > > > > > > > >> > > removed.
> > > > > > > > >> > >
> > > > > > > > >> > Agree.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > > 7. Is recursive descent intentionally excluded? That
> was
> > > an
> > > > > > > > important
> > > > > > > > >> > part
> > > > > > > > >> > > of Joshua's KIP and his feedback on this KIP; I think
> > it's
> > > > > worth
> > > > > > > > >> pursuing
> > > > > > > > >> > > if we can.
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> > My understanding from Joshua's feedback is that by
> > including
> > > > > > support
> > > > > > > > for
> > > > > > > > >> > deep-scan, we are already covering the recursive
> > > > functionality.
> > > > > > > > Though,
> > > > > > > > >> I
> > > > > > > > >> > may be missing something.
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Cheers,
> > > > > > > > >> > >
> > > > > > > > >> > > Chris
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate
> > > > Otoya <
> > > > > > > > >> > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Thanks, Chris!
> > > > > > > > >> > > >
> > > > > > > > >> > > > I have updated the KIP with the rejected
> alternatives
> > > > > updated.
> > > > > > > > >> Also, I
> > > > > > > > >> > > have
> > > > > > > > >> > > > drafted the support for arrays and deep scans as
> part
> > of
> > > > the
> > > > > > > > >> proposed
> > > > > > > > >> > > > notation to make it more complete, even though these
> > can
> > > > be
> > > > > > > > >> implemented
> > > > > > > > >> > > in
> > > > > > > > >> > > > multiple PRs.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Looking forward to your feedback.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Cheers,
> > > > > > > > >> > > > Jorge.
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > > > > > > >> fearthecellos@gmail.com>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi Jorge,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > I really appreciate the effort you've made to
> > simplify
> > > > the
> > > > > > > > syntax
> > > > > > > > >> and
> > > > > > > > >> > > > > feature set of a JSONPath-based approach as much
> as
> > > > > > possible.
> > > > > > > > I'm
> > > > > > > > >> > still
> > > > > > > > >> > > > > hesitant to continue with it, though.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > 1. The syntax is much less friendly. Just compare
> > > > > > > > >> "top.mid.bottom" to
> > > > > > > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses
> > > > JSONPath
> > > > > or
> > > > > > > > even
> > > > > > > > >> > JSON,
> > > > > > > > >> > > > and
> > > > > > > > >> > > > > the learning curve for the former is going to be
> > > > steeper.
> > > > > > The
> > > > > > > > >> > examples
> > > > > > > > >> > > in
> > > > > > > > >> > > > > the new KIP draft you published demonstrate this
> > > pretty
> > > > > > well,
> > > > > > > > and
> > > > > > > > >> > this
> > > > > > > > >> > > is
> > > > > > > > >> > > > > without diving into the details of what escape
> > syntax
> > > > > would
> > > > > > > look
> > > > > > > > >> > like.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > 2. The three new major features that this syntax
> > adds
> > > > > (array
> > > > > > > > >> > accesses,
> > > > > > > > >> > > > deep
> > > > > > > > >> > > > > scans, and multi-value paths) could all be added
> > > pretty
> > > > > > easily
> > > > > > > > to
> > > > > > > > >> the
> > > > > > > > >> > > dot
> > > > > > > > >> > > > > notation syntax without introducing brackets and
> > > dollar
> > > > > > signs.
> > > > > > > > >> Array
> > > > > > > > >> > > > > accesses can be described using the same syntax as
> > > > > > struct/map
> > > > > > > > >> field
> > > > > > > > >> > > > access,
> > > > > > > > >> > > > > deep scans can be described using '*', and
> > multi-value
> > > > > paths
> > > > > > > can
> > > > > > > > >> be
> > > > > > > > >> > > > > described by referencing the name of a field
> that's
> > > > > expected
> > > > > > > to
> > > > > > > > >> have
> > > > > > > > >> > > > > children. These are all top-of-the-head ideas and
> > can
> > > > > > probably
> > > > > > > > be
> > > > > > > > >> > > > refined,
> > > > > > > > >> > > > > but hopefully they demonstrate that we can keep
> the
> > > > syntax
> > > > > > > > simple
> > > > > > > > >> > > without
> > > > > > > > >> > > > > sacrificing features. Of course, the question of
> > > leaving
> > > > > > room
> > > > > > > > for
> > > > > > > > >> > > future
> > > > > > > > >> > > > > features might arise... given that these are the
> > > > > > > out-of-the-box
> > > > > > > > >> SMTs
> > > > > > > > >> > > that
> > > > > > > > >> > > > > are likely to be the first that many people
> > encounter,
> > > > I'd
> > > > > > err
> > > > > > > > on
> > > > > > > > >> the
> > > > > > > > >> > > > side
> > > > > > > > >> > > > > of simplicity and a gentle learning curve; if
> people
> > > > need
> > > > > to
> > > > > > > get
> > > > > > > > >> more
> > > > > > > > >> > > out
> > > > > > > > >> > > > > of transforms, the option to implement their own
> is
> > > > still
> > > > > > > there.
> > > > > > > > >> If
> > > > > > > > >> > we
> > > > > > > > >> > > > can
> > > > > > > > >> > > > > address 95% of use cases with something easy to
> use,
> > > > it's
> > > > > > not
> > > > > > > > >> worth
> > > > > > > > >> > > > making
> > > > > > > > >> > > > > the feature harder for everyone to use just to
> > > > accommodate
> > > > > > the
> > > > > > > > >> > > remaining
> > > > > > > > >> > > > > 5%.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > 3. The advantage of leveraging an existing syntax
> is
> > > > > > twofold:
> > > > > > > > >> users
> > > > > > > > >> > who
> > > > > > > > >> > > > are
> > > > > > > > >> > > > > already familiar with that syntax don't need to
> > learn
> > > a
> > > > > new
> > > > > > > > >> syntax,
> > > > > > > > >> > and
> > > > > > > > >> > > > > maintainers of the syntax get to leverage existing
> > > > > libraries
> > > > > > > and
> > > > > > > > >> > > > > documentation for the syntax. With the current
> > > proposal,
> > > > > > > reusing
> > > > > > > > >> > > > libraries
> > > > > > > > >> > > > > is off the table, which means that we're going to
> > have
> > > > to
> > > > > > > parse
> > > > > > > > >> this
> > > > > > > > >> > > > > ourselves (including all the escape syntax edge
> > > cases),
> > > > > and
> > > > > > we
> > > > > > > > >> won't
> > > > > > > > >> > be
> > > > > > > > >> > > > > able to automatically leverage new features added
> to
> > > > that
> > > > > > > > syntax.
> > > > > > > > >> And
> > > > > > > > >> > > > given
> > > > > > > > >> > > > > how stripped-down the syntax is in comparison to
> > full
> > > > > > > JSONPath,
> > > > > > > > >> > there's
> > > > > > > > >> > > > > still going to be a learning curve for users who
> are
> > > > > already
> > > > > > > > >> familiar
> > > > > > > > >> > > > with
> > > > > > > > >> > > > > it, and we'll still have to document how Connect's
> > > > variant
> > > > > > of
> > > > > > > > >> > JSONPath
> > > > > > > > >> > > > > works either instead of or in addition to just
> > linking
> > > > to
> > > > > a
> > > > > > > > >> > > > well-maintained
> > > > > > > > >> > > > > third-party docs site.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On the topic of "field.path" and "include.path"
> vs.
> > > > > > > > >> "field.style", I
> > > > > > > > >> > > > > actually think that a single property per SMT is
> > > cleaner
> > > > > and
> > > > > > > > >> simpler.
> > > > > > > > >> > > > > Allowing users to mix and match styles within the
> > same
> > > > SMT
> > > > > > > > config
> > > > > > > > >> > seems
> > > > > > > > >> > > > > like a recipe for confusion, and with a single
> > > property
> > > > > that
> > > > > > > > >> dictates
> > > > > > > > >> > > > field
> > > > > > > > >> > > > > syntax behavior, we leave the door open to change
> > the
> > > > > > default
> > > > > > > > at a
> > > > > > > > >> > > later
> > > > > > > > >> > > > > date. We could even fully remove support for plain
> > > field
> > > > > > > > notation
> > > > > > > > >> at
> > > > > > > > >> > > some
> > > > > > > > >> > > > > point and still be able to retain the simple
> > property
> > > > > names
> > > > > > of
> > > > > > > > >> > "field",
> > > > > > > > >> > > > > "include", etc. instead of forcing people to use
> > > > > > "field.path"
> > > > > > > > and
> > > > > > > > >> the
> > > > > > > > >> > > > like.
> > > > > > > > >> > > > > That said, the term "field.style" and the
> permitted
> > > > values
> > > > > > > might
> > > > > > > > >> be a
> > > > > > > > >> > > > > little ambiguous. One alternative, though a little
> > > > > > > heavy-handed,
> > > > > > > > >> is
> > > > > > > > >> > to
> > > > > > > > >> > > > > change it to "field.syntax.version" with permitted
> > > > values
> > > > > of
> > > > > > > > "V1"
> > > > > > > > >> > > > (default,
> > > > > > > > >> > > > > equivalent to "field.style = plain") and "V2"
> > > > (equivalent
> > > > > to
> > > > > > > > >> > > > "field.style =
> > > > > > > > >> > > > > nested"). This would leave us room in the future
> to
> > > make
> > > > > > > further
> > > > > > > > >> > > changes
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > the syntax without having to come up with new
> names,
> > > > > > although
> > > > > > > it
> > > > > > > > >> does
> > > > > > > > >> > > > > sacrifice a little bit in that the permitted
> values
> > > are
> > > > no
> > > > > > > > longer
> > > > > > > > >> > > > > self-describing.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Cheers,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Chris
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban
> > Quilcate
> > > > > > Otoya <
> > > > > > > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Thank you all for your feedback, and sorry for
> the
> > > > long
> > > > > > wait
> > > > > > > > >> for a
> > > > > > > > >> > > > reply.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I would like to explore the idea of
> > > > > > JSONPath-inspired/subset
> > > > > > > > >> > > notation a
> > > > > > > > >> > > > > bit
> > > > > > > > >> > > > > > further:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > It will need to be a much-reduced version of
> > > JSONPath:
> > > > > > > > >> > > > > > - No full support for JsonPath therefore an
> > > additional
> > > > > > > > >> dependency.
> > > > > > > > >> > > > > > - All paths must start with '$'
> > > > > > > > >> > > > > > - No functions support or other operators
> allowed.
> > > > > > > > >> > > > > > - JsonPath dotted and square-bracket notations
> can
> > > be
> > > > > > > > supported
> > > > > > > > >> to
> > > > > > > > >> > > > avoid
> > > > > > > > >> > > > > > escaping dots or other characters: `$a.b.c` and
> > > > > > > > `$['a.b']['c']`
> > > > > > > > >> -
> > > > > > > > >> > or
> > > > > > > > >> > > we
> > > > > > > > >> > > > > > could only support the second one as it's more
> > > > complete.
> > > > > > > > >> > > > > > - Add support for arrays with `[<integer>]`,
> e.g.
> > > > > > `$a.[1].b`
> > > > > > > > >> > > > > > - Add support for multiple-value paths using
> array
> > > > > access
> > > > > > > > >> `$a.*.b`
> > > > > > > > >> > or
> > > > > > > > >> > > > > deep
> > > > > > > > >> > > > > > scan `$a..b`.
> > > > > > > > >> > > > > >     - Some SMTs that will benefit from this:
> > > > > `MaskField`,
> > > > > > > > >> `Cast`,
> > > > > > > > >> > > > > > `ReplaceField`.
> > > > > > > > >> > > > > > - We could introduce a `path[s]` config under
> the
> > > > > current
> > > > > > > > >> > > > configurations
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > apply this feature so no compatibility issues
> are
> > > > > > > introduced:
> > > > > > > > >> e.g.
> > > > > > > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > With these,  100, 101, 102, and 103 will be
> > > > effectively
> > > > > > > > solved.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > The main challenge that I see at the moment is
> > that
> > > by
> > > > > > being
> > > > > > > > >> JSON
> > > > > > > > >> > > > > > path-like, there may be some edge cases that I
> > can't
> > > > > > foresee
> > > > > > > > at
> > > > > > > > >> the
> > > > > > > > >> > > > > moment,
> > > > > > > > >> > > > > > that could make this hard to implement, test,
> and
> > > > > maintain
> > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > long
> > > > > > > > >> > > > > run.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I'll appreciate your feedback on how this
> > > > JSONPath-based
> > > > > > > > >> > alternative
> > > > > > > > >> > > > > > compares to the dotted notation initially
> > proposed,
> > > > and
> > > > > if
> > > > > > > it
> > > > > > > > >> > matches
> > > > > > > > >> > > > > your
> > > > > > > > >> > > > > > feedback.
> > > > > > > > >> > > > > > I have drafted a copy of the KIP to change the
> > > > examples
> > > > > to
> > > > > > > > >> JSONPath
> > > > > > > > >> > > > > > approach and validate some differences:
> > > > > > > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > About 104. I agreed with Chris. We can handle
> this
> > > as
> > > > > part
> > > > > > > of
> > > > > > > > a
> > > > > > > > >> new
> > > > > > > > >> > > > KIP.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Thanks,
> > > > > > > > >> > > > > > Jorge.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > > > > > > >> > fearthecellos@gmail.com
> > > > > > > > >> > > >
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi Joshua,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > I have a few reservations about using JsonPath
> > > > > notation
> > > > > > > > here.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 1. There's likely to be a substantial
> > performance
> > > > > > penalty
> > > > > > > > for
> > > > > > > > >> > > > > converting
> > > > > > > > >> > > > > > > between the Kafka Connect format and something
> > > that
> > > > a
> > > > > > > > JsonPath
> > > > > > > > >> > > > library
> > > > > > > > >> > > > > > > would understand.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 2. The complexity of the feature will be
> > > > significantly
> > > > > > > > >> higher. It
> > > > > > > > >> > > > will
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > > harder to test, document, and implement. There
> > > will
> > > > be
> > > > > > > many
> > > > > > > > >> more
> > > > > > > > >> > > edge
> > > > > > > > >> > > > > > cases
> > > > > > > > >> > > > > > > to consider and support, and we'll be on the
> > hook
> > > to
> > > > > > > handle
> > > > > > > > >> any
> > > > > > > > >> > > bugs
> > > > > > > > >> > > > or
> > > > > > > > >> > > > > > > inconsistencies that arise, either as a result
> > of
> > > > our
> > > > > > use
> > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > > > > JsonPath
> > > > > > > > >> > > > > > > library we choose, or as a result of bugs in
> > that
> > > > > > library.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 3. It's not clear that JsonPath is superior or
> > > even
> > > > > > > suitable
> > > > > > > > >> for
> > > > > > > > >> > > some
> > > > > > > > >> > > > > of
> > > > > > > > >> > > > > > > the SMTs proposed here. What would be the
> > > advantages
> > > > > of
> > > > > > > > >> JsonPath
> > > > > > > > >> > > with
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > InsertField or HoistField SMTs?
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > I also don't think that adding dot notation is
> > > > > > unfriendly
> > > > > > > to
> > > > > > > > >> > users;
> > > > > > > > >> > > > > many
> > > > > > > > >> > > > > > > have proposed this type of syntax in the past,
> > and
> > > > > it's
> > > > > > > > >> > frequently
> > > > > > > > >> > > > used
> > > > > > > > >> > > > > > in
> > > > > > > > >> > > > > > > informal discussions to refer to nested
> fields.
> > If
> > > > the
> > > > > > > > >> proposed
> > > > > > > > >> > > > syntax
> > > > > > > > >> > > > > > was
> > > > > > > > >> > > > > > > not already intuitive then a case against
> > deciding
> > > > on
> > > > > > our
> > > > > > > > own
> > > > > > > > >> > might
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > more
> > > > > > > > >> > > > > > > convincing, but as things stand, simple dot
> > > notation
> > > > > is
> > > > > > > > likely
> > > > > > > > >> > > going
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > be
> > > > > > > > >> > > > > > > easier for users to understand than JsonPath
> > > syntax.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Cheers,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Chris
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua
> Grisham <
> > > > > > > > >> > > > grishamjosh@gmail.com>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hello all!
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Sorry that I come a bit later to the party
> > here,
> > > > > but I
> > > > > > > am
> > > > > > > > >> the
> > > > > > > > >> > one
> > > > > > > > >> > > > who
> > > > > > > > >> > > > > > > wrote
> > > > > > > > >> > > > > > > > KIP-683 [1] for recursive support (just
> simply
> > > > > looping
> > > > > > > > >> through
> > > > > > > > >> > > all
> > > > > > > > >> > > > > > child
> > > > > > > > >> > > > > > > > non-primitive structures for the same
> matching
> > > > > > name(s))
> > > > > > > > >> which
> > > > > > > > >> > is
> > > > > > > > >> > > a
> > > > > > > > >> > > > > > > slightly
> > > > > > > > >> > > > > > > > different way to try and solve a similar
> > > > requirement
> > > > > > --
> > > > > > > > >> > > > unfortunately
> > > > > > > > >> > > > > > at
> > > > > > > > >> > > > > > > > the time the dev community was not quite as
> > > active
> > > > > and
> > > > > > > > then
> > > > > > > > >> I
> > > > > > > > >> > > also
> > > > > > > > >> > > > > got
> > > > > > > > >> > > > > > > busy
> > > > > > > > >> > > > > > > > with work and just life in general so wasn't
> > > able
> > > > to
> > > > > > > > follow
> > > > > > > > >> up
> > > > > > > > >> > or
> > > > > > > > >> > > > > push
> > > > > > > > >> > > > > > > it.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > I do think it is a very good idea to have
> some
> > > > kind
> > > > > of
> > > > > > > > >> > path-like
> > > > > > > > >> > > > > > > expression
> > > > > > > > >> > > > > > > > to be able to specifically address a nested
> > > field,
> > > > > as
> > > > > > I
> > > > > > > > can
> > > > > > > > >> see
> > > > > > > > >> > > > that
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > simple "recursive" case could potentially
> > result
> > > > in
> > > > > > > > >> unwanted or
> > > > > > > > >> > > > > > > unexpected
> > > > > > > > >> > > > > > > > behavior, plus there is the potential to
> > > > introduce a
> > > > > > bit
> > > > > > > > of
> > > > > > > > >> a
> > > > > > > > >> > > > > > performance
> > > > > > > > >> > > > > > > > hit to always loop through everything in
> cases
> > > > where
> > > > > > the
> > > > > > > > >> > > > > schemas/values
> > > > > > > > >> > > > > > > > might be quite large.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > One thing I wanted to ask: instead of
> > creating a
> > > > new
> > > > > > > "path
> > > > > > > > >> > > parser"
> > > > > > > > >> > > > > > > > including some bespoke or "borrowed" syntax,
> > why
> > > > not
> > > > > > > just
> > > > > > > > >> use
> > > > > > > > >> > > > > something
> > > > > > > > >> > > > > > > > that already exists? Specifically here I am
> > > > thinking
> > > > > > > about
> > > > > > > > >> > > > JsonPath (
> > > > > > > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > There is already quite nice support in
> > JsonPath
> > > > for
> > > > > > > > handling
> > > > > > > > >> > > > special
> > > > > > > > >> > > > > > > > characters in field names, for handling
> > > different
> > > > > > > > >> non-primitive
> > > > > > > > >> > > > types
> > > > > > > > >> > > > > > > > (arrays etc), for handling multiple levels
> of
> > > > > nesting,
> > > > > > > etc
> > > > > > > > >> etc.
> > > > > > > > >> > > > > Would
> > > > > > > > >> > > > > > it
> > > > > > > > >> > > > > > > > be possible to instead to re-think this and
> > > maybe
> > > > > have
> > > > > > > > some
> > > > > > > > >> > kind
> > > > > > > > >> > > of
> > > > > > > > >> > > > > > > > JsonPath-based Schema selector / updater
> > and/or
> > > > > > > > >> JsonPath-based
> > > > > > > > >> > > > Value
> > > > > > > > >> > > > > > > > selector / updater? Conceptually this feels
> > like
> > > > it
> > > > > > > makes
> > > > > > > > >> sense
> > > > > > > > >> > > to
> > > > > > > > >> > > > > me,
> > > > > > > > >> > > > > > as
> > > > > > > > >> > > > > > > > from the top of my head it would be quite a
> > > > natural
> > > > > > fit
> > > > > > > to
> > > > > > > > >> map
> > > > > > > > >> > a
> > > > > > > > >> > > > Json
> > > > > > > > >> > > > > > > data
> > > > > > > > >> > > > > > > > structure to the Connect API data structure
> > (and
> > > > you
> > > > > > > could
> > > > > > > > >> > > > > potentially
> > > > > > > > >> > > > > > > even
> > > > > > > > >> > > > > > > > try to leverage the existing Json-to-Connect
> > > > > > > > >> > > > serializer/deserializer
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > help out with this even in a more "out of
> the
> > > > > > > box"-feeling
> > > > > > > > >> kind
> > > > > > > > >> > > of
> > > > > > > > >> > > > > > way).
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Maybe also as Tom mentioned, this part (in
> my
> > > > > example,
> > > > > > > > this
> > > > > > > > >> > > > > > > JsonPath-based
> > > > > > > > >> > > > > > > > "thing") could even be a generic API that
> > could
> > > be
> > > > > > used
> > > > > > > by
> > > > > > > > >> any
> > > > > > > > >> > > SMT,
> > > > > > > > >> > > > > > > > including used in custom ones built by the
> > > > > community.
> > > > > > > > Then
> > > > > > > > >> I
> > > > > > > > >> > > think
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > use
> > > > > > > > >> > > > > > > > a completely separate config property
> somehow
> > > > > related
> > > > > > to
> > > > > > > > >> "path"
> > > > > > > > >> > > (as
> > > > > > > > >> > > > > Tom
> > > > > > > > >> > > > > > > > also mentioned) would also make a lot of
> sense
> > > > here
> > > > > as
> > > > > > > > well.
> > > > > > > > >> > This
> > > > > > > > >> > > > > way,
> > > > > > > > >> > > > > > if
> > > > > > > > >> > > > > > > > you select based on "path" then this
> > > > JsonPath-based
> > > > > > API
> > > > > > > > >> would
> > > > > > > > >> > be
> > > > > > > > >> > > > > used,
> > > > > > > > >> > > > > > > > otherwise it could use something similar to
> > the
> > > > > > existing
> > > > > > > > >> > > get-field
> > > > > > > > >> > > > > > based
> > > > > > > > >> > > > > > > > approach (which I guess could also be
> > refactored
> > > > > into
> > > > > > > some
> > > > > > > > >> kind
> > > > > > > > >> > > of
> > > > > > > > >> > > > > > > utility
> > > > > > > > >> > > > > > > > / API as well if it made sense?)
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > And with that in mind, if this was the kind
> of
> > > > > > direction
> > > > > > > > to
> > > > > > > > >> go,
> > > > > > > > >> > > > then
> > > > > > > > >> > > > > a
> > > > > > > > >> > > > > > > > "recursive" capability like I pitched in
> > KIP-683
> > > > > would
> > > > > > > > also
> > > > > > > > >> > > become
> > > > > > > > >> > > > > > > > unnecessary because you could easily write a
> > > > > JsonPath
> > > > > > > > >> > expression
> > > > > > > > >> > > > like
> > > > > > > > >> > > > > > > > "$..someRecuriveField" and it would do the
> > same
> > > > > thing
> > > > > > > (on
> > > > > > > > >> top
> > > > > > > > >> > of
> > > > > > > > >> > > > > > anything
> > > > > > > > >> > > > > > > > else you would want to do that is already
> > > > supported
> > > > > by
> > > > > > > > >> > JsonPath).
> > > > > > > > >> > > > > Then
> > > > > > > > >> > > > > > we
> > > > > > > > >> > > > > > > > could also kill that older KIP and do a bit
> of
> > > > > > clean-up
> > > > > > > :)
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > [1] -
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Just some extra food for thought.
> All-in-all I
> > > > think
> > > > > > > this
> > > > > > > > >> is a
> > > > > > > > >> > > > super
> > > > > > > > >> > > > > > > great
> > > > > > > > >> > > > > > > > initiative!
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Best,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Joshua
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris
> > > Egerton
> > > > <
> > > > > > > > >> > > > > > > fearthecellos@gmail.com
> > > > > > > > >> > > > > > > > >:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Tom,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Thanks for taking a look at this, and for
> > your
> > > > > > > > thoughtful
> > > > > > > > >> > > > comments.
> > > > > > > > >> > > > > > > I'll
> > > > > > > > >> > > > > > > > > leave it up to Jorge to address most of
> your
> > > > > > comments
> > > > > > > > but
> > > > > > > > >> I
> > > > > > > > >> > > > wanted
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > share
> > > > > > > > >> > > > > > > > > a couple quick thoughts I had regarding
> 103
> > > and
> > > > > 104.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > 103. Like you, I was envisioning a
> possible
> > > > syntax
> > > > > > for
> > > > > > > > >> array
> > > > > > > > >> > > > access
> > > > > > > > >> > > > > > > that
> > > > > > > > >> > > > > > > > > used classic C-style brackets; e.g.,
> > > > `arr[index]`.
> > > > > > > > >> However, I
> > > > > > > > >> > > > > wonder
> > > > > > > > >> > > > > > if
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > could keep things simple and use the same
> > > syntax
> > > > > > that
> > > > > > > > >> we're
> > > > > > > > >> > > > > proposing
> > > > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > > > > nested field access? In other words,
> instead
> > > of
> > > > > > > > >> `arr[index]`,
> > > > > > > > >> > > > you'd
> > > > > > > > >> > > > > > > write
> > > > > > > > >> > > > > > > > > `arr.index`. It'd save us and users the
> > > headache
> > > > > of
> > > > > > > > >> reserving
> > > > > > > > >> > > > > > > characters
> > > > > > > > >> > > > > > > > > now that would need to be escaped even if
> > > their
> > > > > > > > unescaped
> > > > > > > > >> > > > brethren
> > > > > > > > >> > > > > > > aren't
> > > > > > > > >> > > > > > > > > used for anything, and also avoid the
> > question
> > > > of
> > > > > > what
> > > > > > > > >> > exactly
> > > > > > > > >> > > we
> > > > > > > > >> > > > > > > should
> > > > > > > > >> > > > > > > > do
> > > > > > > > >> > > > > > > > > when we see a config that uses reserved
> > > > characters
> > > > > > > that
> > > > > > > > >> > aren't
> > > > > > > > >> > > > yet
> > > > > > > > >> > > > > > > > > supported (throwing an exception seems
> > pretty
> > > > > > > unfriendly
> > > > > > > > >> for
> > > > > > > > >> > > new
> > > > > > > > >> > > > > > > users).
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > 104. This would probably be useful, but it
> > > would
> > > > > > come
> > > > > > > > with
> > > > > > > > >> > some
> > > > > > > > >> > > > > nasty
> > > > > > > > >> > > > > > > > > compatibility questions that would need to
> > be
> > > > > > > addressed
> > > > > > > > if
> > > > > > > > >> > we'd
> > > > > > > > >> > > > > want
> > > > > > > > >> > > > > > > SMTs
> > > > > > > > >> > > > > > > > > that leverage this new API to be viable
> for
> > > > older
> > > > > > > > >> versions of
> > > > > > > > >> > > > > > Connect.
> > > > > > > > >> > > > > > > If
> > > > > > > > >> > > > > > > > > we package and distribute this feature as
> a
> > > > > library
> > > > > > > > >> (either
> > > > > > > > >> > via
> > > > > > > > >> > > > an
> > > > > > > > >> > > > > > > > entirely
> > > > > > > > >> > > > > > > > > new artifact, or as part of the existing
> > > > > > > > >> connect-transforms
> > > > > > > > >> > or
> > > > > > > > >> > > > > > > > connect-api
> > > > > > > > >> > > > > > > > > artifacts), then we'd have to either
> > sidestep
> > > > the
> > > > > > > > existing
> > > > > > > > >> > > plugin
> > > > > > > > >> > > > > > > > isolation
> > > > > > > > >> > > > > > > > > logic [1] that basically makes it
> impossible
> > > for
> > > > > > > Connect
> > > > > > > > >> > > plugins
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > ship
> > > > > > > > >> > > > > > > > > their own versions of Connect artifacts,
> or
> > > > issue
> > > > > a
> > > > > > > big
> > > > > > > > >> > warning
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > people
> > > > > > > > >> > > > > > > > > that any SMT that uses this API won't work
> > > with
> > > > > any
> > > > > > > > older
> > > > > > > > >> > > > versions
> > > > > > > > >> > > > > of
> > > > > > > > >> > > > > > > > > Connect. There's also some other features
> we
> > > > might
> > > > > > > want
> > > > > > > > to
> > > > > > > > >> > add
> > > > > > > > >> > > in
> > > > > > > > >> > > > > an
> > > > > > > > >> > > > > > > > > SMT-utils library such as the existing,
> > > > internal,
> > > > > > > utils
> > > > > > > > >> that
> > > > > > > > >> > > > > Connect
> > > > > > > > >> > > > > > > uses
> > > > > > > > >> > > > > > > > > right now [2]. It may be worth exploring
> > this
> > > > in a
> > > > > > > > >> separate
> > > > > > > > >> > KIP
> > > > > > > > >> > > > of
> > > > > > > > >> > > > > > its
> > > > > > > > >> > > > > > > > own.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > [1] -
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > [2] -
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Chris
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom
> Bentley
> > <
> > > > > > > > >> > > tbentley@redhat.com
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Thanks for the KIP, especially for the
> > > > examples
> > > > > > > which
> > > > > > > > >> are
> > > > > > > > >> > > > > > > super-clear.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > 100. The name `field.style` isn't so
> clear
> > > for
> > > > > > > > something
> > > > > > > > >> > like
> > > > > > > > >> > > > > > > > > ReplaceField:
> > > > > > > > >> > > > > > > > > > it's not so obvious that field.style
> > applies
> > > > to
> > > > > > > > >> `include`
> > > > > > > > >> > and
> > > > > > > > >> > > > > > > > `exclude`.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > 101. The permitted values for
> > `field.style`
> > > > > don't
> > > > > > > seem
> > > > > > > > >> > > terribly
> > > > > > > > >> > > > > > > > intuitive
> > > > > > > > >> > > > > > > > > > (to me anyway): the meaning of `plain`
> > isn't
> > > > > very
> > > > > > > > >> > guessable.
> > > > > > > > >> > > > Why
> > > > > > > > >> > > > > > not
> > > > > > > > >> > > > > > > > > > `top-level` or `root` instead? Also
> > `nested`
> > > > > could
> > > > > > > be
> > > > > > > > >> > > > > misconstrued
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > > mean
> > > > > > > > >> > > > > > > > > > nested-but-not-top-level, so perhaps
> > > > `recursive`
> > > > > > or
> > > > > > > > >> > > `cascading`
> > > > > > > > >> > > > > > might
> > > > > > > > >> > > > > > > > be
> > > > > > > > >> > > > > > > > > > better?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > 102. I'm torn on whether making the
> > > > > interpretation
> > > > > > > of
> > > > > > > > >> > > existing
> > > > > > > > >> > > > > > > configs
> > > > > > > > >> > > > > > > > > like
> > > > > > > > >> > > > > > > > > > `field` be dependent on `field.style`
> is a
> > > > good
> > > > > > > idea.
> > > > > > > > I
> > > > > > > > >> can
> > > > > > > > >> > > see
> > > > > > > > >> > > > > > that
> > > > > > > > >> > > > > > > > it's
> > > > > > > > >> > > > > > > > > > the simplest thing to do, but it just
> > feels
> > > a
> > > > > bit
> > > > > > > odd
> > > > > > > > >> that
> > > > > > > > >> > > > > > sometimes
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > `field` would actually be a path and
> have
> > > > > > different
> > > > > > > > >> > escaping
> > > > > > > > >> > > > > rules.
> > > > > > > > >> > > > > > > An
> > > > > > > > >> > > > > > > > > > alternative would be to come up with a
> > > > parallel
> > > > > > set
> > > > > > > of
> > > > > > > > >> > config
> > > > > > > > >> > > > > names
> > > > > > > > >> > > > > > > > (e.g.
> > > > > > > > >> > > > > > > > > > as well as "field" an SMT might support
> > > > "path")
> > > > > > > which
> > > > > > > > >> were
> > > > > > > > >> > > > > defined
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > > > always take paths, thus avoiding the
> > > > changeable
> > > > > > > > >> > > interpretation
> > > > > > > > >> > > > of
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > > > existing configs. The SMT's #configure()
> > > would
> > > > > > need
> > > > > > > to
> > > > > > > > >> > throw
> > > > > > > > >> > > in
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > case
> > > > > > > > >> > > > > > > > > > that both configs were given. I can see
> > that
> > > > > that
> > > > > > > > would
> > > > > > > > >> be
> > > > > > > > >> > > more
> > > > > > > > >> > > > > > work
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > 103. I think in order to allow for
> > > supporting
> > > > > > arrays
> > > > > > > > in
> > > > > > > > >> a
> > > > > > > > >> > > later
> > > > > > > > >> > > > > KIP
> > > > > > > > >> > > > > > > > > (which
> > > > > > > > >> > > > > > > > > > certainly seems like it could be
> useful),
> > > we'd
> > > > > > want
> > > > > > > to
> > > > > > > > >> > > specify
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > syntax
> > > > > > > > >> > > > > > > > > > now, even if it wasn't implemented under
> > > this
> > > > > KIP.
> > > > > > > > >> That's
> > > > > > > > >> > > > > because I
> > > > > > > > >> > > > > > > > don't
> > > > > > > > >> > > > > > > > > > think you can't exclude the possibility
> > that
> > > > > > > > characters
> > > > > > > > >> > such
> > > > > > > > >> > > as
> > > > > > > > >> > > > > `[`
> > > > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > > > > `]`
> > > > > > > > >> > > > > > > > > > appear in field names. So you'd have a
> > > > > > compatibility
> > > > > > > > >> > problem
> > > > > > > > >> > > if
> > > > > > > > >> > > > > > > people
> > > > > > > > >> > > > > > > > > > started using the features of this KIP
> to
> > > > access
> > > > > > > such
> > > > > > > > >> > fields,
> > > > > > > > >> > > > > only
> > > > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > > > > > those characters to change their meaning
> > > > under a
> > > > > > > later
> > > > > > > > >> KIP.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > 104. I also wonder whether making paths
> > > into a
> > > > > > > public
> > > > > > > > >> Java
> > > > > > > > >> > > API,
> > > > > > > > >> > > > > for
> > > > > > > > >> > > > > > > use
> > > > > > > > >> > > > > > > > > by
> > > > > > > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Thanks again,
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Tom
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris
> > Egerton
> > > <
> > > > > > > > >> > > > > > fearthecellos@gmail.com
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge
> > Esteban
> > > > > > Quilcate
> > > > > > > > >> Otoya
> > > > > > > > >> > <
> > > > > > > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Thank you, Chris! Not possible
> without
> > > > your
> > > > > > > > >> feedback.
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04, Chris
> > > > Egerton
> > > > > <
> > > > > > > > >> > > > > > > > fearthecellos@gmail.com
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Thank you for sticking through
> > this. I
> > > > > have
> > > > > > > one
> > > > > > > > >> small
> > > > > > > > >> > > > > remark
> > > > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > > > > one
> > > > > > > > >> > > > > > > > > > > > small
> > > > > > > > >> > > > > > > > > > > > > clarification; assuming you agree
> > with
> > > > me
> > > > > on
> > > > > > > > them
> > > > > > > > >> > then
> > > > > > > > >> > > > I'm
> > > > > > > > >> > > > > > > ready
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > > vote
> > > > > > > > >> > > > > > > > > > > > on
> > > > > > > > >> > > > > > > > > > > > > the KIP.
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > 1. InsertField: The
> > > > > > "field.on.missing.parent"
> > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > "field.on.existing.field"
> > > > > > > > >> > > > > > > > > > > > > docs both mention a permitted
> value
> > of
> > > > > > > "ingore";
> > > > > > > > >> this
> > > > > > > > >> > > > > should
> > > > > > > > >> > > > > > be
> > > > > > > > >> > > > > > > > > > > "ignore",
> > > > > > > > >> > > > > > > > > > > > > right?
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Of course, one more typo :)
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > 2. InsertField: The examples are
> > still
> > > > > > missing
> > > > > > > > the
> > > > > > > > >> > > > > > > "field.style"
> > > > > > > > >> > > > > > > > > > > property
> > > > > > > > >> > > > > > > > > > > > > from the configurations. They
> should
> > > all
> > > > > > > include
> > > > > > > > >> the
> > > > > > > > >> > > > > property
> > > > > > > > >> > > > > > > > > > > > > "transforms.smt1.field.style":
> > > "nested",
> > > > > > > > correct?
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Yes, it is there. I think I know
> what
> > > you
> > > > > mean
> > > > > > > > now,
> > > > > > > > >> > seems
> > > > > > > > >> > > > > that
> > > > > > > > >> > > > > > > > > > Confluence
> > > > > > > > >> > > > > > > > > > > > is putting everything in one line
> when
> > > > it's
> > > > > in
> > > > > > > > >> separate
> > > > > > > > >> > > > lines
> > > > > > > > >> > > > > > in
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > editor.
> > > > > > > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Thanks again for working through
> > this,
> > > > and
> > > > > > > > >> > > > congratulations
> > > > > > > > >> > > > > > on a
> > > > > > > > >> > > > > > > > > > > > > well-written KIP!
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Chris
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM
> > Jorge
> > > > > > Esteban
> > > > > > > > >> > Quilcate
> > > > > > > > >> > > > > Otoya
> > > > > > > > >> > > > > > <
> > > > > > > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Thank you, Chris! I apply these
> > > > > > improvements
> > > > > > > > to
> > > > > > > > >> the
> > > > > > > > >> > > > KIP,
> > > > > > > > >> > > > > > let
> > > > > > > > >> > > > > > > me
> > > > > > > > >> > > > > > > > > > know
> > > > > > > > >> > > > > > > > > > > > how
> > > > > > > > >> > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > looks now.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43,
> > Chris
> > > > > > Egerton
> > > > > > > <
> > > > > > > > >> > > > > > > > > > fearthecellos@gmail.com
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > Wow, those examples are
> great! A
> > > few
> > > > > > more
> > > > > > > > >> > remarks,
> > > > > > > > >> > > > but
> > > > > > > > >> > > > > I
> > > > > > > > >> > > > > > > > think
> > > > > > > > >> > > > > > > > > > > we're
> > > > > > > > >> > > > > > > > > > > > > > > getting close:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > 1. The examples differ across
> > SMTs
> > > > > with
> > > > > > > the
> > > > > > > > >> name
> > > > > > > > >> > of
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > > > > > > newly-introduced
> > > > > > > > >> > > > > > > > > > > > > > > style property; some of them
> use
> > > > > > > > >> "field.style",
> > > > > > > > >> > and
> > > > > > > > >> > > > > some
> > > > > > > > >> > > > > > > use
> > > > > > > > >> > > > > > > > > > > > > > > "fields.style". I think for
> > > > > > consistency's
> > > > > > > > >> sake we
> > > > > > > > >> > > > > should
> > > > > > > > >> > > > > > > > stick
> > > > > > > > >> > > > > > > > > > with
> > > > > > > > >> > > > > > > > > > > > > just
> > > > > > > > >> > > > > > > > > > > > > > > "field.style"; otherwise it
> > could
> > > be
> > > > > > > painful
> > > > > > > > >> for
> > > > > > > > >> > > > users
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > have
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > remember
> > > > > > > > >> > > > > > > > > > > > > > > which to use.
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed the
> > > config
> > > > > > names
> > > > > > > > to
> > > > > > > > >> > > > > > > `field.style`.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > 2. Some of the examples are
> off:
> > > > > > > > >> > > > > > > > > > > > > > > - TimestampConverter: the
> input
> > in
> > > > the
> > > > > > > > second
> > > > > > > > >> > > example
> > > > > > > > >> > > > > > > ("when
> > > > > > > > >> > > > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > names
> > > > > > > > >> > > > > > > > > > > > > > > include dots") doesn't
> contain a
> > > > field
> > > > > > > with
> > > > > > > > a
> > > > > > > > >> > > dotted
> > > > > > > > >> > > > > name
> > > > > > > > >> > > > > > > > > > > > > > > - ValueToKey: the config in
> the
> > > > third
> > > > > > > > example
> > > > > > > > >> > > ("when
> > > > > > > > >> > > > > > field
> > > > > > > > >> > > > > > > > > names
> > > > > > > > >> > > > > > > > > > > > > include
> > > > > > > > >> > > > > > > > > > > > > > > dots") should probably use
> > > > > > > > "parent..child.k2"
> > > > > > > > >> as
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.fields"
> > property
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > 3. RE changes to InsertField:
> > > > > > > > >> > > > > > > > > > > > > > > - The InsertField SMT should
> > also
> > > > come
> > > > > > > with
> > > > > > > > >> the
> > > > > > > > >> > new
> > > > > > > > >> > > > > > > > > "field.style"
> > > > > > > > >> > > > > > > > > > > > > > property
> > > > > > > > >> > > > > > > > > > > > > > > in order to preserve backwards
> > > > > > > > compatibility,
> > > > > > > > >> > > right?
> > > > > > > > >> > > > I
> > > > > > > > >> > > > > > > don't
> > > > > > > > >> > > > > > > > > see
> > > > > > > > >> > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > included in the example
> configs
> > > for
> > > > > that
> > > > > > > > one,
> > > > > > > > >> > just
> > > > > > > > >> > > > want
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > make
> > > > > > > > >> > > > > > > > > > > sure
> > > > > > > > >> > > > > > > > > > > > > > > - I don't know of any cases
> > where
> > > we
> > > > > use
> > > > > > > > >> > snake_case
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > > > property
> > > > > > > > >> > > > > > > > > > > > names
> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > > Kafka; we should probably use
> > > > > > > > >> "on.missing.parent"
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > > > > > > > > "on.existing.field"
> > > > > > > > >> > > > > > > > > > > > > > > as the new property names for
> > > > > > InsertField.
> > > > > > > > >> > > > > > > > > > > > > > > - Why is the
> "on_existing_field"
> > > (or
> > > > > > > > >> > > > > "on.existing.field")
> > > > > > > > >> > > > > > > > > > property
> > > > > > > > >> > > > > > > > > > > > only
> > > > > > > > >> > > > > > > > > > > > > > > applied when the field style
> is
> > > > > nested?
> > > > > > > > >> Couldn't
> > > > > > > > >> > it
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > > useful
> > > > > > > > >> > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Great points! I have applied
> these
> > > > > > > suggestions
> > > > > > > > >> to
> > > > > > > > >> > the
> > > > > > > > >> > > > > KIP.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > Chris
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40
> PM
> > > > Jorge
> > > > > > > > Esteban
> > > > > > > > >> > > > Quilcate
> > > > > > > > >> > > > > > > Otoya
> > > > > > > > >> > > > > > > > <
> > > > > > > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > wrote:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Again, great feedback Chris.
> > > Much
> > > > > > > > >> appreciated.
> > > > > > > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at 20:22,
> > > Chris
> > > > > > > > Egerton <
> > > > > > > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > Looking good! I have a few
> > > > > comments
> > > > > > > left
> > > > > > > > >> but
> > > > > > > > >> > > all
> > > > > > > > >> > > > > but
> > > > > > > > >> > > > > > > one
> > > > > > > > >> > > > > > > > or
> > > > > > > > >> > > > > > > > > > two
> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > >> > > > > > > > > > > > > > > > minor.
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > 1. The motivation section
> > > states
> > > > > > "This
> > > > > > > > >> KIP is
> > > > > > > > >> > > > aimed
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > > > include
> > > > > > > > >> > > > > > > > > > > > > > support
> > > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > nested structures on the
> > > > existing
> > > > > > > > SMTs...
> > > > > > > > >> and
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > include
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > abstractions
> > > > > > > > >> > > > > > > > > > > > > > > > > to reuse this in future
> > > SMTs". A
> > > > > > good
> > > > > > > > >> > > > > implementation
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > KIP
> > > > > > > > >> > > > > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > > > > > > definitely isolate
> reusable
> > > > logic
> > > > > > > into a
> > > > > > > > >> > > separate
> > > > > > > > >> > > > > > > > > abstraction
> > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > can
> > > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > > easily pulled in to the
> SMTs
> > > we
> > > > > want
> > > > > > > to
> > > > > > > > >> add
> > > > > > > > >> > > > nested
> > > > > > > > >> > > > > > > field
> > > > > > > > >> > > > > > > > > > > support
> > > > > > > > >> > > > > > > > > > > > > to,
> > > > > > > > >> > > > > > > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > > > > > > unless we plan on making
> > this
> > > > kind
> > > > > > of
> > > > > > > > >> > > abstraction
> > > > > > > > >> > > > > > > > publicly
> > > > > > > > >> > > > > > > > > > > > > available
> > > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > > >> > > > > > > > > > > > > > > > > some kind of utility
> method
> > or
> > > > > class
> > > > > > > > that
> > > > > > > > >> > > > external
> > > > > > > > >> > > > > > SMT
> > > > > > > > >> > > > > > > > > > > developers
> > > > > > > > >> > > > > > > > > > > > > can
> > > > > > > > >> > > > > > > > > > > > > > > > > leverage, we can probably
> > > leave
> > > > > this
> > > > > > > > part
> > > > > > > > >> out
> > > > > > > > >> > > as
> > > > > > > > >> > > > > it's
> > > > > > > > >> > > > > > > > more
> > > > > > > > >> > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > an
> > > > > > > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Make sense, will leave this
> > out
> > > of
> > > > > the
> > > > > > > > KIP.
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a
> > > little
> > > > > > > > >> misleading,
> > > > > > > > >> > > isn't
> > > > > > > > >> > > > > it?
> > > > > > > > >> > > > > > > It
> > > > > > > > >> > > > > > > > > > > > > demonstrates
> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > escape syntax for fields
> > with
> > > > dot
> > > > > > > > >> literals in
> > > > > > > > >> > > > their
> > > > > > > > >> > > > > > > > names,
> > > > > > > > >> > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > doesn't
> > > > > > > > >> > > > > > > > > > > > > > > > > demonstrate a way to
> > actually
> > > > use
> > > > > > the
> > > > > > > > Cast
> > > > > > > > >> > (or
> > > > > > > > >> > > > any
> > > > > > > > >> > > > > > > other)
> > > > > > > > >> > > > > > > > > SMT
> > > > > > > > >> > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > access a
> > > > > > > > >> > > > > > > > > > > > > > > > > nested field in a record,
> > > which
> > > > is
> > > > > > the
> > > > > > > > >> whole
> > > > > > > > >> > > > point
> > > > > > > > >> > > > > of
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > KIP.
> > > > > > > > >> > > > > > > > > > > I
> > > > > > > > >> > > > > > > > > > > > > like
> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > example of escape syntax
> but
> > > we
> > > > > > should
> > > > > > > > >> > probably
> > > > > > > > >> > > > > also
> > > > > > > > >> > > > > > > add
> > > > > > > > >> > > > > > > > > one
> > > > > > > > >> > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > > > field access.
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Agree. I have added examples
> > to
> > > > each
> > > > > > SMT
> > > > > > > > to
> > > > > > > > >> be
> > > > > > > > >> > > more
> > > > > > > > >> > > > > > clear
> > > > > > > > >> > > > > > > > > about
> > > > > > > > >> > > > > > > > > > > how
> > > > > > > > >> > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > > affects each function
> > > > > > > > >> > > > > > > > > > > > > > > > .
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > 3. With the InsertField
> SMT,
> > > I'm
> > > > > > > > wondering
> > > > > > > > >> > what
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > > specific
> > > > > > > > >> > > > > > > > > > > > > behavior
> > > > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > > > > > > be when some or all of the
> > > > "middle
> > > > > > > > layer"
> > > > > > > > >> > > nested
> > > > > > > > >> > > > > > fields
> > > > > > > > >> > > > > > > > are
> > > > > > > > >> > > > > > > > > > > > > missing.
> > > > > > > > >> > > > > > > > > > > > > > > For
> > > > > > > > >> > > > > > > > > > > > > > > > > example, if I have a
> record
> > > > with a
> > > > > > > value
> > > > > > > > >> of {
> > > > > > > > >> > > > "k1":
> > > > > > > > >> > > > > > > "v1 }
> > > > > > > > >> > > > > > > > > > and I
> > > > > > > > >> > > > > > > > > > > > > apply
> > > > > > > > >> > > > > > > > > > > > > > > > > InsertField with
> > topic.field =
> > > > > > > > >> > n1.n2.n3.topic,
> > > > > > > > >> > > > what
> > > > > > > > >> > > > > > > will
> > > > > > > > >> > > > > > > > > > > happen?
> > > > > > > > >> > > > > > > > > > > > > Will
> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > updated value become {
> "k1":
> > > > "v1",
> > > > > > > > "n1": {
> > > > > > > > >> > > "n2":
> > > > > > > > >> > > > {
> > > > > > > > >> > > > > > > "n3":
> > > > > > > > >> > > > > > > > > > > "topic"
> > > > > > > > >> > > > > > > > > > > > > }}},
> > > > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > > > > > > an exception be thrown, or
> > > > > something
> > > > > > > > else?
> > > > > > > > >> > This
> > > > > > > > >> > > > > seems
> > > > > > > > >> > > > > > > > > > analogous
> > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > command line mkdir
> command,
> > > > which
> > > > > > (at
> > > > > > > > >> least
> > > > > > > > >> > on
> > > > > > > > >> > > > some
> > > > > > > > >> > > > > > > > > operating
> > > > > > > > >> > > > > > > > > > > > > > systems)
> > > > > > > > >> > > > > > > > > > > > > > > > > fails by default if you
> try
> > to
> > > > > > create
> > > > > > > a
> > > > > > > > >> new
> > > > > > > > >> > > > nested
> > > > > > > > >> > > > > > > > > directory
> > > > > > > > >> > > > > > > > > > > > where
> > > > > > > > >> > > > > > > > > > > > > > > > anything
> > > > > > > > >> > > > > > > > > > > > > > > > > but the last element in
> the
> > > path
> > > > > > > doesn't
> > > > > > > > >> > exist,
> > > > > > > > >> > > > but
> > > > > > > > >> > > > > > can
> > > > > > > > >> > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > invoked
> > > > > > > > >> > > > > > > > > > > > > > > with a
> > > > > > > > >> > > > > > > > > > > > > > > > > flag that instructs it to
> go
> > > > ahead
> > > > > > and
> > > > > > > > >> create
> > > > > > > > >> > > all
> > > > > > > > >> > > > > > > levels
> > > > > > > > >> > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > > > directory instead. I'm
> > leaning
> > > > on
> > > > > > the
> > > > > > > > >> side of
> > > > > > > > >> > > > "just
> > > > > > > > >> > > > > > > > create
> > > > > > > > >> > > > > > > > > > > > > > everything"
> > > > > > > > >> > > > > > > > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > > > > > > would be interested in
> your
> > > > > > thoughts,
> > > > > > > > and
> > > > > > > > >> > > either
> > > > > > > > >> > > > > way,
> > > > > > > > >> > > > > > > we
> > > > > > > > >> > > > > > > > > > should
> > > > > > > > >> > > > > > > > > > > > > > > probably
> > > > > > > > >> > > > > > > > > > > > > > > > > make sure the intended
> > > behavior
> > > > is
> > > > > > > > >> > well-defined
> > > > > > > > >> > > > > > before
> > > > > > > > >> > > > > > > > > > voting.
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > This is an interesting case,
> > > > thanks
> > > > > > for
> > > > > > > > >> > catching
> > > > > > > > >> > > > > this!
> > > > > > > > >> > > > > > > > > > > > > > > > The default behavior I see
> is
> > to
> > > > > > create
> > > > > > > > >> parents
> > > > > > > > >> > > if
> > > > > > > > >> > > > > they
> > > > > > > > >> > > > > > > are
> > > > > > > > >> > > > > > > > > > > missing
> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > overwrite fields
> > > > > > > > >> > > > > > > > > > > > > > > > if they already exist.
> > > > > > > > >> > > > > > > > > > > > > > > > I'm planning to include the
> > > > > following
> > > > > > > two
> > > > > > > > >> flags
> > > > > > > > >> > > if
> > > > > > > > >> > > > > > there
> > > > > > > > >> > > > > > > > is a
> > > > > > > > >> > > > > > > > > > > need
> > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > > > > > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> > > > > > (CREATE|IGNORE),
> > > > > > > > >> > > > > default=CREATE
> > > > > > > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > > > > > > (OVERWRITE|IGNORE),
> > > > > > > > >> > > > > > > > default=OVERWRITE
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will
> the
> > > > > behavior
> > > > > > > be
> > > > > > > > if
> > > > > > > > >> > any
> > > > > > > > >> > > of
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > elements
> > > > > > > > >> > > > > > > > > > > > > > > > > specified with InsertField
> > > > already
> > > > > > > exist
> > > > > > > > >> in
> > > > > > > > >> > the
> > > > > > > > >> > > > > > record
> > > > > > > > >> > > > > > > > > value?
> > > > > > > > >> > > > > > > > > > > > Will
> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > >> > > > > > > > > > > > > > > > just
> > > > > > > > >> > > > > > > > > > > > > > > > > overwrite them? What's the
> > > > > behavior
> > > > > > of
> > > > > > > > >> > > > InsertField
> > > > > > > > >> > > > > > > today
> > > > > > > > >> > > > > > > > > > under
> > > > > > > > >> > > > > > > > > > > > > > similar
> > > > > > > > >> > > > > > > > > > > > > > > > > circumstances?
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > The current behavior is to
> > > > overwrite
> > > > > > the
> > > > > > > > >> value.
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > Chris
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at
> 4:15
> > > PM
> > > > > > Jorge
> > > > > > > > >> Esteban
> > > > > > > > >> > > > > > Quilcate
> > > > > > > > >> > > > > > > > > Otoya
> > > > > > > > >> > > > > > > > > > <
> > > > > > > > >> > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much
> > > > appreciated
> > > > > > all
> > > > > > > > the
> > > > > > > > >> > > > feedback
> > > > > > > > >> > > > > > > here.
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > 1. You nailed it setting
> > the
> > > > > > design
> > > > > > > > goal
> > > > > > > > >> > > here:
> > > > > > > > >> > > > > "it
> > > > > > > > >> > > > > > > > > > shouldn't
> > > > > > > > >> > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > > impossible
> > > > > > > > >> > > > > > > > > > > > > > > > > > to use this new feature
> > for
> > > > any
> > > > > > > field
> > > > > > > > >> name,
> > > > > > > > >> > > no
> > > > > > > > >> > > > > > matter
> > > > > > > > >> > > > > > > > how
> > > > > > > > >> > > > > > > > > > > > > > convoluted.
> > > > > > > > >> > > > > > > > > > > > > > > > > It's
> > > > > > > > >> > > > > > > > > > > > > > > > > > fine if edge cases
> > introduce
> > > > > > > > difficulty
> > > > > > > > >> > (such
> > > > > > > > >> > > > as
> > > > > > > > >> > > > > > > > > > > less-readable
> > > > > > > > >> > > > > > > > > > > > > > > > > > configurations), but
> it's
> > > not
> > > > > fine
> > > > > > > if
> > > > > > > > >> they
> > > > > > > > >> > > > can't
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > > > > > addressed
> > > > > > > > >> > > > > > > > > > > > at
> > > > > > > > >> > > > > > > > > > > > > > > all."
> > > > > > > > >> > > > > > > > > > > > > > > > > > Back to the previous
> > > proposals
> > > > > > > (using
> > > > > > > > >> only
> > > > > > > > >> > > dots
> > > > > > > > >> > > > > as
> > > > > > > > >> > > > > > > > > > > separators)
> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > >> > > > > > > > > > > > > > > have
> > > > > > > > >> > > > > > > > > > > > > > > > 2
> > > > > > > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > > > > > > >> > > > > > > > > > > > > > > > > > 1. escaping with
> > backslashes
> > > > > > > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots
> > itself
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > I'll lean for
> alternative
> > 2,
> > > > as
> > > > > > you
> > > > > > > > >> > proposed
> > > > > > > > >> > > > > > before.
> > > > > > > > >> > > > > > > > > Feels
> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > me
> > > > > > > > >> > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > backslashes have more
> > > > potential
> > > > > to
> > > > > > > > lead
> > > > > > > > >> to
> > > > > > > > >> > > > > > confusion
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > > JSON
> > > > > > > > >> > > > > > > > > > > > > > configs,
> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > CSV seems like a good
> > > > precedent
> > > > > to
> > > > > > > use
> > > > > > > > >> the
> > > > > > > > >> > > same
> > > > > > > > >> > > > > > > > character
> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > escape
> > > > > > > > >> > > > > > > > > > > > > > > > > itself.
> > > > > > > > >> > > > > > > > > > > > > > > > > > KIP is updated to
> reflect
> > > > this.
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an
> > > > example,
> > > > > > and
> > > > > > > > >> stick
> > > > > > > > >> > > with
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > > current
> > > > > > > > >> > > > > > > > > > > > > approach
> > > > > > > > >> > > > > > > > > > > > > > > > > > defining the style per
> > > > > individual
> > > > > > > > >> transform
> > > > > > > > >> > > > > > > > > configuration.
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP
> > updated.
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP
> updated.
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at
> > > 21:59,
> > > > > > Chris
> > > > > > > > >> > Egerton <
> > > > > > > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for addressing
> my
> > > > > > comments;
> > > > > > > > the
> > > > > > > > >> > KIP
> > > > > > > > >> > > > > looks
> > > > > > > > >> > > > > > > > > > up-to-date
> > > > > > > > >> > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > pretty
> > > > > > > > >> > > > > > > > > > > > > > > > > > > readable now, and the
> > > > rejected
> > > > > > > > >> > alternatives
> > > > > > > > >> > > > > > section
> > > > > > > > >> > > > > > > > > does
> > > > > > > > >> > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > great
> > > > > > > > >> > > > > > > > > > > > > > > job
> > > > > > > > >> > > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > outlining the
> discussion
> > > so
> > > > > far
> > > > > > > and
> > > > > > > > >> > > providing
> > > > > > > > >> > > > > > > context
> > > > > > > > >> > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > anyone
> > > > > > > > >> > > > > > > > > > > > > > > else
> > > > > > > > >> > > > > > > > > > > > > > > > > who
> > > > > > > > >> > > > > > > > > > > > > > > > > > > might want to join in.
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on choice
> of
> > > > > > > delimiter:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > - I like the
> > optimization
> > > > for
> > > > > > > simple
> > > > > > > > >> > cases,
> > > > > > > > >> > > > > but I
> > > > > > > > >> > > > > > > > think
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > >> > > > > > > > > > > > > > > > > proposal
> > > > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > > > > a little too
> > restrictive.
> > > > What
> > > > > > if
> > > > > > > > >> > there's a
> > > > > > > > >> > > > > field
> > > > > > > > >> > > > > > > > whose
> > > > > > > > >> > > > > > > > > > > name
> > > > > > > > >> > > > > > > > > > > > > > > contains
> > > > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > > > >> > > > > > > > > > > > > > > > > > > of the permitted
> options
> > > > > > > (currently
> > > > > > > > >> just
> > > > > > > > >> > > ".",
> > > > > > > > >> > > > > > ",",
> > > > > > > > >> > > > > > > > and
> > > > > > > > >> > > > > > > > > > > "/")?
> > > > > > > > >> > > > > > > > > > > > > > > > > > > - If we expand the set
> > of
> > > > > > > permitted
> > > > > > > > >> > > > delimiters
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > allow
> > > > > > > > >> > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > any
> > > > > > > > >> > > > > > > > > > > > > > > > > > > single-character
> string,
> > > > > > > > configuration
> > > > > > > > >> > > > > complexity
> > > > > > > > >> > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > increase
> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > readability may
> decrease
> > > > > > > > >> > > > > > > > > > > > > > > > > > > - Also worth pointing
> > out
> > > > that
> > > > > > > there
> > > > > > > > >> is
> > > > > > > > >> > > some
> > > > > > > > >> > > > > > > > convention
> > > > > > > > >> > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > doubling
> > > > > > > > >> > > > > > > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > > > > > delimiter character as
> > an
> > > > > escape
> > > > > > > > >> > mechanism
> > > > > > > > >> > > > with
> > > > > > > > >> > > > > > > > formats
> > > > > > > > >> > > > > > > > > > > like
> > > > > > > > >> > > > > > > > > > > > > CSV
> > > > > > > > >> > > > > > > > > > > > > > > [1]
> > > > > > > > >> > > > > > > > > > > > > > > > > > > - Overall I think we
> may
> > > be
> > > > > > > > >> approaching
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > > saturation
> > > > > > > > >> > > > > > > > > > > point
> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > productive
> > > > > > > > >> > > > > > > > > > > > > > > > > > > discussion on
> delimiter
> > > > syntax
> > > > > > so
> > > > > > > I
> > > > > > > > >> don't
> > > > > > > > >> > > > want
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > spend
> > > > > > > > >> > > > > > > > > > too
> > > > > > > > >> > > > > > > > > > > > > much
> > > > > > > > >> > > > > > > > > > > > > > > more
> > > > > > > > >> > > > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > your time on it. I
> think
> > > the
> > > > > one
> > > > > > > > point
> > > > > > > > >> > I'd
> > > > > > > > >> > > > like
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > > leave
> > > > > > > > >> > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > now
> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > > > > > shouldn't be
> impossible
> > to
> > > > use
> > > > > > > this
> > > > > > > > >> new
> > > > > > > > >> > > > feature
> > > > > > > > >> > > > > > for
> > > > > > > > >> > > > > > > > any
> > > > > > > > >> > > > > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > > name,
> > > > > > > > >> > > > > > > > > > > > > > > > no
> > > > > > > > >> > > > > > > > > > > > > > > > > > > matter how convoluted.
> > > It's
> > > > > fine
> > > > > > > if
> > > > > > > > >> edge
> > > > > > > > >> > > > cases
> > > > > > > > >> > > > > > > > > introduce
> > > > > > > > >> > > > > > > > > > > > > > difficulty
> > > > > > > > >> > > > > > > > > > > > > > > > > (such
> > > > > > > > >> > > > > > > > > > > > > > > > > > > as less-readable
> > > > > > configurations),
> > > > > > > > but
> > > > > > > > >> > it's
> > > > > > > > >> > > > not
> > > > > > > > >> > > > > > fine
> > > > > > > > >> > > > > > > > if
> > > > > > > > >> > > > > > > > > > they
> > > > > > > > >> > > > > > > > > > > > > can't
> > > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > 2.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > The configuration
> style
> > > > where
> > > > > > you
> > > > > > > > >> define
> > > > > > > > >> > > > > > > > > > > > > "transforms.field.style"
> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > connector config, and
> > then
> > > > > this
> > > > > > > > >> applies
> > > > > > > > >> > to
> > > > > > > > >> > > > all
> > > > > > > > >> > > > > > SMTs
> > > > > > > > >> > > > > > > > for
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > connector,
> > > > > > > > >> > > > > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > > > > very interesting.
> > However,
> > > > it
> > > > > > > > doesn't
> > > > > > > > >> > > follow
> > > > > > > > >> > > > > > > > convention
> > > > > > > > >> > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > existing
> > > > > > > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Right now, if you want
> > to
> > > > > > > configure
> > > > > > > > an
> > > > > > > > >> > SMT,
> > > > > > > > >> > > > you
> > > > > > > > >> > > > > > > > define
> > > > > > > > >> > > > > > > > > > its
> > > > > > > > >> > > > > > > > > > > > name
> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > connector config (for
> > > > example,
> > > > > > > > >> > > "transforms":
> > > > > > > > >> > > > > > > "smt1"),
> > > > > > > > >> > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > then
> > > > > > > > >> > > > > > > > > > > > > > > define
> > > > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > > > >> > > > > > > > > > > > > > > > > > > of the properties for
> > that
> > > > SMT
> > > > > > in
> > > > > > > > the
> > > > > > > > >> > > > connector
> > > > > > > > >> > > > > > > > config
> > > > > > > > >> > > > > > > > > > > using
> > > > > > > > >> > > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > > > > namespacing
> > > > > > > > >> > > > > > > > > > > > > > > > > > > mechanism specific to
> > that
> > > > SMT
> > > > > > > (for
> > > > > > > > >> > > example,
> > > > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > > > > > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT then
> > > sees
> > > > > only
> > > > > > > the
> > > > > > > > >> > > > properties
> > > > > > > > >> > > > > > > > defined
> > > > > > > > >> > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > namespace,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > with the prefix
> stripped
> > > > (for
> > > > > > > > example,
> > > > > > > > >> > > > "prop1":
> > > > > > > > >> > > > > > > > "val1")
> > > > > > > > >> > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > its
> > > > > > > > >> > > > > > > > > > > > > > > > > configure
> > > > > > > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > If we want to continue
> > to
> > > > > follow
> > > > > > > > this
> > > > > > > > >> > > > > convention,
> > > > > > > > >> > > > > > > > then
> > > > > > > > >> > > > > > > > > > > > instead
> > > > > > > > >> > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > specifying
> > > > > > > "transforms.field.style"
> > > > > > > > >> in a
> > > > > > > > >> > > > > > connector
> > > > > > > > >> > > > > > > > > > config,
> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > > > > > expect
> > > > > > > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > > > > > > >> > > > > > "transforms.<name>.field.style",
> > > > > > > > >> > > > > > > > for
> > > > > > > > >> > > > > > > > > > > each
> > > > > > > > >> > > > > > > > > > > > > SMT
> > > > > > > > >> > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > they
> > > > > > > > >> > > > > > > > > > > > > > > > > > > want to configure a
> > field
> > > > > style
> > > > > > > for.
> > > > > > > > >> This
> > > > > > > > >> > > > would
> > > > > > > > >> > > > > > > > require
> > > > > > > > >> > > > > > > > > > > more
> > > > > > > > >> > > > > > > > > > > > > work
> > > > > > > > >> > > > > > > > > > > > > > > on
> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > part of the user, but
> > > would
> > > > be
> > > > > > > > >> simpler to
> > > > > > > > >> > > > > reason
> > > > > > > > >> > > > > > > > about
> > > > > > > > >> > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > easier
> > > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > implement.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > If we want to explore
> an
> > > > > > > alternative
> > > > > > > > >> > where
> > > > > > > > >> > > > > users
> > > > > > > > >> > > > > > > can
> > > > > > > > >> > > > > > > > > > > specify
> > > > > > > > >> > > > > > > > > > > > > > global
> > > > > > > > >> > > > > > > > > > > > > > > > > > > properties that apply
> to
> > > all
> > > > > > > > >> transforms
> > > > > > > > >> > in
> > > > > > > > >> > > a
> > > > > > > > >> > > > > > > > connector
> > > > > > > > >> > > > > > > > > > > > config,
> > > > > > > > >> > > > > > > > > > > > > > then
> > > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > semantics for this
> need
> > to
> > > > be
> > > > > > > > defined
> > > > > > > > >> in
> > > > > > > > >> > > the
> > > > > > > > >> > > > > KIP.
> > > > > > > > >> > > > > > > > This
> > > > > > > > >> > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > have
> > > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > include whether this
> > will
> > > > > apply
> > > > > > > only
> > > > > > > > >> for
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > special
> > > > > > > > >> > > > > > > > > case
> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > "field.style" and
> > possibly
> > > > > > > > >> > > "field.separator"
> > > > > > > > >> > > > > > > > properties
> > > > > > > > >> > > > > > > > > > or
> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > >> > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > > > > available more
> generally
> > > for
> > > > > > other
> > > > > > > > >> > > > properties,
> > > > > > > > >> > > > > > > > whether
> > > > > > > > >> > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > > > > apply
> > > > > > > > >> > > > > > > > > > > > > > > > > only
> > > > > > > > >> > > > > > > > > > > > > > > > > > > for the SMTs outlined
> in
> > > the
> > > > > KIP
> > > > > > > or
> > > > > > > > if
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > > > "field.style"
> > > > > > > > >> > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > possibly
> > > > > > > > >> > > > > > > > > > > > > > > > > > > "field.separator"
> > > properties
> > > > > > would
> > > > > > > > >> also
> > > > > > > > >> > be
> > > > > > > > >> > > > > passed
> > > > > > > > >> > > > > > > > into
> > > > > > > > >> > > > > > > > > > > custom
> > > > > > > > >> > > > > > > > > > > > > > SMTs
> > > > > > > > >> > > > > > > > > > > > > > > so
> > > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > they could choose to
> act
> > > on
> > > > > them
> > > > > > > if
> > > > > > > > >> > > > applicable,
> > > > > > > > >> > > > > > how
> > > > > > > > >> > > > > > > > > edge
> > > > > > > > >> > > > > > > > > > > > cases
> > > > > > > > >> > > > > > > > > > > > > > like
> > > > > > > > >> > > > > > > > > > > > > > > > > > having
> > > > > > > > >> > > > > > > > > > > > > > > > > > > an SMT named "field"
> in
> > > your
> > > > > > > > connector
> > > > > > > > >> > > config
> > > > > > > > >> > > > > > would
> > > > > > > > >> > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > handled,
> > > > > > > > >> > > > > > > > > > > > > > > etc.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Either way, it might
> > help
> > > to
> > > > > > have
> > > > > > > an
> > > > > > > > >> > > example
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > KIP
> > > > > > > > >> > > > > > > > > > > > > outlining
> > > > > > > > >> > > > > > > > > > > > > > > how
> > > > > > > > >> > > > > > > > > > > > > > > > > one
> > > > > > > > >> > > > > > > > > > > > > > > > > > > of the to-be-augmented
> > > SMTs
> > > > > can
> > > > > > be
> > > > > > > > >> > > configured
> > > > > > > > >> > > > > > with
> > > > > > > > >> > > > > > > > this
> > > > > > > > >> > > > > > > > > > new
> > > > > > > > >> > > > > > > > > > > > > > feature
> > > > > > > > >> > > > > > > > > > > > > > > > > and a
> > > > > > > > >> > > > > > > > > > > > > > > > > > > before/after of how a
> > > record
> > > > > > value
> > > > > > > > >> would
> > > > > > > > >> > be
> > > > > > > > >> > > > > > > > transformed
> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > >> > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for
> the
> > > > > > > > >> > > > > "transforms.field.style"
> > > > > > > > >> > > > > > > > > > property
> > > > > > > > >> > > > > > > > > > > > > > mentions
> > > > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > the permitted values
> are
> > > > > "plain"
> > > > > > > and
> > > > > > > > >> > > > "nested",
> > > > > > > > >> > > > > > but
> > > > > > > > >> > > > > > > > then
> > > > > > > > >> > > > > > > > > > > > > describes
> > > > > > > > >> > > > > > > > > > > > > > > > > > behavior
> > > > > > > > >> > > > > > > > > > > > > > > > > > > with a value of
> "root".
> > > > Should
> > > > > > > that
> > > > > > > > be
> > > > > > > > >> > > > "plain"
> > > > > > > > >> > > > > > > > instead?
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for
> the
> > > > > > > > >> > > > > > > "transforms.field.separator"
> > > > > > > > >> > > > > > > > > > > > property
> > > > > > > > >> > > > > > > > > > > > > > > > > > exclusively
> > > > > > > > >> > > > > > > > > > > > > > > > > > > mentions structs, but
> > the
> > > > > > feature
> > > > > > > is
> > > > > > > > >> > > intended
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > work
> > > > > > > > >> > > > > > > > > > with
> > > > > > > > >> > > > > > > > > > > > maps
> > > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > > >> > > > > > > > > > > > > > > > > well.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Can we update it to
> > > reflect
> > > > > > this?
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > References:
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > > > > > > >> > https://stackoverflow.com/a/17808731
> > > > > > > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > > > > > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022
> at
> > > 1:32
> > > > > PM
> > > > > > > > Jorge
> > > > > > > > >> > > Esteban
> > > > > > > > >> > > > > > > > Quilcate
> > > > > > > > >> > > > > > > > > > > Otoya
> > > > > > > > >> > > > > > > > > > > > <
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > quilcate.jorge@gmail.com>
> > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> > > > > > "this..field.child"
> > > > > > > > >> could
> > > > > > > > >> > be
> > > > > > > > >> > > > > > harder
> > > > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > > > > > grasp
> > > > > > > > >> > > > > > > > > > > > > than
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > "this.field/child" +
> > > > > > separator:
> > > > > > > > "/".
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > Even though this
> > > > represents
> > > > > > > > >> additional
> > > > > > > > >> > > > > > > information,
> > > > > > > > >> > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > follows
> > > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > > > > similar
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > > > > > > "Flatten#delimeter"
> > > > > > > > >> > > > > > > configuration.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > I want to give the
> > > > separator
> > > > > > > > >> approach
> > > > > > > > >> > > > another
> > > > > > > > >> > > > > > > try,
> > > > > > > > >> > > > > > > > > so I
> > > > > > > > >> > > > > > > > > > > > have
> > > > > > > > >> > > > > > > > > > > > > > > > updated
> > > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > KIP with the
> separator
> > > > > > proposal,
> > > > > > > > >> > sticking
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > only 2
> > > > > > > > >> > > > > > > > > > > > > > alternatives
> > > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > should hopefully
> cover
> > > > most
> > > > > > > > >> scenarios.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has
> been
> > > > > updated
> > > > > > > > with
> > > > > > > > >> > this
> > > > > > > > >> > > > > > > > improvement.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I
> > have
> > > > > > updated
> > > > > > > > this
> > > > > > > > >> > > > section
> > > > > > > > >> > > > > > > > > > accordingly.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch! I've
> > > > replaced
> > > > > > it
> > > > > > > > with
> > > > > > > > >> > > > > > > `DropHeaders`.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > Looking forward to
> > your
> > > > > > > feedback.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022
> at
> > > > 21:33,
> > > > > > > Chris
> > > > > > > > >> > > Egerton
> > > > > > > > >> > > > <
> > > > > > > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Looking good! Got
> a
> > > few
> > > > > more
> > > > > > > > >> > thoughts.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to
> revisit
> > > > this,
> > > > > > but
> > > > > > > I
> > > > > > > > >> think
> > > > > > > > >> > > we
> > > > > > > > >> > > > > may
> > > > > > > > >> > > > > > > want
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > adopt
> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > > > slightly
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > different escape
> > > syntax
> > > > > > style.
> > > > > > > > >> > > > Backslashes
> > > > > > > > >> > > > > > are
> > > > > > > > >> > > > > > > > > great,
> > > > > > > > >> > > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > > > since
> > > > > > > > >> > > > > > > > > > > > > > > > > > they're
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > already used by
> > JSON,
> > > > > using
> > > > > > > them
> > > > > > > > >> as
> > > > > > > > >> > an
> > > > > > > > >> > > > > escape
> > > > > > > > >> > > > > > > > > > sequence
> > > > > > > > >> > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > > > > > > > notation
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > would also lead to
> > > some
> > > > > > pretty
> > > > > > > > >> ugly
> > > > > > > > >> > > > > connector
> > > > > > > > >> > > > > > > > > > configs.
> > > > > > > > >> > > > > > > > > > > > > Anyone
> > > > > > > > >> > > > > > > > > > > > > > > > who's
> > > > > > > > >> > > > > > > > > > > > > > > > > > had
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > write regular
> > > > expressions
> > > > > > with
> > > > > > > > >> > > > backslashes
> > > > > > > > >> > > > > in
> > > > > > > > >> > > > > > > > Java
> > > > > > > > >> > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > probably
> > > > > > > > >> > > > > > > > > > > > > > > > > > already
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > familiar with
> this:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > > > > > > >> > > > > > > > > > > > > > > > > What
> > > > > > > > >> > > > > > > > > > > > > > > > > > > do
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > you think about
> > using
> > > > the
> > > > > > dot
> > > > > > > > >> > character
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > escape
> > > > > > > > >> > > > > > > > > > > itself?
> > > > > > > > >> > > > > > > > > > > > > In
> > > > > > > > >> > > > > > > > > > > > > > > > other
> > > > > > > > >> > > > > > > > > > > > > > > > > > > words,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > to access a single
> > > field
> > > > > > named
> > > > > > > > >> > > > > "this.field",
> > > > > > > > >> > > > > > > > > instead
> > > > > > > > >> > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > using
> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > syntax
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > "this\.field"
> (which
> > > in
> > > > > JSON
> > > > > > > > would
> > > > > > > > >> > have
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > > > > > expressed
> > > > > > > > >> > > > > > > > > > > > as
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > > > "this..field",
> > > > > > > and
> > > > > > > > >> for a
> > > > > > > > >> > > > > single
> > > > > > > > >> > > > > > > > field
> > > > > > > > >> > > > > > > > > > > named
> > > > > > > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > instead of using
> the
> > > > > syntax
> > > > > > > > >> > > "this\\field"
> > > > > > > > >> > > > > > (or,
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > > JSON,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > > "this\field"
> > > > > > (or,
> > > > > > > > in
> > > > > > > > >> > JSON,
> > > > > > > > >> > > > > > > > > > "this\\field").
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you flesh
> > out
> > > > the
> > > > > > > > >> details on
> > > > > > > > >> > > the
> > > > > > > > >> > > > > new
> > > > > > > > >> > > > > > > > > > > > "field.style"
> > > > > > > > >> > > > > > > > > > > > > > > > > property,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > including the
> type,
> > > > > default
> > > > > > > > value,
> > > > > > > > >> > > > > > importance,
> > > > > > > > >> > > > > > > > and
> > > > > > > > >> > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > preliminary
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > See
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the
> > > > "Compatibility,
> > > > > > > > >> > Deprecation,
> > > > > > > > >> > > > and
> > > > > > > > >> > > > > > > > > Migration
> > > > > > > > >> > > > > > > > > > > > Plan"
> > > > > > > > >> > > > > > > > > > > > > > > > section
> > > > > > > > >> > > > > > > > > > > > > > > > > > > still
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > accurate after the
> > > > latest
> > > > > > > > update?
> > > > > > > > >> > Seems
> > > > > > > > >> > > > > like
> > > > > > > > >> > > > > > > it's
> > > > > > > > >> > > > > > > > > > still
> > > > > > > > >> > > > > > > > > > > > > > written
> > > > > > > > >> > > > > > > > > > > > > > > > > with
> > > > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > assumption that
> > nested
> > > > > field
> > > > > > > > >> syntax
> > > > > > > > >> > > will
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > > > > hardcoded
> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > >> > > > > > > > > > > > > > > opt-in,
> > > > > > > > >> > > > > > > > > > > > > > > > > > which
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > isn't the case
> > > anymore.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The "These
> > > SMTs
> > > > do
> > > > > > not
> > > > > > > > >> > require
> > > > > > > > >> > > > > nested
> > > > > > > > >> > > > > > > > > > structure
> > > > > > > > >> > > > > > > > > > > > > > > support"
> > > > > > > > >> > > > > > > > > > > > > > > > > > > section
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > mentions a "Drop"
> > > SMT. I
> > > > > > think
> > > > > > > > >> this
> > > > > > > > >> > may
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > > > > referring
> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > Confluent
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > SMT, which isn't a
> > > part
> > > > of
> > > > > > > > Apache
> > > > > > > > >> > > Kafka.
> > > > > > > > >> > > > > > Should
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > > drop
> > > > > > > > >> > > > > > > > > > > > > (heh)
> > > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > SMT
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > the list? Or
> perhaps
> > > > just
> > > > > > > > replace
> > > > > > > > >> it
> > > > > > > > >> > > with
> > > > > > > > >> > > > > > > > > > > "DropHeaders",
> > > > > > > > >> > > > > > > > > > > > > > which
> > > > > > > > >> > > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > currently
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > missing from the
> > list
> > > > and
> > > > > > > > >> shouldn't
> > > > > > > > >> > > > require
> > > > > > > > >> > > > > > any
> > > > > > > > >> > > > > > > > > > > > > nested-field
> > > > > > > > >> > > > > > > > > > > > > > > > > related
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28,
> 2022
> > > at
> > > > > 2:12
> > > > > > > PM
> > > > > > > > >> Jorge
> > > > > > > > >> > > > > Esteban
> > > > > > > > >> > > > > > > > > > Quilcate
> > > > > > > > >> > > > > > > > > > > > > Otoya
> > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > quilcate.jorge@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thank you,
> Chris!
> > > and
> > > > > > sorry
> > > > > > > > for
> > > > > > > > >> the
> > > > > > > > >> > > > > delayed
> > > > > > > > >> > > > > > > > > > response.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Please, find my
> > > > comments
> > > > > > > > below:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb
> > 2022
> > > at
> > > > > > > 17:34,
> > > > > > > > >> Chris
> > > > > > > > >> > > > > Egerton
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > <chrise@confluent.io.invalid
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> > > KIP!
> > > > > I'd
> > > > > > > love
> > > > > > > > >> to
> > > > > > > > >> > see
> > > > > > > > >> > > > > > support
> > > > > > > > >> > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > fields
> > > > > > > > >> > > > > > > > > > > > > > > > > > added
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > out-of-the-box
> > > SMTs
> > > > > > > provided
> > > > > > > > >> with
> > > > > > > > >> > > > > > Connect.
> > > > > > > > >> > > > > > > > Here
> > > > > > > > >> > > > > > > > > > are
> > > > > > > > >> > > > > > > > > > > > my
> > > > > > > > >> > > > > > > > > > > > > > > > initial
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree
> that
> > > > > there's
> > > > > > a
> > > > > > > > >> case to
> > > > > > > > >> > > be
> > > > > > > > >> > > > > made
> > > > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > > > > > > > expanding
> > > > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > new config
> > > property
> > > > > for
> > > > > > > > >> > > identifying a
> > > > > > > > >> > > > > > > nested,
> > > > > > > > >> > > > > > > > > > > > > > to-be-hoisted
> > > > > > > > >> > > > > > > > > > > > > > > > > > field,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example in the
> > KIP
> > > > > > doesn't
> > > > > > > > >> really
> > > > > > > > >> > > > > > > demonstrate
> > > > > > > > >> > > > > > > > > why
> > > > > > > > >> > > > > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd be
> > > > helpful
> > > > > to
> > > > > > > > >> expand
> > > > > > > > >> > the
> > > > > > > > >> > > > > > example
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > add
> > > > > > > > >> > > > > > > > > > > > > other
> > > > > > > > >> > > > > > > > > > > > > > > > fields
> > > > > > > > >> > > > > > > > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > order
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > show how
> adding
> > > > nested
> > > > > > > field
> > > > > > > > >> > > support
> > > > > > > > >> > > > > > > enables
> > > > > > > > >> > > > > > > > > > users
> > > > > > > > >> > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > hoist a
> > > > > > > > >> > > > > > > > > > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > without
> dropping
> > > > other
> > > > > > > > fields
> > > > > > > > >> > from
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > value.
> > > > > > > > >> > > > > > > > > > Maybe
> > > > > > > > >> > > > > > > > > > > > > > > something
> > > > > > > > >> > > > > > > > > > > > > > > > > > like
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     source =
> > > > > nested.val
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     field =
> line
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> > > (before):
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "nested":
> > > > > {
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "val":
> > > > > > 42,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "other
> > > > > > > val":
> > > > > > > > >> 96
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> > (after):
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "nested":
> > > > > {
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "line":
> > > > > > {
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >  "val":
> > > > > > > > 42,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  }
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "other
> > > > > > > val":
> > > > > > > > >> 96
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I
> think
> > > > > "source"
> > > > > > > is
> > > > > > > > a
> > > > > > > > >> > > little
> > > > > > > > >> > > > > > > strange
> > > > > > > > >> > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > new
> > > > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property name.
> > > Maybe
> > > > > > > > >> "hoisted" or
> > > > > > > > >> > > > > > > > > "hoisted.field"
> > > > > > > > >> > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and 2.:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The
> example
> > > for
> > > > > > this
> > > > > > > > SMT
> > > > > > > > >> is
> > > > > > > > >> > > > > updated
> > > > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > > > > have
> > > > > > > > >> > > > > > > > > > > > added
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there a
> > > > > reasonable
> > > > > > > use
> > > > > > > > >> case
> > > > > > > > >> > > for
> > > > > > > > >> > > > > > > > expanding
> > > > > > > > >> > > > > > > > > > > > Flatten
> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten
> specific
> > > > > fields?
> > > > > > > My
> > > > > > > > >> > > > > understanding
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > it's
> > > > > > > > >> > > > > > > > > > > > > > > mostly
> > > > > > > > >> > > > > > > > > > > > > > > > > > > useful
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > writing to
> > systems
> > > > > like
> > > > > > > > >> databases
> > > > > > > > >> > > > that
> > > > > > > > >> > > > > > > don't
> > > > > > > > >> > > > > > > > > > > support
> > > > > > > > >> > > > > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > > > > values
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > require
> > everything
> > > > to
> > > > > > be a
> > > > > > > > >> flat
> > > > > > > > >> > > list
> > > > > > > > >> > > > of
> > > > > > > > >> > > > > > > > > key-value
> > > > > > > > >> > > > > > > > > > > > > pairs.
> > > > > > > > >> > > > > > > > > > > > > > > > Being
> > > > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a
> nested
> > > > field
> > > > > > > > >> wouldn't
> > > > > > > > >> > > > provide
> > > > > > > > >> > > > > > any
> > > > > > > > >> > > > > > > > > > > advantage
> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > use
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Are there
> other
> > > > cases
> > > > > > > where
> > > > > > > > it
> > > > > > > > >> > > would?
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't
> think
> > > we
> > > > > > should
> > > > > > > > >> > > > > > unconditionally
> > > > > > > > >> > > > > > > > > change
> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > default
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten. It's
> a
> > > > > > > > >> > > > backwards-incompatible,
> > > > > > > > >> > > > > > > > > breaking
> > > > > > > > >> > > > > > > > > > > > change
> > > > > > > > >> > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > could
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > headaches for
> > > users.
> > > > > It
> > > > > > > > might
> > > > > > > > >> be
> > > > > > > > >> > > > > > reasonable
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > > change
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > value
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically
> > based
> > > on
> > > > > > > whether
> > > > > > > > >> the
> > > > > > > > >> > > user
> > > > > > > > >> > > > > has
> > > > > > > > >> > > > > > > > > > > specified a
> > > > > > > > >> > > > > > > > > > > > > > value
> > > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property, but
> > > > > > considering
> > > > > > > > the
> > > > > > > > >> > > > > motivation
> > > > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > > > > > > changing
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > > > >> > > > > > > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it creates
> > > conflicts
> > > > > > with
> > > > > > > > the
> > > > > > > > >> > > > > > > > to-be-introduced
> > > > > > > > >> > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > > > > > > syntax
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > could arise
> with
> > > > > > > downstream
> > > > > > > > >> SMTs
> > > > > > > > >> > > > > > regardless
> > > > > > > > >> > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > whether
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > user
> > > > > > > > >> > > > > > > > > > > > > > > > > > has
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly
> > > > configured
> > > > > > > > Flatten
> > > > > > > > >> > with
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > "field"
> > > > > > > > >> > > > > > > > > > > > > > property), I
> > > > > > > > >> > > > > > > > > > > > > > > > > don't
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > know
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this would be
> > too
> > > > > useful
> > > > > > > > >> either.
> > > > > > > > >> > I
> > > > > > > > >> > > > have
> > > > > > > > >> > > > > > > some
> > > > > > > > >> > > > > > > > > > > thoughts
> > > > > > > > >> > > > > > > > > > > > > > below
> > > > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > > > >> > > > > > > > > > > > > > > > > > how
> > > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > handle
> possible
> > > > > > conflicts
> > > > > > > > >> between
> > > > > > > > >> > > > names
> > > > > > > > >> > > > > > > with
> > > > > > > > >> > > > > > > > > dots
> > > > > > > > >> > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > their
> > > > > > > > >> > > > > > > > > > > > > > > > > names
> > > > > > > > >> > > > > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for
> > nested
> > > > > field
> > > > > > > > >> > references
> > > > > > > > >> > > > that
> > > > > > > > >> > > > > > > > should
> > > > > > > > >> > > > > > > > > > > > > hopefully
> > > > > > > > >> > > > > > > > > > > > > > > make
> > > > > > > > >> > > > > > > > > > > > > > > > > > > either
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough.
> With
> > > the
> > > > > > > support
> > > > > > > > >> for
> > > > > > > > >> > > > nested
> > > > > > > > >> > > > > > > fields
> > > > > > > > >> > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > other
> > > > > > > > >> > > > > > > > > > > > > > > SMTs,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > could stay as it
> > is.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > This removes the
> > > need
> > > > > for
> > > > > > > (4)
> > > > > > > > >> > > changing
> > > > > > > > >> > > > > > > Flatten
> > > > > > > > >> > > > > > > > > > config
> > > > > > > > >> > > > > > > > > > > > as
> > > > > > > > >> > > > > > > > > > > > > > > well.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think
> it's
> > > fine
> > > > > to
> > > > > > > > expand
> > > > > > > > >> > > > > > ExtractField
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > support
> > > > > > > > >> > > > > > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it might be
> > worth
> > > > > noting
> > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > > > rejected
> > > > > > > > >> > > > > > > > > > > alternatives
> > > > > > > > >> > > > > > > > > > > > > > > section
> > > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > isn't strictly
> > > > > necessary
> > > > > > > > since
> > > > > > > > >> > you
> > > > > > > > >> > > > can
> > > > > > > > >> > > > > > > > replace
> > > > > > > > >> > > > > > > > > > any
> > > > > > > > >> > > > > > > > > > > > > single
> > > > > > > > >> > > > > > > > > > > > > > > > > > > invocation
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT that
> > uses
> > > > > > nested
> > > > > > > > >> field
> > > > > > > > >> > > > > notation
> > > > > > > > >> > > > > > > with
> > > > > > > > >> > > > > > > > > > > > multiple
> > > > > > > > >> > > > > > > > > > > > > > > > > > invocations
> > > > > > > > >> > > > > > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that use
> > > non-nested
> > > > > > > > notation.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding
> it.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit:
> > > > "RegerRouter"
> > > > > > > should
> > > > > > > > >> be
> > > > > > > > >> > > > > > > "RegexRouter"
> > > > > > > > >> > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > list
> > > > > > > > >> > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > SMTs
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > not require
> > nested
> > > > > > > structure
> > > > > > > > >> > > support.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing it.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be
> > rare
> > > > for
> > > > > > dots
> > > > > > > > in
> > > > > > > > >> > field
> > > > > > > > >> > > > > names
> > > > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > > > > > occur
> > > > > > > > >> > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > wild
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be so
> > > > certain
> > > > > > of
> > > > > > > > >> this),
> > > > > > > > >> > > but
> > > > > > > > >> > > > > > unless
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > > want
> > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > inflict
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > users of
> > Flatten,
> > > I
> > > > > > think
> > > > > > > > >> we're
> > > > > > > > >> > > going
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > have
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > think
> > > > > > > > >> > > > > > > > > > > > > > > about
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between dotted
> > > > > notation
> > > > > > > and
> > > > > > > > >> > > > non-nested
> > > > > > > > >> > > > > > > fields
> > > > > > > > >> > > > > > > > > > whose
> > > > > > > > >> > > > > > > > > > > > > names
> > > > > > > > >> > > > > > > > > > > > > > > > > contain
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > don't think
> this
> > > is
> > > > > > > actually
> > > > > > > > >> > such a
> > > > > > > > >> > > > bad
> > > > > > > > >> > > > > > > > thing,
> > > > > > > > >> > > > > > > > > > > > though.
> > > > > > > > >> > > > > > > > > > > > > I
> > > > > > > > >> > > > > > > > > > > > > > > > agree
> > > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > notation is
> > > > intuitive
> > > > > > and
> > > > > > > > >> pretty
> > > > > > > > >> > > > > > > commonplace
> > > > > > > > >> > > > > > > > > (in
> > > > > > > > >> > > > > > > > > > > > tools
> > > > > > > > >> > > > > > > > > > > > > > like
> > > > > > > > >> > > > > > > > > > > > > > > > jq,
> > > > > > > > >> > > > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example), so
> I'd
> > > > like
> > > > > it
> > > > > > > if
> > > > > > > > we
> > > > > > > > >> > > could
> > > > > > > > >> > > > > > stick
> > > > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > > > > it.
> > > > > > > > >> > > > > > > > > > > > What
> > > > > > > > >> > > > > > > > > > > > > > > about
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > escape syntax,
> > > > using a
> > > > > > > > >> backslash?
> > > > > > > > >> > > > That
> > > > > > > > >> > > > > > way,
> > > > > > > > >> > > > > > > > > users
> > > > > > > > >> > > > > > > > > > > > could
> > > > > > > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between
> > > "this.field"
> > > > > > > (which
> > > > > > > > >> would
> > > > > > > > >> > > > refer
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > > >> > > > > > > > > > > > > > > > > > "field"
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > under
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the top-level
> > > "this"
> > > > > > > field),
> > > > > > > > >> and
> > > > > > > > >> > > > > > > > "this\.field"
> > > > > > > > >> > > > > > > > > > > (which
> > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > > > > refer
> > > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> > > > > > "this.field").
> > > > > > > > >> Like
> > > > > > > > >> > > with
> > > > > > > > >> > > > > most
> > > > > > > > >> > > > > > > > > > languages
> > > > > > > > >> > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > > use
> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > for escape
> > > > sequences,
> > > > > it
> > > > > > > > could
> > > > > > > > >> > also
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > used
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > > escape
> > > > > > > > >> > > > > > > > > > > > > > > itself,
> > > > > > > > >> > > > > > > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that a field
> > name
> > > > > > > contains a
> > > > > > > > >> > > > > backslash. I
> > > > > > > > >> > > > > > > > think
> > > > > > > > >> > > > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > more
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > simpler than,
> > > e.g.,
> > > > > > > adding a
> > > > > > > > >> new
> > > > > > > > >> > > > config
> > > > > > > > >> > > > > > > > > property
> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > toggle
> > > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > be used when
> > > parsing
> > > > > > > nested
> > > > > > > > >> field
> > > > > > > > >> > > > > > > references.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like this
> > > approach.
> > > > > > Adding
> > > > > > > > to
> > > > > > > > >> the
> > > > > > > > >> > > > KIP.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't
> think
> > > we
> > > > > can
> > > > > > > > >> > > > unconditionally
> > > > > > > > >> > > > > > > turn
> > > > > > > > >> > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > > feature
> > > > > > > > >> > > > > > > > > > > > > > > on.
> > > > > > > > >> > > > > > > > > > > > > > > > > The
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > risk
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > breaking
> > existing
> > > > > > > pipelines
> > > > > > > > >> > > > (especially
> > > > > > > > >> > > > > > > ones
> > > > > > > > >> > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > involve,
> > > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > combination of
> > the
> > > > > > Flatten
> > > > > > > > and
> > > > > > > > >> > Cast
> > > > > > > > >> > > > > SMTs)
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > > > pretty
> > > > > > > > >> > > > > > > > > > > > > > high. I
> > > > > > > > >> > > > > > > > > > > > > > > > > think
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > should be an
> > > opt-in
> > > > > > > feature,
> > > > > > > > >> at
> > > > > > > > >> > > least
> > > > > > > > >> > > > > > until
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > > next
> > > > > > > > >> > > > > > > > > > > > > > major
> > > > > > > > >> > > > > > > > > > > > > > > > > > release.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > One
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > we could
> > > accomplish
> > > > > this
> > > > > > > is
> > > > > > > > by
> > > > > > > > >> > > > > > introducing
> > > > > > > > >> > > > > > > a
> > > > > > > > >> > > > > > > > > new
> > > > > > > > >> > > > > > > > > > > > > > > > "field.style"
> > > > > > > > >> > > > > > > > > > > > > > > > > > > (name
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > obviously
> > subject
> > > to
> > > > > > > change)
> > > > > > > > >> > > property
> > > > > > > > >> > > > > > with
> > > > > > > > >> > > > > > > > > values
> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > "plain"
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If
> set
> > > to
> > > > > > > "plain"
> > > > > > > > >> then
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > current
> > > > > > > > >> > > > > > > > > > > > non-nested
> > > > > > > > >> > > > > > > > > > > > > > > > > behavior
> > > > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > and if set to
> > > > > "nested",
> > > > > > > then
> > > > > > > > >> the
> > > > > > > > >> > > > > proposed
> > > > > > > > >> > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > behavior
> > > > > > > > >> > > > > > > > > > > > > > > is.
> > > > > > > > >> > > > > > > > > > > > > > > > > We
> > > > > > > > >> > > > > > > > > > > > > > > > > > > can
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > consider
> > updating
> > > > the
> > > > > > > > default
> > > > > > > > >> > value
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > > "nested"
> > > > > > > > >> > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > future
> > > > > > > > >> > > > > > > > > > > > > > > > > major
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > release
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > (or even
> codify
> > > that
> > > > > > plan
> > > > > > > in
> > > > > > > > >> this
> > > > > > > > >> > > > KIP,
> > > > > > > > >> > > > > if
> > > > > > > > >> > > > > > > > > there's
> > > > > > > > >> > > > > > > > > > > > > enough
> > > > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > > > >> > > > > > > > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > This would
> also
> > > > leave
> > > > > > the
> > > > > > > > door
> > > > > > > > >> > open
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > > adding
> > > > > > > > >> > > > > > > > > > > > > recursive
> > > > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > > > >> > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the future by
> > > > adding a
> > > > > > > > >> permitted
> > > > > > > > >> > > > value
> > > > > > > > >> > > > > of
> > > > > > > > >> > > > > > > > > > > > "recursive".
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of the
> > > linked
> > > > > > > tickets
> > > > > > > > >> in
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > > > "Motivation"
> > > > > > > > >> > > > > > > > > > > > > section,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-10640
> > > > > > ,
> > > > > > > > >> > > > > > > > > > > > has
> > > > > > > > >> > > > > > > > > > > > > > an
> > > > > > > > >> > > > > > > > > > > > > > > > open
> > > > > > > > >> > > > > > > > > > > > > > > > > > PR
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that propose
> > > adding
> > > > > > > > recursive
> > > > > > > > >> > > support
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > some
> > > > > > > > >> > > > > > > > > > SMTs.
> > > > > > > > >> > > > > > > > > > > > > Have
> > > > > > > > >> > > > > > > > > > > > > > > you
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > considered
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> > > > > > functionality
> > > > > > > > for
> > > > > > > > >> > your
> > > > > > > > >> > > > > KIP?
> > > > > > > > >> > > > > > Or
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > > your
> > > > > > > > >> > > > > > > > > > > > aim
> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > stick
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > solely
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > nested field
> > > > support?
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like the
> > > > `field.style`
> > > > > > > > >> > > configuration
> > > > > > > > >> > > > > flag
> > > > > > > > >> > > > > > > > > > approach.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for
> > pointing
> > > > out
> > > > > > the
> > > > > > > > >> > > `recursive`
> > > > > > > > >> > > > > > > > approach.
> > > > > > > > >> > > > > > > > > > > Will
> > > > > > > > >> > > > > > > > > > > > > add
> > > > > > > > >> > > > > > > > > > > > > > > > > `nested`
> > > > > > > > >> > > > > > > > > > > > > > > > > > > at
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's
> > check
> > > > the
> > > > > > > demand
> > > > > > > > >> for
> > > > > > > > >> > > > > > > `recursive`
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > consider
> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > > as
> > > > > > > > >> > > > > > > > > > > > > > > > > > part
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > this or another
> > KIP.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I have added the
> > > > > following
> > > > > > > on
> > > > > > > > >> the
> > > > > > > > >> > > KIP:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs
> could
> > > > extend
> > > > > > > this
> > > > > > > > >> > support
> > > > > > > > >> > > > > for:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive
> > > notation:
> > > > > > name a
> > > > > > > > >> field
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > apply
> > > > > > > > >> > > > > > > it
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > all
> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > >> > > > > > > > > > > > > > > > > > across
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > schema matching
> > that
> > > > > name.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Access to
> > arrays:
> > > > > Adding
> > > > > > > > `[]`
> > > > > > > > >> > > > notation
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > > > represent
> > > > > > > > >> > > > > > > > > > > > > > access
> > > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > > > > > > arrays
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs to
> > > > fields
> > > > > > > within
> > > > > > > > >> an
> > > > > > > > >> > > > array.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 8,
> > > 2022
> > > > at
> > > > > > > 1:23
> > > > > > > > PM
> > > > > > > > >> > > Jorge
> > > > > > > > >> > > > > > > Esteban
> > > > > > > > >> > > > > > > > > > > Quilcate
> > > > > > > > >> > > > > > > > > > > > > > > Otoya <
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > quilcate.jorge@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev team,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like to
> > > start
> > > > a
> > > > > > new
> > > > > > > > >> > > discussion
> > > > > > > > >> > > > > > thread
> > > > > > > > >> > > > > > > > on
> > > > > > > > >> > > > > > > > > > > Kafka
> > > > > > > > >> > > > > > > > > > > > > > > Connect
> > > > > > > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP is
> > > aimed
> > > > to
> > > > > > > > include
> > > > > > > > >> > > > support
> > > > > > > > >> > > > > > for
> > > > > > > > >> > > > > > > > > nested
> > > > > > > > >> > > > > > > > > > > > > > > structures
> > > > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs — where
> > > this
> > > > > make
> > > > > > > > >> sense.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking
> > forward
> > > to
> > > > > > your
> > > > > > > > >> > feedback.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Robert Yokota <ra...@gmail.com>]

Also, a double backslash can be used to escape backslash.

Robert

On Fri, Aug 12, 2022 at 10:56 AM Jorge Esteban Quilcate Otoya <
quilcate.jorge@gmail.com> wrote:

> Awesome, thanks Robert!
> Will work on updated the proposal around this. Also, @Chris Egerton
> <fe...@gmail.com> if you have any feedback about this, would be
> much appreciated.
>
> Cheers,
> Jorge.
>
> On Fri, 12 Aug 2022 at 18:45, Robert Yokota <ra...@gmail.com> wrote:
>
> > Hi Jorge,
> >
> > Yes, to escape a backtick I would recommend a backslash.
> >
> > For example, MySQL allows identifiers to be surrounded with single
> quotes,
> > double quotes, or backticks, and they use backslash to escape.
> >
> > Jsonata went with primarily backticks as they are less common in strings
> > that appear in JSON, and it seems to have worked nicely for them.
> >
> > Regards,
> > Robert
> >
> > On Fri, Aug 12, 2022 at 10:01 AM Jorge Esteban Quilcate Otoya <
> > quilcate.jorge@gmail.com> wrote:
> >
> > > Thanks Robert! And sorry for the late reply.
> > >
> > > That's a great catch and will require the current proposal to be
> updated.
> > >
> > > Using back-ticks is a good proposal. Though, the challenge we got is
> that
> > > JSON allows really any character into attribute names, including
> > > back-ticks.
> > > I wonder what's the best approach to escape back-ticks then. Given that
> > > those use-cases should be minimal we could fallback to backslash to
> > escape
> > > back-ticks?
> > >
> > > What do you think?
> > >
> > > Many thanks,
> > > Jorge.
> > >
> > >
> > > On Wed, 20 Jul 2022 at 23:03, Robert Yokota <ra...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm late to this thread, but would like to comment on the double
> > > dot/double
> > > > asterisk syntax.
> > > >
> > > > Unfortunately double dot is often used in JSON Path as a descendant
> > > > selector, see
> https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html
> > > >
> > > > I think a better notation would be to use backticks to quote any
> names
> > > with
> > > > a dot, such as  root.`child.with.dot`.  See Jsonata syntax for
> example:
> > > > http://docs.jsonata.org/simple
> > > >
> > > > Robert
> > > >
> > > > On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
> > > > quilcate.jorge@gmail.com> wrote:
> > > >
> > > > > Thanks Chris! I have updated the KIP to include this fix.
> > > > >
> > > > > I will keep the array as a potential improvement at the moment, and
> > out
> > > > of
> > > > > the scope of this KIP.
> > > > >
> > > > > Thanks,
> > > > > Jorge.
> > > > >
> > > > > On Tue, 28 Jun 2022 at 23:19, Chris Egerton <
> fearthecellos@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jorge,
> > > > > >
> > > > > > Apologies for the long delay, had my own KIP-related work to
> focus
> > > on.
> > > > > >
> > > > > > I think it's fine to include array accesses but it's not a
> blocker.
> > > I'm
> > > > > +1
> > > > > > either way. On that front though, I think the MaskField section
> > might
> > > > > need
> > > > > > to be updated as it still mentions arrays and deep-scan?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >
> > > > > > > Hi there,
> > > > > > >
> > > > > > > I have update the KIP to the previous state voted, including
> the
> > > > > > > configuration change from `field.style` to
> > `field.syntax.version`.
> > > > > > >
> > > > > > > I'll bump the vote thread as well to see if there's agreement
> on
> > > > adding
> > > > > > > this feature to Connect.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jorge.
> > > > > > >
> > > > > > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > >
> > > > > > > > Thanks, Chris. Your feedback is much appreciated!
> > > > > > > >
> > > > > > > > I see how the current proposal might be underestimating some
> > edge
> > > > > > cases.
> > > > > > > > I'm happy to move the design for deep-scan and multi-values
> to
> > > > future
> > > > > > > > developments related with this KIP and reduce its scope,
> though
> > > > open
> > > > > > for
> > > > > > > > more feedback.
> > > > > > > >
> > > > > > > > Also, just to be sure, are you proposing also to not include
> > > array
> > > > > > access
> > > > > > > > at this stage?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jorge.
> > > > > > > >
> > > > > > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <
> > > > fearthecellos@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Jorge,
> > > > > > > >>
> > > > > > > >> I've done some more thinking and I hate to say it, but I
> think
> > > the
> > > > > > > syntax
> > > > > > > >> does need to be expanded. Right now it's clear what "a.b"
> > refers
> > > > to
> > > > > > and
> > > > > > > >> what "a..b" refers to, but what about "a...b"? Is that
> > referring
> > > > to
> > > > > > > >> subfield ".b" of field "a", or subfield "b" of field "a."?
> > This
> > > > gets
> > > > > > > even
> > > > > > > >> more complicated when thinking about fields whose names are
> > > > > > exclusively
> > > > > > > >> made up of dots.
> > > > > > > >>
> > > > > > > >> I'm also a little hesitant to mix the cases of multi-value
> > paths
> > > > and
> > > > > > > deep
> > > > > > > >> scans. What if you only want to access one subfield deep for
> > an
> > > > SMT,
> > > > > > > >> instead of recursing through all the children of a given
> > field?
> > > > It's
> > > > > > > akin
> > > > > > > >> to the distinction between * and ** with file globbing
> > patterns,
> > > > and
> > > > > > > there
> > > > > > > >> could be a substantial performance difference if you have
> > > > > > heavily-nested
> > > > > > > >> fields.
> > > > > > > >>
> > > > > > > >> Ultimately, I think that if the proposed
> > "field.syntax.version"
> > > > > > property
> > > > > > > >> sits well with people, it might be better to reduce the
> scope
> > of
> > > > the
> > > > > > KIP
> > > > > > > >> back to the original proposal and just focus on adding
> support
> > > for
> > > > > > > >> explicitly-specified nested values, with no multi-value
> paths
> > > > > > > whatsoever,
> > > > > > > >> knowing that we have an easy way to introduce new syntax and
> > > > > features
> > > > > > in
> > > > > > > >> the future. (We could probably leave the "a...b" case for
> that
> > > > next
> > > > > > > >> version
> > > > > > > >> too.)
> > > > > > > >>
> > > > > > > >> I was a huge fan of this KIP before we started trying to
> > address
> > > > > more
> > > > > > > >> complex use cases, and although I don't want to write those
> > > off, I
> > > > > > think
> > > > > > > >> we
> > > > > > > >> may have bitten off more than we can chew in time for the
> > 3.3.0
> > > > > > release
> > > > > > > >> and
> > > > > > > >> would hate to see this KIP get delayed as a result.
> > > > > > > >>
> > > > > > > >> I'd be really curious to hear from Joshua and Tom on this
> > front,
> > > > > > though.
> > > > > > > >> Is
> > > > > > > >> it acceptable to move more incrementally here and settle on
> > the
> > > > > syntax
> > > > > > > >> version property as our means of introducing new features,
> or
> > is
> > > > it
> > > > > > > >> preferable to implement things monolithically and try to get
> > > > > > everything
> > > > > > > >> (or
> > > > > > > >> at least, as much as possible) right the first time?
> > > > > > > >>
> > > > > > > >> Thanks again for your continued effort on this KIP!
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >>
> > > > > > > >> Chris
> > > > > > > >>
> > > > > > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate Otoya
> <
> > > > > > > >> quilcate.jorge@gmail.com> wrote:
> > > > > > > >>
> > > > > > > >> > Thanks, Chris!
> > > > > > > >> >
> > > > > > > >> > Please, find my comments below:
> > > > > > > >> >
> > > > > > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> > > > > fearthecellos@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Jorge,
> > > > > > > >> > >
> > > > > > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > > > > > >> > >
> > > > > > > >> > > 1. Under the "Accessing multiple values by deep-scan"
> > header
> > > > > it's
> > > > > > > >> stated
> > > > > > > >> > > that "If deep-scan is used, it must have only one field
> > > after
> > > > > the
> > > > > > > >> > asterisk
> > > > > > > >> > > level.". However, in example 3 for the Cast SMT and
> other
> > > > > examples
> > > > > > > for
> > > > > > > >> > > other SMTs, the spec contains a field of "*.child.k2",
> > which
> > > > > > appears
> > > > > > > >> to
> > > > > > > >> > > have two fields after the asterisk level. I may be
> > > > > > misunderstanding
> > > > > > > >> the
> > > > > > > >> > > proposal, but it seems like the two contradict each
> other.
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > Thanks for catching this. I have clarified it by removing
> > this
> > > > > > > >> restriction.
> > > > > > > >> > Also, have extended the deep-scan scenarios.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > 2. I'm a little unclear on why we need the special
> > handling
> > > > for
> > > > > > > arrays
> > > > > > > >> > > where, for an array field "a", the field name "a" can be
> > > > treated
> > > > > > as
> > > > > > > >> > either
> > > > > > > >> > > the array itself, or every element in the array. Is
> there
> > a
> > > > > reason
> > > > > > > we
> > > > > > > >> > can't
> > > > > > > >> > > use the field name "a.*" to handle the latter case, and
> > "a"
> > > to
> > > > > > > handle
> > > > > > > >> the
> > > > > > > >> > > former?
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > Agree, this is confusing. I like the `a.*` approach to
> > access
> > > > > array
> > > > > > > >> items.
> > > > > > > >> > I have added this to the proposal.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > 3. How would a user specify that they'd like to access a
> > > field
> > > > > > with
> > > > > > > >> the
> > > > > > > >> > > literal name "*"?
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > Good one. I'm proposing an approach similar to how it's
> > > proposed
> > > > > to
> > > > > > > >> escape
> > > > > > > >> > dots, with a double-asterisk. Curious on your thoughts
> > around
> > > > > this.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > 4. For the Cast SMT, do you think it might bite some
> > people
> > > if
> > > > > > > fields
> > > > > > > >> > that
> > > > > > > >> > > can't be cast correctly are silently ignored? I'm
> > imagining
> > > > the
> > > > > > case
> > > > > > > >> > where
> > > > > > > >> > > none of the fields in a multi-path expression can be
> cast
> > > > > > correctly
> > > > > > > >> and
> > > > > > > >> > it
> > > > > > > >> > > ends up eating half of someone's day to track down why
> > their
> > > > SMT
> > > > > > > isn't
> > > > > > > >> > > doing anything.
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > If I understand correctly, this challenge could be
> relevant
> > > > across
> > > > > > > SMTs.
> > > > > > > >> > At the moment, most/all? SMTs just silently ignore.
> > > > > > > >> > Was thinking about adding a flag `field.on.path.not.found`
> > to
> > > > > either
> > > > > > > >> ignore
> > > > > > > >> > or fail when no paths are found. What do your think?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > 5. For the ExtractField and ValueToKey SMTs, what
> happens
> > > if a
> > > > > > > >> deep-scan
> > > > > > > >> > > field name is used, but only one field is found? Is the
> > > > > resulting
> > > > > > > >> field
> > > > > > > >> > > still an array, or is it just the single field that was
> > > found?
> > > > > > (FWIW
> > > > > > > >> I'm
> > > > > > > >> > > leaning towards keeping it an array just to keep schemas
> > > > > > consistent
> > > > > > > >> in a
> > > > > > > >> > > pipeline in case the number of fields found fluctuates
> > > across
> > > > > > > >> records.)
> > > > > > > >> > >
> > > > > > > >> > > Agree. Will clarify that an array is always produced
> even
> > > for
> > > > 1
> > > > > > or 0
> > > > > > > >> > fields are found.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that "As
> this
> > > SMT
> > > > > > > affects
> > > > > > > >> > only
> > > > > > > >> > > existing fields, additional configurations will not be
> > > > > required.".
> > > > > > > >> Given
> > > > > > > >> > > the new "field.syntax.version" property, this part
> should
> > > > > probably
> > > > > > > be
> > > > > > > >> > > removed.
> > > > > > > >> > >
> > > > > > > >> > Agree.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > 7. Is recursive descent intentionally excluded? That was
> > an
> > > > > > > important
> > > > > > > >> > part
> > > > > > > >> > > of Joshua's KIP and his feedback on this KIP; I think
> it's
> > > > worth
> > > > > > > >> pursuing
> > > > > > > >> > > if we can.
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > My understanding from Joshua's feedback is that by
> including
> > > > > support
> > > > > > > for
> > > > > > > >> > deep-scan, we are already covering the recursive
> > > functionality.
> > > > > > > Though,
> > > > > > > >> I
> > > > > > > >> > may be missing something.
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Cheers,
> > > > > > > >> > >
> > > > > > > >> > > Chris
> > > > > > > >> > >
> > > > > > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate
> > > Otoya <
> > > > > > > >> > > quilcate.jorge@gmail.com> wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Thanks, Chris!
> > > > > > > >> > > >
> > > > > > > >> > > > I have updated the KIP with the rejected alternatives
> > > > updated.
> > > > > > > >> Also, I
> > > > > > > >> > > have
> > > > > > > >> > > > drafted the support for arrays and deep scans as part
> of
> > > the
> > > > > > > >> proposed
> > > > > > > >> > > > notation to make it more complete, even though these
> can
> > > be
> > > > > > > >> implemented
> > > > > > > >> > > in
> > > > > > > >> > > > multiple PRs.
> > > > > > > >> > > >
> > > > > > > >> > > > Looking forward to your feedback.
> > > > > > > >> > > >
> > > > > > > >> > > > Cheers,
> > > > > > > >> > > > Jorge.
> > > > > > > >> > > >
> > > > > > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > > > > > >> fearthecellos@gmail.com>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi Jorge,
> > > > > > > >> > > > >
> > > > > > > >> > > > > I really appreciate the effort you've made to
> simplify
> > > the
> > > > > > > syntax
> > > > > > > >> and
> > > > > > > >> > > > > feature set of a JSONPath-based approach as much as
> > > > > possible.
> > > > > > > I'm
> > > > > > > >> > still
> > > > > > > >> > > > > hesitant to continue with it, though.
> > > > > > > >> > > > >
> > > > > > > >> > > > > 1. The syntax is much less friendly. Just compare
> > > > > > > >> "top.mid.bottom" to
> > > > > > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses
> > > JSONPath
> > > > or
> > > > > > > even
> > > > > > > >> > JSON,
> > > > > > > >> > > > and
> > > > > > > >> > > > > the learning curve for the former is going to be
> > > steeper.
> > > > > The
> > > > > > > >> > examples
> > > > > > > >> > > in
> > > > > > > >> > > > > the new KIP draft you published demonstrate this
> > pretty
> > > > > well,
> > > > > > > and
> > > > > > > >> > this
> > > > > > > >> > > is
> > > > > > > >> > > > > without diving into the details of what escape
> syntax
> > > > would
> > > > > > look
> > > > > > > >> > like.
> > > > > > > >> > > > >
> > > > > > > >> > > > > 2. The three new major features that this syntax
> adds
> > > > (array
> > > > > > > >> > accesses,
> > > > > > > >> > > > deep
> > > > > > > >> > > > > scans, and multi-value paths) could all be added
> > pretty
> > > > > easily
> > > > > > > to
> > > > > > > >> the
> > > > > > > >> > > dot
> > > > > > > >> > > > > notation syntax without introducing brackets and
> > dollar
> > > > > signs.
> > > > > > > >> Array
> > > > > > > >> > > > > accesses can be described using the same syntax as
> > > > > struct/map
> > > > > > > >> field
> > > > > > > >> > > > access,
> > > > > > > >> > > > > deep scans can be described using '*', and
> multi-value
> > > > paths
> > > > > > can
> > > > > > > >> be
> > > > > > > >> > > > > described by referencing the name of a field that's
> > > > expected
> > > > > > to
> > > > > > > >> have
> > > > > > > >> > > > > children. These are all top-of-the-head ideas and
> can
> > > > > probably
> > > > > > > be
> > > > > > > >> > > > refined,
> > > > > > > >> > > > > but hopefully they demonstrate that we can keep the
> > > syntax
> > > > > > > simple
> > > > > > > >> > > without
> > > > > > > >> > > > > sacrificing features. Of course, the question of
> > leaving
> > > > > room
> > > > > > > for
> > > > > > > >> > > future
> > > > > > > >> > > > > features might arise... given that these are the
> > > > > > out-of-the-box
> > > > > > > >> SMTs
> > > > > > > >> > > that
> > > > > > > >> > > > > are likely to be the first that many people
> encounter,
> > > I'd
> > > > > err
> > > > > > > on
> > > > > > > >> the
> > > > > > > >> > > > side
> > > > > > > >> > > > > of simplicity and a gentle learning curve; if people
> > > need
> > > > to
> > > > > > get
> > > > > > > >> more
> > > > > > > >> > > out
> > > > > > > >> > > > > of transforms, the option to implement their own is
> > > still
> > > > > > there.
> > > > > > > >> If
> > > > > > > >> > we
> > > > > > > >> > > > can
> > > > > > > >> > > > > address 95% of use cases with something easy to use,
> > > it's
> > > > > not
> > > > > > > >> worth
> > > > > > > >> > > > making
> > > > > > > >> > > > > the feature harder for everyone to use just to
> > > accommodate
> > > > > the
> > > > > > > >> > > remaining
> > > > > > > >> > > > > 5%.
> > > > > > > >> > > > >
> > > > > > > >> > > > > 3. The advantage of leveraging an existing syntax is
> > > > > twofold:
> > > > > > > >> users
> > > > > > > >> > who
> > > > > > > >> > > > are
> > > > > > > >> > > > > already familiar with that syntax don't need to
> learn
> > a
> > > > new
> > > > > > > >> syntax,
> > > > > > > >> > and
> > > > > > > >> > > > > maintainers of the syntax get to leverage existing
> > > > libraries
> > > > > > and
> > > > > > > >> > > > > documentation for the syntax. With the current
> > proposal,
> > > > > > reusing
> > > > > > > >> > > > libraries
> > > > > > > >> > > > > is off the table, which means that we're going to
> have
> > > to
> > > > > > parse
> > > > > > > >> this
> > > > > > > >> > > > > ourselves (including all the escape syntax edge
> > cases),
> > > > and
> > > > > we
> > > > > > > >> won't
> > > > > > > >> > be
> > > > > > > >> > > > > able to automatically leverage new features added to
> > > that
> > > > > > > syntax.
> > > > > > > >> And
> > > > > > > >> > > > given
> > > > > > > >> > > > > how stripped-down the syntax is in comparison to
> full
> > > > > > JSONPath,
> > > > > > > >> > there's
> > > > > > > >> > > > > still going to be a learning curve for users who are
> > > > already
> > > > > > > >> familiar
> > > > > > > >> > > > with
> > > > > > > >> > > > > it, and we'll still have to document how Connect's
> > > variant
> > > > > of
> > > > > > > >> > JSONPath
> > > > > > > >> > > > > works either instead of or in addition to just
> linking
> > > to
> > > > a
> > > > > > > >> > > > well-maintained
> > > > > > > >> > > > > third-party docs site.
> > > > > > > >> > > > >
> > > > > > > >> > > > > On the topic of "field.path" and "include.path" vs.
> > > > > > > >> "field.style", I
> > > > > > > >> > > > > actually think that a single property per SMT is
> > cleaner
> > > > and
> > > > > > > >> simpler.
> > > > > > > >> > > > > Allowing users to mix and match styles within the
> same
> > > SMT
> > > > > > > config
> > > > > > > >> > seems
> > > > > > > >> > > > > like a recipe for confusion, and with a single
> > property
> > > > that
> > > > > > > >> dictates
> > > > > > > >> > > > field
> > > > > > > >> > > > > syntax behavior, we leave the door open to change
> the
> > > > > default
> > > > > > > at a
> > > > > > > >> > > later
> > > > > > > >> > > > > date. We could even fully remove support for plain
> > field
> > > > > > > notation
> > > > > > > >> at
> > > > > > > >> > > some
> > > > > > > >> > > > > point and still be able to retain the simple
> property
> > > > names
> > > > > of
> > > > > > > >> > "field",
> > > > > > > >> > > > > "include", etc. instead of forcing people to use
> > > > > "field.path"
> > > > > > > and
> > > > > > > >> the
> > > > > > > >> > > > like.
> > > > > > > >> > > > > That said, the term "field.style" and the permitted
> > > values
> > > > > > might
> > > > > > > >> be a
> > > > > > > >> > > > > little ambiguous. One alternative, though a little
> > > > > > heavy-handed,
> > > > > > > >> is
> > > > > > > >> > to
> > > > > > > >> > > > > change it to "field.syntax.version" with permitted
> > > values
> > > > of
> > > > > > > "V1"
> > > > > > > >> > > > (default,
> > > > > > > >> > > > > equivalent to "field.style = plain") and "V2"
> > > (equivalent
> > > > to
> > > > > > > >> > > > "field.style =
> > > > > > > >> > > > > nested"). This would leave us room in the future to
> > make
> > > > > > further
> > > > > > > >> > > changes
> > > > > > > >> > > > to
> > > > > > > >> > > > > the syntax without having to come up with new names,
> > > > > although
> > > > > > it
> > > > > > > >> does
> > > > > > > >> > > > > sacrifice a little bit in that the permitted values
> > are
> > > no
> > > > > > > longer
> > > > > > > >> > > > > self-describing.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Cheers,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Chris
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban
> Quilcate
> > > > > Otoya <
> > > > > > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Thank you all for your feedback, and sorry for the
> > > long
> > > > > wait
> > > > > > > >> for a
> > > > > > > >> > > > reply.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I would like to explore the idea of
> > > > > JSONPath-inspired/subset
> > > > > > > >> > > notation a
> > > > > > > >> > > > > bit
> > > > > > > >> > > > > > further:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > It will need to be a much-reduced version of
> > JSONPath:
> > > > > > > >> > > > > > - No full support for JsonPath therefore an
> > additional
> > > > > > > >> dependency.
> > > > > > > >> > > > > > - All paths must start with '$'
> > > > > > > >> > > > > > - No functions support or other operators allowed.
> > > > > > > >> > > > > > - JsonPath dotted and square-bracket notations can
> > be
> > > > > > > supported
> > > > > > > >> to
> > > > > > > >> > > > avoid
> > > > > > > >> > > > > > escaping dots or other characters: `$a.b.c` and
> > > > > > > `$['a.b']['c']`
> > > > > > > >> -
> > > > > > > >> > or
> > > > > > > >> > > we
> > > > > > > >> > > > > > could only support the second one as it's more
> > > complete.
> > > > > > > >> > > > > > - Add support for arrays with `[<integer>]`, e.g.
> > > > > `$a.[1].b`
> > > > > > > >> > > > > > - Add support for multiple-value paths using array
> > > > access
> > > > > > > >> `$a.*.b`
> > > > > > > >> > or
> > > > > > > >> > > > > deep
> > > > > > > >> > > > > > scan `$a..b`.
> > > > > > > >> > > > > >     - Some SMTs that will benefit from this:
> > > > `MaskField`,
> > > > > > > >> `Cast`,
> > > > > > > >> > > > > > `ReplaceField`.
> > > > > > > >> > > > > > - We could introduce a `path[s]` config under the
> > > > current
> > > > > > > >> > > > configurations
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > apply this feature so no compatibility issues are
> > > > > > introduced:
> > > > > > > >> e.g.
> > > > > > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > With these,  100, 101, 102, and 103 will be
> > > effectively
> > > > > > > solved.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > The main challenge that I see at the moment is
> that
> > by
> > > > > being
> > > > > > > >> JSON
> > > > > > > >> > > > > > path-like, there may be some edge cases that I
> can't
> > > > > foresee
> > > > > > > at
> > > > > > > >> the
> > > > > > > >> > > > > moment,
> > > > > > > >> > > > > > that could make this hard to implement, test, and
> > > > maintain
> > > > > > in
> > > > > > > >> the
> > > > > > > >> > > long
> > > > > > > >> > > > > run.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I'll appreciate your feedback on how this
> > > JSONPath-based
> > > > > > > >> > alternative
> > > > > > > >> > > > > > compares to the dotted notation initially
> proposed,
> > > and
> > > > if
> > > > > > it
> > > > > > > >> > matches
> > > > > > > >> > > > > your
> > > > > > > >> > > > > > feedback.
> > > > > > > >> > > > > > I have drafted a copy of the KIP to change the
> > > examples
> > > > to
> > > > > > > >> JSONPath
> > > > > > > >> > > > > > approach and validate some differences:
> > > > > > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > About 104. I agreed with Chris. We can handle this
> > as
> > > > part
> > > > > > of
> > > > > > > a
> > > > > > > >> new
> > > > > > > >> > > > KIP.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > Jorge.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > > > > > >> > fearthecellos@gmail.com
> > > > > > > >> > > >
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi Joshua,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I have a few reservations about using JsonPath
> > > > notation
> > > > > > > here.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 1. There's likely to be a substantial
> performance
> > > > > penalty
> > > > > > > for
> > > > > > > >> > > > > converting
> > > > > > > >> > > > > > > between the Kafka Connect format and something
> > that
> > > a
> > > > > > > JsonPath
> > > > > > > >> > > > library
> > > > > > > >> > > > > > > would understand.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 2. The complexity of the feature will be
> > > significantly
> > > > > > > >> higher. It
> > > > > > > >> > > > will
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > > harder to test, document, and implement. There
> > will
> > > be
> > > > > > many
> > > > > > > >> more
> > > > > > > >> > > edge
> > > > > > > >> > > > > > cases
> > > > > > > >> > > > > > > to consider and support, and we'll be on the
> hook
> > to
> > > > > > handle
> > > > > > > >> any
> > > > > > > >> > > bugs
> > > > > > > >> > > > or
> > > > > > > >> > > > > > > inconsistencies that arise, either as a result
> of
> > > our
> > > > > use
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > > > JsonPath
> > > > > > > >> > > > > > > library we choose, or as a result of bugs in
> that
> > > > > library.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 3. It's not clear that JsonPath is superior or
> > even
> > > > > > suitable
> > > > > > > >> for
> > > > > > > >> > > some
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > > the SMTs proposed here. What would be the
> > advantages
> > > > of
> > > > > > > >> JsonPath
> > > > > > > >> > > with
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > InsertField or HoistField SMTs?
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I also don't think that adding dot notation is
> > > > > unfriendly
> > > > > > to
> > > > > > > >> > users;
> > > > > > > >> > > > > many
> > > > > > > >> > > > > > > have proposed this type of syntax in the past,
> and
> > > > it's
> > > > > > > >> > frequently
> > > > > > > >> > > > used
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > > informal discussions to refer to nested fields.
> If
> > > the
> > > > > > > >> proposed
> > > > > > > >> > > > syntax
> > > > > > > >> > > > > > was
> > > > > > > >> > > > > > > not already intuitive then a case against
> deciding
> > > on
> > > > > our
> > > > > > > own
> > > > > > > >> > might
> > > > > > > >> > > > be
> > > > > > > >> > > > > > more
> > > > > > > >> > > > > > > convincing, but as things stand, simple dot
> > notation
> > > > is
> > > > > > > likely
> > > > > > > >> > > going
> > > > > > > >> > > > to
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > > easier for users to understand than JsonPath
> > syntax.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Cheers,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Chris
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua Grisham <
> > > > > > > >> > > > grishamjosh@gmail.com>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hello all!
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Sorry that I come a bit later to the party
> here,
> > > > but I
> > > > > > am
> > > > > > > >> the
> > > > > > > >> > one
> > > > > > > >> > > > who
> > > > > > > >> > > > > > > wrote
> > > > > > > >> > > > > > > > KIP-683 [1] for recursive support (just simply
> > > > looping
> > > > > > > >> through
> > > > > > > >> > > all
> > > > > > > >> > > > > > child
> > > > > > > >> > > > > > > > non-primitive structures for the same matching
> > > > > name(s))
> > > > > > > >> which
> > > > > > > >> > is
> > > > > > > >> > > a
> > > > > > > >> > > > > > > slightly
> > > > > > > >> > > > > > > > different way to try and solve a similar
> > > requirement
> > > > > --
> > > > > > > >> > > > unfortunately
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > > > the time the dev community was not quite as
> > active
> > > > and
> > > > > > > then
> > > > > > > >> I
> > > > > > > >> > > also
> > > > > > > >> > > > > got
> > > > > > > >> > > > > > > busy
> > > > > > > >> > > > > > > > with work and just life in general so wasn't
> > able
> > > to
> > > > > > > follow
> > > > > > > >> up
> > > > > > > >> > or
> > > > > > > >> > > > > push
> > > > > > > >> > > > > > > it.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > I do think it is a very good idea to have some
> > > kind
> > > > of
> > > > > > > >> > path-like
> > > > > > > >> > > > > > > expression
> > > > > > > >> > > > > > > > to be able to specifically address a nested
> > field,
> > > > as
> > > > > I
> > > > > > > can
> > > > > > > >> see
> > > > > > > >> > > > that
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > simple "recursive" case could potentially
> result
> > > in
> > > > > > > >> unwanted or
> > > > > > > >> > > > > > > unexpected
> > > > > > > >> > > > > > > > behavior, plus there is the potential to
> > > introduce a
> > > > > bit
> > > > > > > of
> > > > > > > >> a
> > > > > > > >> > > > > > performance
> > > > > > > >> > > > > > > > hit to always loop through everything in cases
> > > where
> > > > > the
> > > > > > > >> > > > > schemas/values
> > > > > > > >> > > > > > > > might be quite large.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > One thing I wanted to ask: instead of
> creating a
> > > new
> > > > > > "path
> > > > > > > >> > > parser"
> > > > > > > >> > > > > > > > including some bespoke or "borrowed" syntax,
> why
> > > not
> > > > > > just
> > > > > > > >> use
> > > > > > > >> > > > > something
> > > > > > > >> > > > > > > > that already exists? Specifically here I am
> > > thinking
> > > > > > about
> > > > > > > >> > > > JsonPath (
> > > > > > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > There is already quite nice support in
> JsonPath
> > > for
> > > > > > > handling
> > > > > > > >> > > > special
> > > > > > > >> > > > > > > > characters in field names, for handling
> > different
> > > > > > > >> non-primitive
> > > > > > > >> > > > types
> > > > > > > >> > > > > > > > (arrays etc), for handling multiple levels of
> > > > nesting,
> > > > > > etc
> > > > > > > >> etc.
> > > > > > > >> > > > > Would
> > > > > > > >> > > > > > it
> > > > > > > >> > > > > > > > be possible to instead to re-think this and
> > maybe
> > > > have
> > > > > > > some
> > > > > > > >> > kind
> > > > > > > >> > > of
> > > > > > > >> > > > > > > > JsonPath-based Schema selector / updater
> and/or
> > > > > > > >> JsonPath-based
> > > > > > > >> > > > Value
> > > > > > > >> > > > > > > > selector / updater? Conceptually this feels
> like
> > > it
> > > > > > makes
> > > > > > > >> sense
> > > > > > > >> > > to
> > > > > > > >> > > > > me,
> > > > > > > >> > > > > > as
> > > > > > > >> > > > > > > > from the top of my head it would be quite a
> > > natural
> > > > > fit
> > > > > > to
> > > > > > > >> map
> > > > > > > >> > a
> > > > > > > >> > > > Json
> > > > > > > >> > > > > > > data
> > > > > > > >> > > > > > > > structure to the Connect API data structure
> (and
> > > you
> > > > > > could
> > > > > > > >> > > > > potentially
> > > > > > > >> > > > > > > even
> > > > > > > >> > > > > > > > try to leverage the existing Json-to-Connect
> > > > > > > >> > > > serializer/deserializer
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > help out with this even in a more "out of the
> > > > > > box"-feeling
> > > > > > > >> kind
> > > > > > > >> > > of
> > > > > > > >> > > > > > way).
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Maybe also as Tom mentioned, this part (in my
> > > > example,
> > > > > > > this
> > > > > > > >> > > > > > > JsonPath-based
> > > > > > > >> > > > > > > > "thing") could even be a generic API that
> could
> > be
> > > > > used
> > > > > > by
> > > > > > > >> any
> > > > > > > >> > > SMT,
> > > > > > > >> > > > > > > > including used in custom ones built by the
> > > > community.
> > > > > > > Then
> > > > > > > >> I
> > > > > > > >> > > think
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > use
> > > > > > > >> > > > > > > > a completely separate config property somehow
> > > > related
> > > > > to
> > > > > > > >> "path"
> > > > > > > >> > > (as
> > > > > > > >> > > > > Tom
> > > > > > > >> > > > > > > > also mentioned) would also make a lot of sense
> > > here
> > > > as
> > > > > > > well.
> > > > > > > >> > This
> > > > > > > >> > > > > way,
> > > > > > > >> > > > > > if
> > > > > > > >> > > > > > > > you select based on "path" then this
> > > JsonPath-based
> > > > > API
> > > > > > > >> would
> > > > > > > >> > be
> > > > > > > >> > > > > used,
> > > > > > > >> > > > > > > > otherwise it could use something similar to
> the
> > > > > existing
> > > > > > > >> > > get-field
> > > > > > > >> > > > > > based
> > > > > > > >> > > > > > > > approach (which I guess could also be
> refactored
> > > > into
> > > > > > some
> > > > > > > >> kind
> > > > > > > >> > > of
> > > > > > > >> > > > > > > utility
> > > > > > > >> > > > > > > > / API as well if it made sense?)
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > And with that in mind, if this was the kind of
> > > > > direction
> > > > > > > to
> > > > > > > >> go,
> > > > > > > >> > > > then
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > > > "recursive" capability like I pitched in
> KIP-683
> > > > would
> > > > > > > also
> > > > > > > >> > > become
> > > > > > > >> > > > > > > > unnecessary because you could easily write a
> > > > JsonPath
> > > > > > > >> > expression
> > > > > > > >> > > > like
> > > > > > > >> > > > > > > > "$..someRecuriveField" and it would do the
> same
> > > > thing
> > > > > > (on
> > > > > > > >> top
> > > > > > > >> > of
> > > > > > > >> > > > > > anything
> > > > > > > >> > > > > > > > else you would want to do that is already
> > > supported
> > > > by
> > > > > > > >> > JsonPath).
> > > > > > > >> > > > > Then
> > > > > > > >> > > > > > we
> > > > > > > >> > > > > > > > could also kill that older KIP and do a bit of
> > > > > clean-up
> > > > > > :)
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > [1] -
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Just some extra food for thought. All-in-all I
> > > think
> > > > > > this
> > > > > > > >> is a
> > > > > > > >> > > > super
> > > > > > > >> > > > > > > great
> > > > > > > >> > > > > > > > initiative!
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Best,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Joshua
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris
> > Egerton
> > > <
> > > > > > > >> > > > > > > fearthecellos@gmail.com
> > > > > > > >> > > > > > > > >:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Tom,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks for taking a look at this, and for
> your
> > > > > > > thoughtful
> > > > > > > >> > > > comments.
> > > > > > > >> > > > > > > I'll
> > > > > > > >> > > > > > > > > leave it up to Jorge to address most of your
> > > > > comments
> > > > > > > but
> > > > > > > >> I
> > > > > > > >> > > > wanted
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > share
> > > > > > > >> > > > > > > > > a couple quick thoughts I had regarding 103
> > and
> > > > 104.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 103. Like you, I was envisioning a possible
> > > syntax
> > > > > for
> > > > > > > >> array
> > > > > > > >> > > > access
> > > > > > > >> > > > > > > that
> > > > > > > >> > > > > > > > > used classic C-style brackets; e.g.,
> > > `arr[index]`.
> > > > > > > >> However, I
> > > > > > > >> > > > > wonder
> > > > > > > >> > > > > > if
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > could keep things simple and use the same
> > syntax
> > > > > that
> > > > > > > >> we're
> > > > > > > >> > > > > proposing
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > > nested field access? In other words, instead
> > of
> > > > > > > >> `arr[index]`,
> > > > > > > >> > > > you'd
> > > > > > > >> > > > > > > write
> > > > > > > >> > > > > > > > > `arr.index`. It'd save us and users the
> > headache
> > > > of
> > > > > > > >> reserving
> > > > > > > >> > > > > > > characters
> > > > > > > >> > > > > > > > > now that would need to be escaped even if
> > their
> > > > > > > unescaped
> > > > > > > >> > > > brethren
> > > > > > > >> > > > > > > aren't
> > > > > > > >> > > > > > > > > used for anything, and also avoid the
> question
> > > of
> > > > > what
> > > > > > > >> > exactly
> > > > > > > >> > > we
> > > > > > > >> > > > > > > should
> > > > > > > >> > > > > > > > do
> > > > > > > >> > > > > > > > > when we see a config that uses reserved
> > > characters
> > > > > > that
> > > > > > > >> > aren't
> > > > > > > >> > > > yet
> > > > > > > >> > > > > > > > > supported (throwing an exception seems
> pretty
> > > > > > unfriendly
> > > > > > > >> for
> > > > > > > >> > > new
> > > > > > > >> > > > > > > users).
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 104. This would probably be useful, but it
> > would
> > > > > come
> > > > > > > with
> > > > > > > >> > some
> > > > > > > >> > > > > nasty
> > > > > > > >> > > > > > > > > compatibility questions that would need to
> be
> > > > > > addressed
> > > > > > > if
> > > > > > > >> > we'd
> > > > > > > >> > > > > want
> > > > > > > >> > > > > > > SMTs
> > > > > > > >> > > > > > > > > that leverage this new API to be viable for
> > > older
> > > > > > > >> versions of
> > > > > > > >> > > > > > Connect.
> > > > > > > >> > > > > > > If
> > > > > > > >> > > > > > > > > we package and distribute this feature as a
> > > > library
> > > > > > > >> (either
> > > > > > > >> > via
> > > > > > > >> > > > an
> > > > > > > >> > > > > > > > entirely
> > > > > > > >> > > > > > > > > new artifact, or as part of the existing
> > > > > > > >> connect-transforms
> > > > > > > >> > or
> > > > > > > >> > > > > > > > connect-api
> > > > > > > >> > > > > > > > > artifacts), then we'd have to either
> sidestep
> > > the
> > > > > > > existing
> > > > > > > >> > > plugin
> > > > > > > >> > > > > > > > isolation
> > > > > > > >> > > > > > > > > logic [1] that basically makes it impossible
> > for
> > > > > > Connect
> > > > > > > >> > > plugins
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > ship
> > > > > > > >> > > > > > > > > their own versions of Connect artifacts, or
> > > issue
> > > > a
> > > > > > big
> > > > > > > >> > warning
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > people
> > > > > > > >> > > > > > > > > that any SMT that uses this API won't work
> > with
> > > > any
> > > > > > > older
> > > > > > > >> > > > versions
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > > > > Connect. There's also some other features we
> > > might
> > > > > > want
> > > > > > > to
> > > > > > > >> > add
> > > > > > > >> > > in
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > > > > SMT-utils library such as the existing,
> > > internal,
> > > > > > utils
> > > > > > > >> that
> > > > > > > >> > > > > Connect
> > > > > > > >> > > > > > > uses
> > > > > > > >> > > > > > > > > right now [2]. It may be worth exploring
> this
> > > in a
> > > > > > > >> separate
> > > > > > > >> > KIP
> > > > > > > >> > > > of
> > > > > > > >> > > > > > its
> > > > > > > >> > > > > > > > own.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > [1] -
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > [2] -
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Chris
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom Bentley
> <
> > > > > > > >> > > tbentley@redhat.com
> > > > > > > >> > > > >
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks for the KIP, especially for the
> > > examples
> > > > > > which
> > > > > > > >> are
> > > > > > > >> > > > > > > super-clear.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 100. The name `field.style` isn't so clear
> > for
> > > > > > > something
> > > > > > > >> > like
> > > > > > > >> > > > > > > > > ReplaceField:
> > > > > > > >> > > > > > > > > > it's not so obvious that field.style
> applies
> > > to
> > > > > > > >> `include`
> > > > > > > >> > and
> > > > > > > >> > > > > > > > `exclude`.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 101. The permitted values for
> `field.style`
> > > > don't
> > > > > > seem
> > > > > > > >> > > terribly
> > > > > > > >> > > > > > > > intuitive
> > > > > > > >> > > > > > > > > > (to me anyway): the meaning of `plain`
> isn't
> > > > very
> > > > > > > >> > guessable.
> > > > > > > >> > > > Why
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > > > > > `top-level` or `root` instead? Also
> `nested`
> > > > could
> > > > > > be
> > > > > > > >> > > > > misconstrued
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > > mean
> > > > > > > >> > > > > > > > > > nested-but-not-top-level, so perhaps
> > > `recursive`
> > > > > or
> > > > > > > >> > > `cascading`
> > > > > > > >> > > > > > might
> > > > > > > >> > > > > > > > be
> > > > > > > >> > > > > > > > > > better?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 102. I'm torn on whether making the
> > > > interpretation
> > > > > > of
> > > > > > > >> > > existing
> > > > > > > >> > > > > > > configs
> > > > > > > >> > > > > > > > > like
> > > > > > > >> > > > > > > > > > `field` be dependent on `field.style` is a
> > > good
> > > > > > idea.
> > > > > > > I
> > > > > > > >> can
> > > > > > > >> > > see
> > > > > > > >> > > > > > that
> > > > > > > >> > > > > > > > it's
> > > > > > > >> > > > > > > > > > the simplest thing to do, but it just
> feels
> > a
> > > > bit
> > > > > > odd
> > > > > > > >> that
> > > > > > > >> > > > > > sometimes
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > `field` would actually be a path and have
> > > > > different
> > > > > > > >> > escaping
> > > > > > > >> > > > > rules.
> > > > > > > >> > > > > > > An
> > > > > > > >> > > > > > > > > > alternative would be to come up with a
> > > parallel
> > > > > set
> > > > > > of
> > > > > > > >> > config
> > > > > > > >> > > > > names
> > > > > > > >> > > > > > > > (e.g.
> > > > > > > >> > > > > > > > > > as well as "field" an SMT might support
> > > "path")
> > > > > > which
> > > > > > > >> were
> > > > > > > >> > > > > defined
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > > > always take paths, thus avoiding the
> > > changeable
> > > > > > > >> > > interpretation
> > > > > > > >> > > > of
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > > > existing configs. The SMT's #configure()
> > would
> > > > > need
> > > > > > to
> > > > > > > >> > throw
> > > > > > > >> > > in
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > case
> > > > > > > >> > > > > > > > > > that both configs were given. I can see
> that
> > > > that
> > > > > > > would
> > > > > > > >> be
> > > > > > > >> > > more
> > > > > > > >> > > > > > work
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 103. I think in order to allow for
> > supporting
> > > > > arrays
> > > > > > > in
> > > > > > > >> a
> > > > > > > >> > > later
> > > > > > > >> > > > > KIP
> > > > > > > >> > > > > > > > > (which
> > > > > > > >> > > > > > > > > > certainly seems like it could be useful),
> > we'd
> > > > > want
> > > > > > to
> > > > > > > >> > > specify
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > syntax
> > > > > > > >> > > > > > > > > > now, even if it wasn't implemented under
> > this
> > > > KIP.
> > > > > > > >> That's
> > > > > > > >> > > > > because I
> > > > > > > >> > > > > > > > don't
> > > > > > > >> > > > > > > > > > think you can't exclude the possibility
> that
> > > > > > > characters
> > > > > > > >> > such
> > > > > > > >> > > as
> > > > > > > >> > > > > `[`
> > > > > > > >> > > > > > > and
> > > > > > > >> > > > > > > > > `]`
> > > > > > > >> > > > > > > > > > appear in field names. So you'd have a
> > > > > compatibility
> > > > > > > >> > problem
> > > > > > > >> > > if
> > > > > > > >> > > > > > > people
> > > > > > > >> > > > > > > > > > started using the features of this KIP to
> > > access
> > > > > > such
> > > > > > > >> > fields,
> > > > > > > >> > > > > only
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > > > those characters to change their meaning
> > > under a
> > > > > > later
> > > > > > > >> KIP.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 104. I also wonder whether making paths
> > into a
> > > > > > public
> > > > > > > >> Java
> > > > > > > >> > > API,
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > > use
> > > > > > > >> > > > > > > > > by
> > > > > > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks again,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Tom
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris
> Egerton
> > <
> > > > > > > >> > > > > > fearthecellos@gmail.com
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge
> Esteban
> > > > > Quilcate
> > > > > > > >> Otoya
> > > > > > > >> > <
> > > > > > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Thank you, Chris! Not possible without
> > > your
> > > > > > > >> feedback.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04, Chris
> > > Egerton
> > > > <
> > > > > > > >> > > > > > > > fearthecellos@gmail.com
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Thank you for sticking through
> this. I
> > > > have
> > > > > > one
> > > > > > > >> small
> > > > > > > >> > > > > remark
> > > > > > > >> > > > > > > and
> > > > > > > >> > > > > > > > > one
> > > > > > > >> > > > > > > > > > > > small
> > > > > > > >> > > > > > > > > > > > > clarification; assuming you agree
> with
> > > me
> > > > on
> > > > > > > them
> > > > > > > >> > then
> > > > > > > >> > > > I'm
> > > > > > > >> > > > > > > ready
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > > vote
> > > > > > > >> > > > > > > > > > > > on
> > > > > > > >> > > > > > > > > > > > > the KIP.
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > 1. InsertField: The
> > > > > "field.on.missing.parent"
> > > > > > > and
> > > > > > > >> > > > > > > > > > > > "field.on.existing.field"
> > > > > > > >> > > > > > > > > > > > > docs both mention a permitted value
> of
> > > > > > "ingore";
> > > > > > > >> this
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > > > > > > "ignore",
> > > > > > > >> > > > > > > > > > > > > right?
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Of course, one more typo :)
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > 2. InsertField: The examples are
> still
> > > > > missing
> > > > > > > the
> > > > > > > >> > > > > > > "field.style"
> > > > > > > >> > > > > > > > > > > property
> > > > > > > >> > > > > > > > > > > > > from the configurations. They should
> > all
> > > > > > include
> > > > > > > >> the
> > > > > > > >> > > > > property
> > > > > > > >> > > > > > > > > > > > > "transforms.smt1.field.style":
> > "nested",
> > > > > > > correct?
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Yes, it is there. I think I know what
> > you
> > > > mean
> > > > > > > now,
> > > > > > > >> > seems
> > > > > > > >> > > > > that
> > > > > > > >> > > > > > > > > > Confluence
> > > > > > > >> > > > > > > > > > > > is putting everything in one line when
> > > it's
> > > > in
> > > > > > > >> separate
> > > > > > > >> > > > lines
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > > > editor.
> > > > > > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Thanks again for working through
> this,
> > > and
> > > > > > > >> > > > congratulations
> > > > > > > >> > > > > > on a
> > > > > > > >> > > > > > > > > > > > > well-written KIP!
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Chris
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM
> Jorge
> > > > > Esteban
> > > > > > > >> > Quilcate
> > > > > > > >> > > > > Otoya
> > > > > > > >> > > > > > <
> > > > > > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Thank you, Chris! I apply these
> > > > > improvements
> > > > > > > to
> > > > > > > >> the
> > > > > > > >> > > > KIP,
> > > > > > > >> > > > > > let
> > > > > > > >> > > > > > > me
> > > > > > > >> > > > > > > > > > know
> > > > > > > >> > > > > > > > > > > > how
> > > > > > > >> > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > looks now.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43,
> Chris
> > > > > Egerton
> > > > > > <
> > > > > > > >> > > > > > > > > > fearthecellos@gmail.com
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > Wow, those examples are great! A
> > few
> > > > > more
> > > > > > > >> > remarks,
> > > > > > > >> > > > but
> > > > > > > >> > > > > I
> > > > > > > >> > > > > > > > think
> > > > > > > >> > > > > > > > > > > we're
> > > > > > > >> > > > > > > > > > > > > > > getting close:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > 1. The examples differ across
> SMTs
> > > > with
> > > > > > the
> > > > > > > >> name
> > > > > > > >> > of
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > > > > > > newly-introduced
> > > > > > > >> > > > > > > > > > > > > > > style property; some of them use
> > > > > > > >> "field.style",
> > > > > > > >> > and
> > > > > > > >> > > > > some
> > > > > > > >> > > > > > > use
> > > > > > > >> > > > > > > > > > > > > > > "fields.style". I think for
> > > > > consistency's
> > > > > > > >> sake we
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > > > stick
> > > > > > > >> > > > > > > > > > with
> > > > > > > >> > > > > > > > > > > > > just
> > > > > > > >> > > > > > > > > > > > > > > "field.style"; otherwise it
> could
> > be
> > > > > > painful
> > > > > > > >> for
> > > > > > > >> > > > users
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > have
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > remember
> > > > > > > >> > > > > > > > > > > > > > > which to use.
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed the
> > config
> > > > > names
> > > > > > > to
> > > > > > > >> > > > > > > `field.style`.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > 2. Some of the examples are off:
> > > > > > > >> > > > > > > > > > > > > > > - TimestampConverter: the input
> in
> > > the
> > > > > > > second
> > > > > > > >> > > example
> > > > > > > >> > > > > > > ("when
> > > > > > > >> > > > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > names
> > > > > > > >> > > > > > > > > > > > > > > include dots") doesn't contain a
> > > field
> > > > > > with
> > > > > > > a
> > > > > > > >> > > dotted
> > > > > > > >> > > > > name
> > > > > > > >> > > > > > > > > > > > > > > - ValueToKey: the config in the
> > > third
> > > > > > > example
> > > > > > > >> > > ("when
> > > > > > > >> > > > > > field
> > > > > > > >> > > > > > > > > names
> > > > > > > >> > > > > > > > > > > > > include
> > > > > > > >> > > > > > > > > > > > > > > dots") should probably use
> > > > > > > "parent..child.k2"
> > > > > > > >> as
> > > > > > > >> > > the
> > > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.fields"
> property
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > 3. RE changes to InsertField:
> > > > > > > >> > > > > > > > > > > > > > > - The InsertField SMT should
> also
> > > come
> > > > > > with
> > > > > > > >> the
> > > > > > > >> > new
> > > > > > > >> > > > > > > > > "field.style"
> > > > > > > >> > > > > > > > > > > > > > property
> > > > > > > >> > > > > > > > > > > > > > > in order to preserve backwards
> > > > > > > compatibility,
> > > > > > > >> > > right?
> > > > > > > >> > > > I
> > > > > > > >> > > > > > > don't
> > > > > > > >> > > > > > > > > see
> > > > > > > >> > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > included in the example configs
> > for
> > > > that
> > > > > > > one,
> > > > > > > >> > just
> > > > > > > >> > > > want
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > make
> > > > > > > >> > > > > > > > > > > sure
> > > > > > > >> > > > > > > > > > > > > > > - I don't know of any cases
> where
> > we
> > > > use
> > > > > > > >> > snake_case
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > > > property
> > > > > > > >> > > > > > > > > > > > names
> > > > > > > >> > > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > > Kafka; we should probably use
> > > > > > > >> "on.missing.parent"
> > > > > > > >> > > and
> > > > > > > >> > > > > > > > > > > > > "on.existing.field"
> > > > > > > >> > > > > > > > > > > > > > > as the new property names for
> > > > > InsertField.
> > > > > > > >> > > > > > > > > > > > > > > - Why is the "on_existing_field"
> > (or
> > > > > > > >> > > > > "on.existing.field")
> > > > > > > >> > > > > > > > > > property
> > > > > > > >> > > > > > > > > > > > only
> > > > > > > >> > > > > > > > > > > > > > > applied when the field style is
> > > > nested?
> > > > > > > >> Couldn't
> > > > > > > >> > it
> > > > > > > >> > > > be
> > > > > > > >> > > > > > > useful
> > > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Great points! I have applied these
> > > > > > suggestions
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > > > > KIP.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > Chris
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40 PM
> > > Jorge
> > > > > > > Esteban
> > > > > > > >> > > > Quilcate
> > > > > > > >> > > > > > > Otoya
> > > > > > > >> > > > > > > > <
> > > > > > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> wrote:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Again, great feedback Chris.
> > Much
> > > > > > > >> appreciated.
> > > > > > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at 20:22,
> > Chris
> > > > > > > Egerton <
> > > > > > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > Looking good! I have a few
> > > > comments
> > > > > > left
> > > > > > > >> but
> > > > > > > >> > > all
> > > > > > > >> > > > > but
> > > > > > > >> > > > > > > one
> > > > > > > >> > > > > > > > or
> > > > > > > >> > > > > > > > > > two
> > > > > > > >> > > > > > > > > > > > are
> > > > > > > >> > > > > > > > > > > > > > > > minor.
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > 1. The motivation section
> > states
> > > > > "This
> > > > > > > >> KIP is
> > > > > > > >> > > > aimed
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > > > include
> > > > > > > >> > > > > > > > > > > > > > support
> > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > nested structures on the
> > > existing
> > > > > > > SMTs...
> > > > > > > >> and
> > > > > > > >> > > to
> > > > > > > >> > > > > > > include
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > abstractions
> > > > > > > >> > > > > > > > > > > > > > > > > to reuse this in future
> > SMTs". A
> > > > > good
> > > > > > > >> > > > > implementation
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > > this
> > > > > > > >> > > > > > > > > > > KIP
> > > > > > > >> > > > > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > > > > > > definitely isolate reusable
> > > logic
> > > > > > into a
> > > > > > > >> > > separate
> > > > > > > >> > > > > > > > > abstraction
> > > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > can
> > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > > easily pulled in to the SMTs
> > we
> > > > want
> > > > > > to
> > > > > > > >> add
> > > > > > > >> > > > nested
> > > > > > > >> > > > > > > field
> > > > > > > >> > > > > > > > > > > support
> > > > > > > >> > > > > > > > > > > > > to,
> > > > > > > >> > > > > > > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > > > > > > unless we plan on making
> this
> > > kind
> > > > > of
> > > > > > > >> > > abstraction
> > > > > > > >> > > > > > > > publicly
> > > > > > > >> > > > > > > > > > > > > available
> > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > >> > > > > > > > > > > > > > > > > some kind of utility method
> or
> > > > class
> > > > > > > that
> > > > > > > >> > > > external
> > > > > > > >> > > > > > SMT
> > > > > > > >> > > > > > > > > > > developers
> > > > > > > >> > > > > > > > > > > > > can
> > > > > > > >> > > > > > > > > > > > > > > > > leverage, we can probably
> > leave
> > > > this
> > > > > > > part
> > > > > > > >> out
> > > > > > > >> > > as
> > > > > > > >> > > > > it's
> > > > > > > >> > > > > > > > more
> > > > > > > >> > > > > > > > > of
> > > > > > > >> > > > > > > > > > > an
> > > > > > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Make sense, will leave this
> out
> > of
> > > > the
> > > > > > > KIP.
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a
> > little
> > > > > > > >> misleading,
> > > > > > > >> > > isn't
> > > > > > > >> > > > > it?
> > > > > > > >> > > > > > > It
> > > > > > > >> > > > > > > > > > > > > demonstrates
> > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > escape syntax for fields
> with
> > > dot
> > > > > > > >> literals in
> > > > > > > >> > > > their
> > > > > > > >> > > > > > > > names,
> > > > > > > >> > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > doesn't
> > > > > > > >> > > > > > > > > > > > > > > > > demonstrate a way to
> actually
> > > use
> > > > > the
> > > > > > > Cast
> > > > > > > >> > (or
> > > > > > > >> > > > any
> > > > > > > >> > > > > > > other)
> > > > > > > >> > > > > > > > > SMT
> > > > > > > >> > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > access a
> > > > > > > >> > > > > > > > > > > > > > > > > nested field in a record,
> > which
> > > is
> > > > > the
> > > > > > > >> whole
> > > > > > > >> > > > point
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > KIP.
> > > > > > > >> > > > > > > > > > > I
> > > > > > > >> > > > > > > > > > > > > like
> > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > example of escape syntax but
> > we
> > > > > should
> > > > > > > >> > probably
> > > > > > > >> > > > > also
> > > > > > > >> > > > > > > add
> > > > > > > >> > > > > > > > > one
> > > > > > > >> > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > > > field access.
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Agree. I have added examples
> to
> > > each
> > > > > SMT
> > > > > > > to
> > > > > > > >> be
> > > > > > > >> > > more
> > > > > > > >> > > > > > clear
> > > > > > > >> > > > > > > > > about
> > > > > > > >> > > > > > > > > > > how
> > > > > > > >> > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > > affects each function
> > > > > > > >> > > > > > > > > > > > > > > > .
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > 3. With the InsertField SMT,
> > I'm
> > > > > > > wondering
> > > > > > > >> > what
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > > specific
> > > > > > > >> > > > > > > > > > > > > behavior
> > > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > > > > > > be when some or all of the
> > > "middle
> > > > > > > layer"
> > > > > > > >> > > nested
> > > > > > > >> > > > > > fields
> > > > > > > >> > > > > > > > are
> > > > > > > >> > > > > > > > > > > > > missing.
> > > > > > > >> > > > > > > > > > > > > > > For
> > > > > > > >> > > > > > > > > > > > > > > > > example, if I have a record
> > > with a
> > > > > > value
> > > > > > > >> of {
> > > > > > > >> > > > "k1":
> > > > > > > >> > > > > > > "v1 }
> > > > > > > >> > > > > > > > > > and I
> > > > > > > >> > > > > > > > > > > > > apply
> > > > > > > >> > > > > > > > > > > > > > > > > InsertField with
> topic.field =
> > > > > > > >> > n1.n2.n3.topic,
> > > > > > > >> > > > what
> > > > > > > >> > > > > > > will
> > > > > > > >> > > > > > > > > > > happen?
> > > > > > > >> > > > > > > > > > > > > Will
> > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > updated value become { "k1":
> > > "v1",
> > > > > > > "n1": {
> > > > > > > >> > > "n2":
> > > > > > > >> > > > {
> > > > > > > >> > > > > > > "n3":
> > > > > > > >> > > > > > > > > > > "topic"
> > > > > > > >> > > > > > > > > > > > > }}},
> > > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > > > > > > an exception be thrown, or
> > > > something
> > > > > > > else?
> > > > > > > >> > This
> > > > > > > >> > > > > seems
> > > > > > > >> > > > > > > > > > analogous
> > > > > > > >> > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > command line mkdir command,
> > > which
> > > > > (at
> > > > > > > >> least
> > > > > > > >> > on
> > > > > > > >> > > > some
> > > > > > > >> > > > > > > > > operating
> > > > > > > >> > > > > > > > > > > > > > systems)
> > > > > > > >> > > > > > > > > > > > > > > > > fails by default if you try
> to
> > > > > create
> > > > > > a
> > > > > > > >> new
> > > > > > > >> > > > nested
> > > > > > > >> > > > > > > > > directory
> > > > > > > >> > > > > > > > > > > > where
> > > > > > > >> > > > > > > > > > > > > > > > anything
> > > > > > > >> > > > > > > > > > > > > > > > > but the last element in the
> > path
> > > > > > doesn't
> > > > > > > >> > exist,
> > > > > > > >> > > > but
> > > > > > > >> > > > > > can
> > > > > > > >> > > > > > > > be
> > > > > > > >> > > > > > > > > > > > invoked
> > > > > > > >> > > > > > > > > > > > > > > with a
> > > > > > > >> > > > > > > > > > > > > > > > > flag that instructs it to go
> > > ahead
> > > > > and
> > > > > > > >> create
> > > > > > > >> > > all
> > > > > > > >> > > > > > > levels
> > > > > > > >> > > > > > > > of
> > > > > > > >> > > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > > > directory instead. I'm
> leaning
> > > on
> > > > > the
> > > > > > > >> side of
> > > > > > > >> > > > "just
> > > > > > > >> > > > > > > > create
> > > > > > > >> > > > > > > > > > > > > > everything"
> > > > > > > >> > > > > > > > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > > > > > > would be interested in your
> > > > > thoughts,
> > > > > > > and
> > > > > > > >> > > either
> > > > > > > >> > > > > way,
> > > > > > > >> > > > > > > we
> > > > > > > >> > > > > > > > > > should
> > > > > > > >> > > > > > > > > > > > > > > probably
> > > > > > > >> > > > > > > > > > > > > > > > > make sure the intended
> > behavior
> > > is
> > > > > > > >> > well-defined
> > > > > > > >> > > > > > before
> > > > > > > >> > > > > > > > > > voting.
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > This is an interesting case,
> > > thanks
> > > > > for
> > > > > > > >> > catching
> > > > > > > >> > > > > this!
> > > > > > > >> > > > > > > > > > > > > > > > The default behavior I see is
> to
> > > > > create
> > > > > > > >> parents
> > > > > > > >> > > if
> > > > > > > >> > > > > they
> > > > > > > >> > > > > > > are
> > > > > > > >> > > > > > > > > > > missing
> > > > > > > >> > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > overwrite fields
> > > > > > > >> > > > > > > > > > > > > > > > if they already exist.
> > > > > > > >> > > > > > > > > > > > > > > > I'm planning to include the
> > > > following
> > > > > > two
> > > > > > > >> flags
> > > > > > > >> > > if
> > > > > > > >> > > > > > there
> > > > > > > >> > > > > > > > is a
> > > > > > > >> > > > > > > > > > > need
> > > > > > > >> > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > > > > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> > > > > (CREATE|IGNORE),
> > > > > > > >> > > > > default=CREATE
> > > > > > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > > > > > (OVERWRITE|IGNORE),
> > > > > > > >> > > > > > > > default=OVERWRITE
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will the
> > > > behavior
> > > > > > be
> > > > > > > if
> > > > > > > >> > any
> > > > > > > >> > > of
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > elements
> > > > > > > >> > > > > > > > > > > > > > > > > specified with InsertField
> > > already
> > > > > > exist
> > > > > > > >> in
> > > > > > > >> > the
> > > > > > > >> > > > > > record
> > > > > > > >> > > > > > > > > value?
> > > > > > > >> > > > > > > > > > > > Will
> > > > > > > >> > > > > > > > > > > > > we
> > > > > > > >> > > > > > > > > > > > > > > > just
> > > > > > > >> > > > > > > > > > > > > > > > > overwrite them? What's the
> > > > behavior
> > > > > of
> > > > > > > >> > > > InsertField
> > > > > > > >> > > > > > > today
> > > > > > > >> > > > > > > > > > under
> > > > > > > >> > > > > > > > > > > > > > similar
> > > > > > > >> > > > > > > > > > > > > > > > > circumstances?
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > The current behavior is to
> > > overwrite
> > > > > the
> > > > > > > >> value.
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > Chris
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at 4:15
> > PM
> > > > > Jorge
> > > > > > > >> Esteban
> > > > > > > >> > > > > > Quilcate
> > > > > > > >> > > > > > > > > Otoya
> > > > > > > >> > > > > > > > > > <
> > > > > > > >> > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much
> > > appreciated
> > > > > all
> > > > > > > the
> > > > > > > >> > > > feedback
> > > > > > > >> > > > > > > here.
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > 1. You nailed it setting
> the
> > > > > design
> > > > > > > goal
> > > > > > > >> > > here:
> > > > > > > >> > > > > "it
> > > > > > > >> > > > > > > > > > shouldn't
> > > > > > > >> > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > > impossible
> > > > > > > >> > > > > > > > > > > > > > > > > > to use this new feature
> for
> > > any
> > > > > > field
> > > > > > > >> name,
> > > > > > > >> > > no
> > > > > > > >> > > > > > matter
> > > > > > > >> > > > > > > > how
> > > > > > > >> > > > > > > > > > > > > > convoluted.
> > > > > > > >> > > > > > > > > > > > > > > > > It's
> > > > > > > >> > > > > > > > > > > > > > > > > > fine if edge cases
> introduce
> > > > > > > difficulty
> > > > > > > >> > (such
> > > > > > > >> > > > as
> > > > > > > >> > > > > > > > > > > less-readable
> > > > > > > >> > > > > > > > > > > > > > > > > > configurations), but it's
> > not
> > > > fine
> > > > > > if
> > > > > > > >> they
> > > > > > > >> > > > can't
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > > > > > addressed
> > > > > > > >> > > > > > > > > > > > at
> > > > > > > >> > > > > > > > > > > > > > > all."
> > > > > > > >> > > > > > > > > > > > > > > > > > Back to the previous
> > proposals
> > > > > > (using
> > > > > > > >> only
> > > > > > > >> > > dots
> > > > > > > >> > > > > as
> > > > > > > >> > > > > > > > > > > separators)
> > > > > > > >> > > > > > > > > > > > we
> > > > > > > >> > > > > > > > > > > > > > > have
> > > > > > > >> > > > > > > > > > > > > > > > 2
> > > > > > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > > > > > >> > > > > > > > > > > > > > > > > > 1. escaping with
> backslashes
> > > > > > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots
> itself
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > I'll lean for alternative
> 2,
> > > as
> > > > > you
> > > > > > > >> > proposed
> > > > > > > >> > > > > > before.
> > > > > > > >> > > > > > > > > Feels
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > me
> > > > > > > >> > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > backslashes have more
> > > potential
> > > > to
> > > > > > > lead
> > > > > > > >> to
> > > > > > > >> > > > > > confusion
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > > JSON
> > > > > > > >> > > > > > > > > > > > > > configs,
> > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > CSV seems like a good
> > > precedent
> > > > to
> > > > > > use
> > > > > > > >> the
> > > > > > > >> > > same
> > > > > > > >> > > > > > > > character
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > escape
> > > > > > > >> > > > > > > > > > > > > > > > > itself.
> > > > > > > >> > > > > > > > > > > > > > > > > > KIP is updated to reflect
> > > this.
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an
> > > example,
> > > > > and
> > > > > > > >> stick
> > > > > > > >> > > with
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > > current
> > > > > > > >> > > > > > > > > > > > > approach
> > > > > > > >> > > > > > > > > > > > > > > > > > defining the style per
> > > > individual
> > > > > > > >> transform
> > > > > > > >> > > > > > > > > configuration.
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP
> updated.
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP updated.
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at
> > 21:59,
> > > > > Chris
> > > > > > > >> > Egerton <
> > > > > > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for addressing my
> > > > > comments;
> > > > > > > the
> > > > > > > >> > KIP
> > > > > > > >> > > > > looks
> > > > > > > >> > > > > > > > > > up-to-date
> > > > > > > >> > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > pretty
> > > > > > > >> > > > > > > > > > > > > > > > > > > readable now, and the
> > > rejected
> > > > > > > >> > alternatives
> > > > > > > >> > > > > > section
> > > > > > > >> > > > > > > > > does
> > > > > > > >> > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > great
> > > > > > > >> > > > > > > > > > > > > > > job
> > > > > > > >> > > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > outlining the discussion
> > so
> > > > far
> > > > > > and
> > > > > > > >> > > providing
> > > > > > > >> > > > > > > context
> > > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > anyone
> > > > > > > >> > > > > > > > > > > > > > > else
> > > > > > > >> > > > > > > > > > > > > > > > > who
> > > > > > > >> > > > > > > > > > > > > > > > > > > might want to join in.
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on choice of
> > > > > > delimiter:
> > > > > > > >> > > > > > > > > > > > > > > > > > > - I like the
> optimization
> > > for
> > > > > > simple
> > > > > > > >> > cases,
> > > > > > > >> > > > > but I
> > > > > > > >> > > > > > > > think
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > new
> > > > > > > >> > > > > > > > > > > > > > > > > proposal
> > > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > > > > a little too
> restrictive.
> > > What
> > > > > if
> > > > > > > >> > there's a
> > > > > > > >> > > > > field
> > > > > > > >> > > > > > > > whose
> > > > > > > >> > > > > > > > > > > name
> > > > > > > >> > > > > > > > > > > > > > > contains
> > > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > > >> > > > > > > > > > > > > > > > > > > of the permitted options
> > > > > > (currently
> > > > > > > >> just
> > > > > > > >> > > ".",
> > > > > > > >> > > > > > ",",
> > > > > > > >> > > > > > > > and
> > > > > > > >> > > > > > > > > > > "/")?
> > > > > > > >> > > > > > > > > > > > > > > > > > > - If we expand the set
> of
> > > > > > permitted
> > > > > > > >> > > > delimiters
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > allow
> > > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > any
> > > > > > > >> > > > > > > > > > > > > > > > > > > single-character string,
> > > > > > > configuration
> > > > > > > >> > > > > complexity
> > > > > > > >> > > > > > > > will
> > > > > > > >> > > > > > > > > > > > increase
> > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > readability may decrease
> > > > > > > >> > > > > > > > > > > > > > > > > > > - Also worth pointing
> out
> > > that
> > > > > > there
> > > > > > > >> is
> > > > > > > >> > > some
> > > > > > > >> > > > > > > > convention
> > > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > doubling
> > > > > > > >> > > > > > > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > > > > > delimiter character as
> an
> > > > escape
> > > > > > > >> > mechanism
> > > > > > > >> > > > with
> > > > > > > >> > > > > > > > formats
> > > > > > > >> > > > > > > > > > > like
> > > > > > > >> > > > > > > > > > > > > CSV
> > > > > > > >> > > > > > > > > > > > > > > [1]
> > > > > > > >> > > > > > > > > > > > > > > > > > > - Overall I think we may
> > be
> > > > > > > >> approaching
> > > > > > > >> > the
> > > > > > > >> > > > > > > > saturation
> > > > > > > >> > > > > > > > > > > point
> > > > > > > >> > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > productive
> > > > > > > >> > > > > > > > > > > > > > > > > > > discussion on delimiter
> > > syntax
> > > > > so
> > > > > > I
> > > > > > > >> don't
> > > > > > > >> > > > want
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > spend
> > > > > > > >> > > > > > > > > > too
> > > > > > > >> > > > > > > > > > > > > much
> > > > > > > >> > > > > > > > > > > > > > > more
> > > > > > > >> > > > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > your time on it. I think
> > the
> > > > one
> > > > > > > point
> > > > > > > >> > I'd
> > > > > > > >> > > > like
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > > leave
> > > > > > > >> > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > now
> > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > > > > > shouldn't be impossible
> to
> > > use
> > > > > > this
> > > > > > > >> new
> > > > > > > >> > > > feature
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > > any
> > > > > > > >> > > > > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > > name,
> > > > > > > >> > > > > > > > > > > > > > > > no
> > > > > > > >> > > > > > > > > > > > > > > > > > > matter how convoluted.
> > It's
> > > > fine
> > > > > > if
> > > > > > > >> edge
> > > > > > > >> > > > cases
> > > > > > > >> > > > > > > > > introduce
> > > > > > > >> > > > > > > > > > > > > > difficulty
> > > > > > > >> > > > > > > > > > > > > > > > > (such
> > > > > > > >> > > > > > > > > > > > > > > > > > > as less-readable
> > > > > configurations),
> > > > > > > but
> > > > > > > >> > it's
> > > > > > > >> > > > not
> > > > > > > >> > > > > > fine
> > > > > > > >> > > > > > > > if
> > > > > > > >> > > > > > > > > > they
> > > > > > > >> > > > > > > > > > > > > can't
> > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > 2.
> > > > > > > >> > > > > > > > > > > > > > > > > > > The configuration style
> > > where
> > > > > you
> > > > > > > >> define
> > > > > > > >> > > > > > > > > > > > > "transforms.field.style"
> > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > connector config, and
> then
> > > > this
> > > > > > > >> applies
> > > > > > > >> > to
> > > > > > > >> > > > all
> > > > > > > >> > > > > > SMTs
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > connector,
> > > > > > > >> > > > > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > > > > very interesting.
> However,
> > > it
> > > > > > > doesn't
> > > > > > > >> > > follow
> > > > > > > >> > > > > > > > convention
> > > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > existing
> > > > > > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > > > > > >> > > > > > > > > > > > > > > > > > > Right now, if you want
> to
> > > > > > configure
> > > > > > > an
> > > > > > > >> > SMT,
> > > > > > > >> > > > you
> > > > > > > >> > > > > > > > define
> > > > > > > >> > > > > > > > > > its
> > > > > > > >> > > > > > > > > > > > name
> > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > connector config (for
> > > example,
> > > > > > > >> > > "transforms":
> > > > > > > >> > > > > > > "smt1"),
> > > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > then
> > > > > > > >> > > > > > > > > > > > > > > define
> > > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > > >> > > > > > > > > > > > > > > > > > > of the properties for
> that
> > > SMT
> > > > > in
> > > > > > > the
> > > > > > > >> > > > connector
> > > > > > > >> > > > > > > > config
> > > > > > > >> > > > > > > > > > > using
> > > > > > > >> > > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > > > > namespacing
> > > > > > > >> > > > > > > > > > > > > > > > > > > mechanism specific to
> that
> > > SMT
> > > > > > (for
> > > > > > > >> > > example,
> > > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > > > > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT then
> > sees
> > > > only
> > > > > > the
> > > > > > > >> > > > properties
> > > > > > > >> > > > > > > > defined
> > > > > > > >> > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > namespace,
> > > > > > > >> > > > > > > > > > > > > > > > > > > with the prefix stripped
> > > (for
> > > > > > > example,
> > > > > > > >> > > > "prop1":
> > > > > > > >> > > > > > > > "val1")
> > > > > > > >> > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > its
> > > > > > > >> > > > > > > > > > > > > > > > > configure
> > > > > > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > > > > > >> > > > > > > > > > > > > > > > > > > If we want to continue
> to
> > > > follow
> > > > > > > this
> > > > > > > >> > > > > convention,
> > > > > > > >> > > > > > > > then
> > > > > > > >> > > > > > > > > > > > instead
> > > > > > > >> > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > specifying
> > > > > > "transforms.field.style"
> > > > > > > >> in a
> > > > > > > >> > > > > > connector
> > > > > > > >> > > > > > > > > > config,
> > > > > > > >> > > > > > > > > > > we
> > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > > > > > expect
> > > > > > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > > > > > >> > > > > > "transforms.<name>.field.style",
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > > each
> > > > > > > >> > > > > > > > > > > > > SMT
> > > > > > > >> > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > they
> > > > > > > >> > > > > > > > > > > > > > > > > > > want to configure a
> field
> > > > style
> > > > > > for.
> > > > > > > >> This
> > > > > > > >> > > > would
> > > > > > > >> > > > > > > > require
> > > > > > > >> > > > > > > > > > > more
> > > > > > > >> > > > > > > > > > > > > work
> > > > > > > >> > > > > > > > > > > > > > > on
> > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > part of the user, but
> > would
> > > be
> > > > > > > >> simpler to
> > > > > > > >> > > > > reason
> > > > > > > >> > > > > > > > about
> > > > > > > >> > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > easier
> > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > implement.
> > > > > > > >> > > > > > > > > > > > > > > > > > > If we want to explore an
> > > > > > alternative
> > > > > > > >> > where
> > > > > > > >> > > > > users
> > > > > > > >> > > > > > > can
> > > > > > > >> > > > > > > > > > > specify
> > > > > > > >> > > > > > > > > > > > > > global
> > > > > > > >> > > > > > > > > > > > > > > > > > > properties that apply to
> > all
> > > > > > > >> transforms
> > > > > > > >> > in
> > > > > > > >> > > a
> > > > > > > >> > > > > > > > connector
> > > > > > > >> > > > > > > > > > > > config,
> > > > > > > >> > > > > > > > > > > > > > then
> > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > semantics for this need
> to
> > > be
> > > > > > > defined
> > > > > > > >> in
> > > > > > > >> > > the
> > > > > > > >> > > > > KIP.
> > > > > > > >> > > > > > > > This
> > > > > > > >> > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > have
> > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > include whether this
> will
> > > > apply
> > > > > > only
> > > > > > > >> for
> > > > > > > >> > > the
> > > > > > > >> > > > > > > special
> > > > > > > >> > > > > > > > > case
> > > > > > > >> > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > "field.style" and
> possibly
> > > > > > > >> > > "field.separator"
> > > > > > > >> > > > > > > > properties
> > > > > > > >> > > > > > > > > > or
> > > > > > > >> > > > > > > > > > > if
> > > > > > > >> > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > > > > available more generally
> > for
> > > > > other
> > > > > > > >> > > > properties,
> > > > > > > >> > > > > > > > whether
> > > > > > > >> > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > > > > apply
> > > > > > > >> > > > > > > > > > > > > > > > > only
> > > > > > > >> > > > > > > > > > > > > > > > > > > for the SMTs outlined in
> > the
> > > > KIP
> > > > > > or
> > > > > > > if
> > > > > > > >> > the
> > > > > > > >> > > > > > > > > "field.style"
> > > > > > > >> > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > possibly
> > > > > > > >> > > > > > > > > > > > > > > > > > > "field.separator"
> > properties
> > > > > would
> > > > > > > >> also
> > > > > > > >> > be
> > > > > > > >> > > > > passed
> > > > > > > >> > > > > > > > into
> > > > > > > >> > > > > > > > > > > custom
> > > > > > > >> > > > > > > > > > > > > > SMTs
> > > > > > > >> > > > > > > > > > > > > > > so
> > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > they could choose to act
> > on
> > > > them
> > > > > > if
> > > > > > > >> > > > applicable,
> > > > > > > >> > > > > > how
> > > > > > > >> > > > > > > > > edge
> > > > > > > >> > > > > > > > > > > > cases
> > > > > > > >> > > > > > > > > > > > > > like
> > > > > > > >> > > > > > > > > > > > > > > > > > having
> > > > > > > >> > > > > > > > > > > > > > > > > > > an SMT named "field" in
> > your
> > > > > > > connector
> > > > > > > >> > > config
> > > > > > > >> > > > > > would
> > > > > > > >> > > > > > > > be
> > > > > > > >> > > > > > > > > > > > handled,
> > > > > > > >> > > > > > > > > > > > > > > etc.
> > > > > > > >> > > > > > > > > > > > > > > > > > > Either way, it might
> help
> > to
> > > > > have
> > > > > > an
> > > > > > > >> > > example
> > > > > > > >> > > > in
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > KIP
> > > > > > > >> > > > > > > > > > > > > outlining
> > > > > > > >> > > > > > > > > > > > > > > how
> > > > > > > >> > > > > > > > > > > > > > > > > one
> > > > > > > >> > > > > > > > > > > > > > > > > > > of the to-be-augmented
> > SMTs
> > > > can
> > > > > be
> > > > > > > >> > > configured
> > > > > > > >> > > > > > with
> > > > > > > >> > > > > > > > this
> > > > > > > >> > > > > > > > > > new
> > > > > > > >> > > > > > > > > > > > > > feature
> > > > > > > >> > > > > > > > > > > > > > > > > and a
> > > > > > > >> > > > > > > > > > > > > > > > > > > before/after of how a
> > record
> > > > > value
> > > > > > > >> would
> > > > > > > >> > be
> > > > > > > >> > > > > > > > transformed
> > > > > > > >> > > > > > > > > > > with
> > > > > > > >> > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for the
> > > > > > > >> > > > > "transforms.field.style"
> > > > > > > >> > > > > > > > > > property
> > > > > > > >> > > > > > > > > > > > > > mentions
> > > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > the permitted values are
> > > > "plain"
> > > > > > and
> > > > > > > >> > > > "nested",
> > > > > > > >> > > > > > but
> > > > > > > >> > > > > > > > then
> > > > > > > >> > > > > > > > > > > > > describes
> > > > > > > >> > > > > > > > > > > > > > > > > > behavior
> > > > > > > >> > > > > > > > > > > > > > > > > > > with a value of "root".
> > > Should
> > > > > > that
> > > > > > > be
> > > > > > > >> > > > "plain"
> > > > > > > >> > > > > > > > instead?
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for the
> > > > > > > >> > > > > > > "transforms.field.separator"
> > > > > > > >> > > > > > > > > > > > property
> > > > > > > >> > > > > > > > > > > > > > > > > > exclusively
> > > > > > > >> > > > > > > > > > > > > > > > > > > mentions structs, but
> the
> > > > > feature
> > > > > > is
> > > > > > > >> > > intended
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > work
> > > > > > > >> > > > > > > > > > with
> > > > > > > >> > > > > > > > > > > > maps
> > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > >> > > > > > > > > > > > > > > > > well.
> > > > > > > >> > > > > > > > > > > > > > > > > > > Can we update it to
> > reflect
> > > > > this?
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > References:
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > > > > > >> > https://stackoverflow.com/a/17808731
> > > > > > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > > > > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > Chris
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022 at
> > 1:32
> > > > PM
> > > > > > > Jorge
> > > > > > > >> > > Esteban
> > > > > > > >> > > > > > > > Quilcate
> > > > > > > >> > > > > > > > > > > Otoya
> > > > > > > >> > > > > > > > > > > > <
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> quilcate.jorge@gmail.com>
> > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> > > > > "this..field.child"
> > > > > > > >> could
> > > > > > > >> > be
> > > > > > > >> > > > > > harder
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > > grasp
> > > > > > > >> > > > > > > > > > > > > than
> > > > > > > >> > > > > > > > > > > > > > > > > > > > "this.field/child" +
> > > > > separator:
> > > > > > > "/".
> > > > > > > >> > > > > > > > > > > > > > > > > > > > Even though this
> > > represents
> > > > > > > >> additional
> > > > > > > >> > > > > > > information,
> > > > > > > >> > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > follows
> > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > > > > similar
> > > > > > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > > > > > "Flatten#delimeter"
> > > > > > > >> > > > > > > configuration.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > I want to give the
> > > separator
> > > > > > > >> approach
> > > > > > > >> > > > another
> > > > > > > >> > > > > > > try,
> > > > > > > >> > > > > > > > > so I
> > > > > > > >> > > > > > > > > > > > have
> > > > > > > >> > > > > > > > > > > > > > > > updated
> > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > KIP with the separator
> > > > > proposal,
> > > > > > > >> > sticking
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > only 2
> > > > > > > >> > > > > > > > > > > > > > alternatives
> > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > > should hopefully cover
> > > most
> > > > > > > >> scenarios.
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has been
> > > > updated
> > > > > > > with
> > > > > > > >> > this
> > > > > > > >> > > > > > > > improvement.
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I
> have
> > > > > updated
> > > > > > > this
> > > > > > > >> > > > section
> > > > > > > >> > > > > > > > > > accordingly.
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch! I've
> > > replaced
> > > > > it
> > > > > > > with
> > > > > > > >> > > > > > > `DropHeaders`.
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > Looking forward to
> your
> > > > > > feedback.
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022 at
> > > 21:33,
> > > > > > Chris
> > > > > > > >> > > Egerton
> > > > > > > >> > > > <
> > > > > > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > Looking good! Got a
> > few
> > > > more
> > > > > > > >> > thoughts.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to revisit
> > > this,
> > > > > but
> > > > > > I
> > > > > > > >> think
> > > > > > > >> > > we
> > > > > > > >> > > > > may
> > > > > > > >> > > > > > > want
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > adopt
> > > > > > > >> > > > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > > > slightly
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > different escape
> > syntax
> > > > > style.
> > > > > > > >> > > > Backslashes
> > > > > > > >> > > > > > are
> > > > > > > >> > > > > > > > > great,
> > > > > > > >> > > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > > > since
> > > > > > > >> > > > > > > > > > > > > > > > > > they're
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > already used by
> JSON,
> > > > using
> > > > > > them
> > > > > > > >> as
> > > > > > > >> > an
> > > > > > > >> > > > > escape
> > > > > > > >> > > > > > > > > > sequence
> > > > > > > >> > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > > > > > > > notation
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > would also lead to
> > some
> > > > > pretty
> > > > > > > >> ugly
> > > > > > > >> > > > > connector
> > > > > > > >> > > > > > > > > > configs.
> > > > > > > >> > > > > > > > > > > > > Anyone
> > > > > > > >> > > > > > > > > > > > > > > > who's
> > > > > > > >> > > > > > > > > > > > > > > > > > had
> > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > write regular
> > > expressions
> > > > > with
> > > > > > > >> > > > backslashes
> > > > > > > >> > > > > in
> > > > > > > >> > > > > > > > Java
> > > > > > > >> > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > probably
> > > > > > > >> > > > > > > > > > > > > > > > > > already
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > familiar with this:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > > > > > >> > > > > > > > > > > > > > > > > What
> > > > > > > >> > > > > > > > > > > > > > > > > > > do
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > you think about
> using
> > > the
> > > > > dot
> > > > > > > >> > character
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > escape
> > > > > > > >> > > > > > > > > > > itself?
> > > > > > > >> > > > > > > > > > > > > In
> > > > > > > >> > > > > > > > > > > > > > > > other
> > > > > > > >> > > > > > > > > > > > > > > > > > > words,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > to access a single
> > field
> > > > > named
> > > > > > > >> > > > > "this.field",
> > > > > > > >> > > > > > > > > instead
> > > > > > > >> > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > using
> > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > syntax
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > "this\.field" (which
> > in
> > > > JSON
> > > > > > > would
> > > > > > > >> > have
> > > > > > > >> > > > to
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > > > > > expressed
> > > > > > > >> > > > > > > > > > > > as
> > > > > > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > > "this..field",
> > > > > > and
> > > > > > > >> for a
> > > > > > > >> > > > > single
> > > > > > > >> > > > > > > > field
> > > > > > > >> > > > > > > > > > > named
> > > > > > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > instead of using the
> > > > syntax
> > > > > > > >> > > "this\\field"
> > > > > > > >> > > > > > (or,
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > > JSON,
> > > > > > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > "this\field"
> > > > > (or,
> > > > > > > in
> > > > > > > >> > JSON,
> > > > > > > >> > > > > > > > > > "this\\field").
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you flesh
> out
> > > the
> > > > > > > >> details on
> > > > > > > >> > > the
> > > > > > > >> > > > > new
> > > > > > > >> > > > > > > > > > > > "field.style"
> > > > > > > >> > > > > > > > > > > > > > > > > property,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > including the type,
> > > > default
> > > > > > > value,
> > > > > > > >> > > > > > importance,
> > > > > > > >> > > > > > > > and
> > > > > > > >> > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > preliminary
> > > > > > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > See
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the
> > > "Compatibility,
> > > > > > > >> > Deprecation,
> > > > > > > >> > > > and
> > > > > > > >> > > > > > > > > Migration
> > > > > > > >> > > > > > > > > > > > Plan"
> > > > > > > >> > > > > > > > > > > > > > > > section
> > > > > > > >> > > > > > > > > > > > > > > > > > > still
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > accurate after the
> > > latest
> > > > > > > update?
> > > > > > > >> > Seems
> > > > > > > >> > > > > like
> > > > > > > >> > > > > > > it's
> > > > > > > >> > > > > > > > > > still
> > > > > > > >> > > > > > > > > > > > > > written
> > > > > > > >> > > > > > > > > > > > > > > > > with
> > > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > assumption that
> nested
> > > > field
> > > > > > > >> syntax
> > > > > > > >> > > will
> > > > > > > >> > > > be
> > > > > > > >> > > > > > > > > hardcoded
> > > > > > > >> > > > > > > > > > > or
> > > > > > > >> > > > > > > > > > > > > > > opt-in,
> > > > > > > >> > > > > > > > > > > > > > > > > > which
> > > > > > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > isn't the case
> > anymore.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The "These
> > SMTs
> > > do
> > > > > not
> > > > > > > >> > require
> > > > > > > >> > > > > nested
> > > > > > > >> > > > > > > > > > structure
> > > > > > > >> > > > > > > > > > > > > > > support"
> > > > > > > >> > > > > > > > > > > > > > > > > > > section
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > mentions a "Drop"
> > SMT. I
> > > > > think
> > > > > > > >> this
> > > > > > > >> > may
> > > > > > > >> > > > be
> > > > > > > >> > > > > > > > > referring
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > Confluent
> > > > > > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > SMT, which isn't a
> > part
> > > of
> > > > > > > Apache
> > > > > > > >> > > Kafka.
> > > > > > > >> > > > > > Should
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > > drop
> > > > > > > >> > > > > > > > > > > > > (heh)
> > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > SMT
> > > > > > > >> > > > > > > > > > > > > > > > > > > > from
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > the list? Or perhaps
> > > just
> > > > > > > replace
> > > > > > > >> it
> > > > > > > >> > > with
> > > > > > > >> > > > > > > > > > > "DropHeaders",
> > > > > > > >> > > > > > > > > > > > > > which
> > > > > > > >> > > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > > > > > currently
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > missing from the
> list
> > > and
> > > > > > > >> shouldn't
> > > > > > > >> > > > require
> > > > > > > >> > > > > > any
> > > > > > > >> > > > > > > > > > > > > nested-field
> > > > > > > >> > > > > > > > > > > > > > > > > related
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28, 2022
> > at
> > > > 2:12
> > > > > > PM
> > > > > > > >> Jorge
> > > > > > > >> > > > > Esteban
> > > > > > > >> > > > > > > > > > Quilcate
> > > > > > > >> > > > > > > > > > > > > Otoya
> > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > quilcate.jorge@gmail.com>
> > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thank you, Chris!
> > and
> > > > > sorry
> > > > > > > for
> > > > > > > >> the
> > > > > > > >> > > > > delayed
> > > > > > > >> > > > > > > > > > response.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Please, find my
> > > comments
> > > > > > > below:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb
> 2022
> > at
> > > > > > 17:34,
> > > > > > > >> Chris
> > > > > > > >> > > > > Egerton
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > <chrise@confluent.io.invalid
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> > KIP!
> > > > I'd
> > > > > > love
> > > > > > > >> to
> > > > > > > >> > see
> > > > > > > >> > > > > > support
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > fields
> > > > > > > >> > > > > > > > > > > > > > > > > > added
> > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > out-of-the-box
> > SMTs
> > > > > > provided
> > > > > > > >> with
> > > > > > > >> > > > > > Connect.
> > > > > > > >> > > > > > > > Here
> > > > > > > >> > > > > > > > > > are
> > > > > > > >> > > > > > > > > > > > my
> > > > > > > >> > > > > > > > > > > > > > > > initial
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree that
> > > > there's
> > > > > a
> > > > > > > >> case to
> > > > > > > >> > > be
> > > > > > > >> > > > > made
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > > > > > expanding
> > > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > new config
> > property
> > > > for
> > > > > > > >> > > identifying a
> > > > > > > >> > > > > > > nested,
> > > > > > > >> > > > > > > > > > > > > > to-be-hoisted
> > > > > > > >> > > > > > > > > > > > > > > > > > field,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example in the
> KIP
> > > > > doesn't
> > > > > > > >> really
> > > > > > > >> > > > > > > demonstrate
> > > > > > > >> > > > > > > > > why
> > > > > > > >> > > > > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd be
> > > helpful
> > > > to
> > > > > > > >> expand
> > > > > > > >> > the
> > > > > > > >> > > > > > example
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > add
> > > > > > > >> > > > > > > > > > > > > other
> > > > > > > >> > > > > > > > > > > > > > > > fields
> > > > > > > >> > > > > > > > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > order
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > show how adding
> > > nested
> > > > > > field
> > > > > > > >> > > support
> > > > > > > >> > > > > > > enables
> > > > > > > >> > > > > > > > > > users
> > > > > > > >> > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > hoist a
> > > > > > > >> > > > > > > > > > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > without dropping
> > > other
> > > > > > > fields
> > > > > > > >> > from
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > value.
> > > > > > > >> > > > > > > > > > Maybe
> > > > > > > >> > > > > > > > > > > > > > > something
> > > > > > > >> > > > > > > > > > > > > > > > > > like
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     source =
> > > > nested.val
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     field = line
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> > (before):
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "nested":
> > > > {
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "val":
> > > > > 42,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "other
> > > > > > val":
> > > > > > > >> 96
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> (after):
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "nested":
> > > > {
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "line":
> > > > > {
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >  "val":
> > > > > > > 42,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  }
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "other
> > > > > > val":
> > > > > > > >> 96
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I think
> > > > "source"
> > > > > > is
> > > > > > > a
> > > > > > > >> > > little
> > > > > > > >> > > > > > > strange
> > > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > new
> > > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property name.
> > Maybe
> > > > > > > >> "hoisted" or
> > > > > > > >> > > > > > > > > "hoisted.field"
> > > > > > > >> > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and 2.:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The example
> > for
> > > > > this
> > > > > > > SMT
> > > > > > > >> is
> > > > > > > >> > > > > updated
> > > > > > > >> > > > > > > and
> > > > > > > >> > > > > > > > > have
> > > > > > > >> > > > > > > > > > > > added
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there a
> > > > reasonable
> > > > > > use
> > > > > > > >> case
> > > > > > > >> > > for
> > > > > > > >> > > > > > > > expanding
> > > > > > > >> > > > > > > > > > > > Flatten
> > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten specific
> > > > fields?
> > > > > > My
> > > > > > > >> > > > > understanding
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > it's
> > > > > > > >> > > > > > > > > > > > > > > mostly
> > > > > > > >> > > > > > > > > > > > > > > > > > > useful
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > writing to
> systems
> > > > like
> > > > > > > >> databases
> > > > > > > >> > > > that
> > > > > > > >> > > > > > > don't
> > > > > > > >> > > > > > > > > > > support
> > > > > > > >> > > > > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > > > > values
> > > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > require
> everything
> > > to
> > > > > be a
> > > > > > > >> flat
> > > > > > > >> > > list
> > > > > > > >> > > > of
> > > > > > > >> > > > > > > > > key-value
> > > > > > > >> > > > > > > > > > > > > pairs.
> > > > > > > >> > > > > > > > > > > > > > > > Being
> > > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a nested
> > > field
> > > > > > > >> wouldn't
> > > > > > > >> > > > provide
> > > > > > > >> > > > > > any
> > > > > > > >> > > > > > > > > > > advantage
> > > > > > > >> > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > use
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Are there other
> > > cases
> > > > > > where
> > > > > > > it
> > > > > > > >> > > would?
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't think
> > we
> > > > > should
> > > > > > > >> > > > > > unconditionally
> > > > > > > >> > > > > > > > > change
> > > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > default
> > > > > > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten. It's a
> > > > > > > >> > > > backwards-incompatible,
> > > > > > > >> > > > > > > > > breaking
> > > > > > > >> > > > > > > > > > > > change
> > > > > > > >> > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > could
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > headaches for
> > users.
> > > > It
> > > > > > > might
> > > > > > > >> be
> > > > > > > >> > > > > > reasonable
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > > change
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > > >> > > > > > > > > > > > > > > > > > > > value
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically
> based
> > on
> > > > > > whether
> > > > > > > >> the
> > > > > > > >> > > user
> > > > > > > >> > > > > has
> > > > > > > >> > > > > > > > > > > specified a
> > > > > > > >> > > > > > > > > > > > > > value
> > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property, but
> > > > > considering
> > > > > > > the
> > > > > > > >> > > > > motivation
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > > > > changing
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > > >> > > > > > > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it creates
> > conflicts
> > > > > with
> > > > > > > the
> > > > > > > >> > > > > > > > to-be-introduced
> > > > > > > >> > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > > > > > > syntax
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > could arise with
> > > > > > downstream
> > > > > > > >> SMTs
> > > > > > > >> > > > > > regardless
> > > > > > > >> > > > > > > > of
> > > > > > > >> > > > > > > > > > > > whether
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > user
> > > > > > > >> > > > > > > > > > > > > > > > > > has
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly
> > > configured
> > > > > > > Flatten
> > > > > > > >> > with
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > "field"
> > > > > > > >> > > > > > > > > > > > > > property), I
> > > > > > > >> > > > > > > > > > > > > > > > > don't
> > > > > > > >> > > > > > > > > > > > > > > > > > > > know
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this would be
> too
> > > > useful
> > > > > > > >> either.
> > > > > > > >> > I
> > > > > > > >> > > > have
> > > > > > > >> > > > > > > some
> > > > > > > >> > > > > > > > > > > thoughts
> > > > > > > >> > > > > > > > > > > > > > below
> > > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > > >> > > > > > > > > > > > > > > > > > how
> > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > handle possible
> > > > > conflicts
> > > > > > > >> between
> > > > > > > >> > > > names
> > > > > > > >> > > > > > > with
> > > > > > > >> > > > > > > > > dots
> > > > > > > >> > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > their
> > > > > > > >> > > > > > > > > > > > > > > > > names
> > > > > > > >> > > > > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for
> nested
> > > > field
> > > > > > > >> > references
> > > > > > > >> > > > that
> > > > > > > >> > > > > > > > should
> > > > > > > >> > > > > > > > > > > > > hopefully
> > > > > > > >> > > > > > > > > > > > > > > make
> > > > > > > >> > > > > > > > > > > > > > > > > > > either
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough. With
> > the
> > > > > > support
> > > > > > > >> for
> > > > > > > >> > > > nested
> > > > > > > >> > > > > > > fields
> > > > > > > >> > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > other
> > > > > > > >> > > > > > > > > > > > > > > SMTs,
> > > > > > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > could stay as it
> is.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > This removes the
> > need
> > > > for
> > > > > > (4)
> > > > > > > >> > > changing
> > > > > > > >> > > > > > > Flatten
> > > > > > > >> > > > > > > > > > config
> > > > > > > >> > > > > > > > > > > > as
> > > > > > > >> > > > > > > > > > > > > > > well.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think it's
> > fine
> > > > to
> > > > > > > expand
> > > > > > > >> > > > > > ExtractField
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > > > support
> > > > > > > >> > > > > > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it might be
> worth
> > > > noting
> > > > > > in
> > > > > > > >> the
> > > > > > > >> > > > > rejected
> > > > > > > >> > > > > > > > > > > alternatives
> > > > > > > >> > > > > > > > > > > > > > > section
> > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > isn't strictly
> > > > necessary
> > > > > > > since
> > > > > > > >> > you
> > > > > > > >> > > > can
> > > > > > > >> > > > > > > > replace
> > > > > > > >> > > > > > > > > > any
> > > > > > > >> > > > > > > > > > > > > single
> > > > > > > >> > > > > > > > > > > > > > > > > > > invocation
> > > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT that
> uses
> > > > > nested
> > > > > > > >> field
> > > > > > > >> > > > > notation
> > > > > > > >> > > > > > > with
> > > > > > > >> > > > > > > > > > > > multiple
> > > > > > > >> > > > > > > > > > > > > > > > > > invocations
> > > > > > > >> > > > > > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that use
> > non-nested
> > > > > > > notation.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding it.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit:
> > > "RegerRouter"
> > > > > > should
> > > > > > > >> be
> > > > > > > >> > > > > > > "RegexRouter"
> > > > > > > >> > > > > > > > > in
> > > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > list
> > > > > > > >> > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > SMTs
> > > > > > > >> > > > > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > do
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > not require
> nested
> > > > > > structure
> > > > > > > >> > > support.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing it.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be
> rare
> > > for
> > > > > dots
> > > > > > > in
> > > > > > > >> > field
> > > > > > > >> > > > > names
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > > occur
> > > > > > > >> > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > wild
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be so
> > > certain
> > > > > of
> > > > > > > >> this),
> > > > > > > >> > > but
> > > > > > > >> > > > > > unless
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > > want
> > > > > > > >> > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > inflict
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > users of
> Flatten,
> > I
> > > > > think
> > > > > > > >> we're
> > > > > > > >> > > going
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > have
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > think
> > > > > > > >> > > > > > > > > > > > > > > about
> > > > > > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between dotted
> > > > notation
> > > > > > and
> > > > > > > >> > > > non-nested
> > > > > > > >> > > > > > > fields
> > > > > > > >> > > > > > > > > > whose
> > > > > > > >> > > > > > > > > > > > > names
> > > > > > > >> > > > > > > > > > > > > > > > > contain
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > don't think this
> > is
> > > > > > actually
> > > > > > > >> > such a
> > > > > > > >> > > > bad
> > > > > > > >> > > > > > > > thing,
> > > > > > > >> > > > > > > > > > > > though.
> > > > > > > >> > > > > > > > > > > > > I
> > > > > > > >> > > > > > > > > > > > > > > > agree
> > > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > notation is
> > > intuitive
> > > > > and
> > > > > > > >> pretty
> > > > > > > >> > > > > > > commonplace
> > > > > > > >> > > > > > > > > (in
> > > > > > > >> > > > > > > > > > > > tools
> > > > > > > >> > > > > > > > > > > > > > like
> > > > > > > >> > > > > > > > > > > > > > > > jq,
> > > > > > > >> > > > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example), so I'd
> > > like
> > > > it
> > > > > > if
> > > > > > > we
> > > > > > > >> > > could
> > > > > > > >> > > > > > stick
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > it.
> > > > > > > >> > > > > > > > > > > > What
> > > > > > > >> > > > > > > > > > > > > > > about
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > escape syntax,
> > > using a
> > > > > > > >> backslash?
> > > > > > > >> > > > That
> > > > > > > >> > > > > > way,
> > > > > > > >> > > > > > > > > users
> > > > > > > >> > > > > > > > > > > > could
> > > > > > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between
> > "this.field"
> > > > > > (which
> > > > > > > >> would
> > > > > > > >> > > > refer
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > field
> > > > > > > >> > > > > > > > > > > > > > > > > > "field"
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > under
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the top-level
> > "this"
> > > > > > field),
> > > > > > > >> and
> > > > > > > >> > > > > > > > "this\.field"
> > > > > > > >> > > > > > > > > > > (which
> > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > > > > refer
> > > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> > > > > "this.field").
> > > > > > > >> Like
> > > > > > > >> > > with
> > > > > > > >> > > > > most
> > > > > > > >> > > > > > > > > > languages
> > > > > > > >> > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > > use
> > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > for escape
> > > sequences,
> > > > it
> > > > > > > could
> > > > > > > >> > also
> > > > > > > >> > > > be
> > > > > > > >> > > > > > used
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > > escape
> > > > > > > >> > > > > > > > > > > > > > > itself,
> > > > > > > >> > > > > > > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that a field
> name
> > > > > > contains a
> > > > > > > >> > > > > backslash. I
> > > > > > > >> > > > > > > > think
> > > > > > > >> > > > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > more
> > > > > > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > simpler than,
> > e.g.,
> > > > > > adding a
> > > > > > > >> new
> > > > > > > >> > > > config
> > > > > > > >> > > > > > > > > property
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > toggle
> > > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > be used when
> > parsing
> > > > > > nested
> > > > > > > >> field
> > > > > > > >> > > > > > > references.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like this
> > approach.
> > > > > Adding
> > > > > > > to
> > > > > > > >> the
> > > > > > > >> > > > KIP.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't think
> > we
> > > > can
> > > > > > > >> > > > unconditionally
> > > > > > > >> > > > > > > turn
> > > > > > > >> > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > > feature
> > > > > > > >> > > > > > > > > > > > > > > on.
> > > > > > > >> > > > > > > > > > > > > > > > > The
> > > > > > > >> > > > > > > > > > > > > > > > > > > > risk
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > breaking
> existing
> > > > > > pipelines
> > > > > > > >> > > > (especially
> > > > > > > >> > > > > > > ones
> > > > > > > >> > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > involve,
> > > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > combination of
> the
> > > > > Flatten
> > > > > > > and
> > > > > > > >> > Cast
> > > > > > > >> > > > > SMTs)
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > > > pretty
> > > > > > > >> > > > > > > > > > > > > > high. I
> > > > > > > >> > > > > > > > > > > > > > > > > think
> > > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > should be an
> > opt-in
> > > > > > feature,
> > > > > > > >> at
> > > > > > > >> > > least
> > > > > > > >> > > > > > until
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > > next
> > > > > > > >> > > > > > > > > > > > > > major
> > > > > > > >> > > > > > > > > > > > > > > > > > release.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > One
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > we could
> > accomplish
> > > > this
> > > > > > is
> > > > > > > by
> > > > > > > >> > > > > > introducing
> > > > > > > >> > > > > > > a
> > > > > > > >> > > > > > > > > new
> > > > > > > >> > > > > > > > > > > > > > > > "field.style"
> > > > > > > >> > > > > > > > > > > > > > > > > > > (name
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > obviously
> subject
> > to
> > > > > > change)
> > > > > > > >> > > property
> > > > > > > >> > > > > > with
> > > > > > > >> > > > > > > > > values
> > > > > > > >> > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > "plain"
> > > > > > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If set
> > to
> > > > > > "plain"
> > > > > > > >> then
> > > > > > > >> > > the
> > > > > > > >> > > > > > > current
> > > > > > > >> > > > > > > > > > > > non-nested
> > > > > > > >> > > > > > > > > > > > > > > > > behavior
> > > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > and if set to
> > > > "nested",
> > > > > > then
> > > > > > > >> the
> > > > > > > >> > > > > proposed
> > > > > > > >> > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > behavior
> > > > > > > >> > > > > > > > > > > > > > > is.
> > > > > > > >> > > > > > > > > > > > > > > > > We
> > > > > > > >> > > > > > > > > > > > > > > > > > > can
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > consider
> updating
> > > the
> > > > > > > default
> > > > > > > >> > value
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > > "nested"
> > > > > > > >> > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > future
> > > > > > > >> > > > > > > > > > > > > > > > > major
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > release
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > (or even codify
> > that
> > > > > plan
> > > > > > in
> > > > > > > >> this
> > > > > > > >> > > > KIP,
> > > > > > > >> > > > > if
> > > > > > > >> > > > > > > > > there's
> > > > > > > >> > > > > > > > > > > > > enough
> > > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > > >> > > > > > > > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > This would also
> > > leave
> > > > > the
> > > > > > > door
> > > > > > > >> > open
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > > adding
> > > > > > > >> > > > > > > > > > > > > recursive
> > > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > > >> > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the future by
> > > adding a
> > > > > > > >> permitted
> > > > > > > >> > > > value
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > > > > > > > "recursive".
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of the
> > linked
> > > > > > tickets
> > > > > > > >> in
> > > > > > > >> > the
> > > > > > > >> > > > > > > > > "Motivation"
> > > > > > > >> > > > > > > > > > > > > section,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-10640
> > > > > ,
> > > > > > > >> > > > > > > > > > > > has
> > > > > > > >> > > > > > > > > > > > > > an
> > > > > > > >> > > > > > > > > > > > > > > > open
> > > > > > > >> > > > > > > > > > > > > > > > > > PR
> > > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that propose
> > adding
> > > > > > > recursive
> > > > > > > >> > > support
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > some
> > > > > > > >> > > > > > > > > > SMTs.
> > > > > > > >> > > > > > > > > > > > > Have
> > > > > > > >> > > > > > > > > > > > > > > you
> > > > > > > >> > > > > > > > > > > > > > > > > > > > considered
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> > > > > functionality
> > > > > > > for
> > > > > > > >> > your
> > > > > > > >> > > > > KIP?
> > > > > > > >> > > > > > Or
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > > your
> > > > > > > >> > > > > > > > > > > > aim
> > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > stick
> > > > > > > >> > > > > > > > > > > > > > > > > > > > solely
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > nested field
> > > support?
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like the
> > > `field.style`
> > > > > > > >> > > configuration
> > > > > > > >> > > > > flag
> > > > > > > >> > > > > > > > > > approach.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for
> pointing
> > > out
> > > > > the
> > > > > > > >> > > `recursive`
> > > > > > > >> > > > > > > > approach.
> > > > > > > >> > > > > > > > > > > Will
> > > > > > > >> > > > > > > > > > > > > add
> > > > > > > >> > > > > > > > > > > > > > > > > `nested`
> > > > > > > >> > > > > > > > > > > > > > > > > > > at
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's
> check
> > > the
> > > > > > demand
> > > > > > > >> for
> > > > > > > >> > > > > > > `recursive`
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > consider
> > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > > as
> > > > > > > >> > > > > > > > > > > > > > > > > > part
> > > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > this or another
> KIP.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I have added the
> > > > following
> > > > > > on
> > > > > > > >> the
> > > > > > > >> > > KIP:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs could
> > > extend
> > > > > > this
> > > > > > > >> > support
> > > > > > > >> > > > > for:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive
> > notation:
> > > > > name a
> > > > > > > >> field
> > > > > > > >> > > and
> > > > > > > >> > > > > > apply
> > > > > > > >> > > > > > > it
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > all
> > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > >> > > > > > > > > > > > > > > > > > across
> > > > > > > >> > > > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > schema matching
> that
> > > > name.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Access to
> arrays:
> > > > Adding
> > > > > > > `[]`
> > > > > > > >> > > > notation
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > > > represent
> > > > > > > >> > > > > > > > > > > > > > access
> > > > > > > >> > > > > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > > > > > > arrays
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs to
> > > fields
> > > > > > within
> > > > > > > >> an
> > > > > > > >> > > > array.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 8,
> > 2022
> > > at
> > > > > > 1:23
> > > > > > > PM
> > > > > > > >> > > Jorge
> > > > > > > >> > > > > > > Esteban
> > > > > > > >> > > > > > > > > > > Quilcate
> > > > > > > >> > > > > > > > > > > > > > > Otoya <
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > quilcate.jorge@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev team,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like to
> > start
> > > a
> > > > > new
> > > > > > > >> > > discussion
> > > > > > > >> > > > > > thread
> > > > > > > >> > > > > > > > on
> > > > > > > >> > > > > > > > > > > Kafka
> > > > > > > >> > > > > > > > > > > > > > > Connect
> > > > > > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP is
> > aimed
> > > to
> > > > > > > include
> > > > > > > >> > > > support
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > > > nested
> > > > > > > >> > > > > > > > > > > > > > > structures
> > > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs — where
> > this
> > > > make
> > > > > > > >> sense.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking
> forward
> > to
> > > > > your
> > > > > > > >> > feedback.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Jorge Esteban Quilcate Otoya <qu...@gmail.com>]

Awesome, thanks Robert!
Will work on updated the proposal around this. Also, @Chris Egerton
<fe...@gmail.com> if you have any feedback about this, would be
much appreciated.

Cheers,
Jorge.

On Fri, 12 Aug 2022 at 18:45, Robert Yokota <ra...@gmail.com> wrote:

> Hi Jorge,
>
> Yes, to escape a backtick I would recommend a backslash.
>
> For example, MySQL allows identifiers to be surrounded with single quotes,
> double quotes, or backticks, and they use backslash to escape.
>
> Jsonata went with primarily backticks as they are less common in strings
> that appear in JSON, and it seems to have worked nicely for them.
>
> Regards,
> Robert
>
> On Fri, Aug 12, 2022 at 10:01 AM Jorge Esteban Quilcate Otoya <
> quilcate.jorge@gmail.com> wrote:
>
> > Thanks Robert! And sorry for the late reply.
> >
> > That's a great catch and will require the current proposal to be updated.
> >
> > Using back-ticks is a good proposal. Though, the challenge we got is that
> > JSON allows really any character into attribute names, including
> > back-ticks.
> > I wonder what's the best approach to escape back-ticks then. Given that
> > those use-cases should be minimal we could fallback to backslash to
> escape
> > back-ticks?
> >
> > What do you think?
> >
> > Many thanks,
> > Jorge.
> >
> >
> > On Wed, 20 Jul 2022 at 23:03, Robert Yokota <ra...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I'm late to this thread, but would like to comment on the double
> > dot/double
> > > asterisk syntax.
> > >
> > > Unfortunately double dot is often used in JSON Path as a descendant
> > > selector, see https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html
> > >
> > > I think a better notation would be to use backticks to quote any names
> > with
> > > a dot, such as  root.`child.with.dot`.  See Jsonata syntax for example:
> > > http://docs.jsonata.org/simple
> > >
> > > Robert
> > >
> > > On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
> > > quilcate.jorge@gmail.com> wrote:
> > >
> > > > Thanks Chris! I have updated the KIP to include this fix.
> > > >
> > > > I will keep the array as a potential improvement at the moment, and
> out
> > > of
> > > > the scope of this KIP.
> > > >
> > > > Thanks,
> > > > Jorge.
> > > >
> > > > On Tue, 28 Jun 2022 at 23:19, Chris Egerton <fearthecellos@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi Jorge,
> > > > >
> > > > > Apologies for the long delay, had my own KIP-related work to focus
> > on.
> > > > >
> > > > > I think it's fine to include array accesses but it's not a blocker.
> > I'm
> > > > +1
> > > > > either way. On that front though, I think the MaskField section
> might
> > > > need
> > > > > to be updated as it still mentions arrays and deep-scan?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jorge@gmail.com> wrote:
> > > > >
> > > > > > Hi there,
> > > > > >
> > > > > > I have update the KIP to the previous state voted, including the
> > > > > > configuration change from `field.style` to
> `field.syntax.version`.
> > > > > >
> > > > > > I'll bump the vote thread as well to see if there's agreement on
> > > adding
> > > > > > this feature to Connect.
> > > > > >
> > > > > > Cheers,
> > > > > > Jorge.
> > > > > >
> > > > > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks, Chris. Your feedback is much appreciated!
> > > > > > >
> > > > > > > I see how the current proposal might be underestimating some
> edge
> > > > > cases.
> > > > > > > I'm happy to move the design for deep-scan and multi-values to
> > > future
> > > > > > > developments related with this KIP and reduce its scope, though
> > > open
> > > > > for
> > > > > > > more feedback.
> > > > > > >
> > > > > > > Also, just to be sure, are you proposing also to not include
> > array
> > > > > access
> > > > > > > at this stage?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jorge.
> > > > > > >
> > > > > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <
> > > fearthecellos@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Jorge,
> > > > > > >>
> > > > > > >> I've done some more thinking and I hate to say it, but I think
> > the
> > > > > > syntax
> > > > > > >> does need to be expanded. Right now it's clear what "a.b"
> refers
> > > to
> > > > > and
> > > > > > >> what "a..b" refers to, but what about "a...b"? Is that
> referring
> > > to
> > > > > > >> subfield ".b" of field "a", or subfield "b" of field "a."?
> This
> > > gets
> > > > > > even
> > > > > > >> more complicated when thinking about fields whose names are
> > > > > exclusively
> > > > > > >> made up of dots.
> > > > > > >>
> > > > > > >> I'm also a little hesitant to mix the cases of multi-value
> paths
> > > and
> > > > > > deep
> > > > > > >> scans. What if you only want to access one subfield deep for
> an
> > > SMT,
> > > > > > >> instead of recursing through all the children of a given
> field?
> > > It's
> > > > > > akin
> > > > > > >> to the distinction between * and ** with file globbing
> patterns,
> > > and
> > > > > > there
> > > > > > >> could be a substantial performance difference if you have
> > > > > heavily-nested
> > > > > > >> fields.
> > > > > > >>
> > > > > > >> Ultimately, I think that if the proposed
> "field.syntax.version"
> > > > > property
> > > > > > >> sits well with people, it might be better to reduce the scope
> of
> > > the
> > > > > KIP
> > > > > > >> back to the original proposal and just focus on adding support
> > for
> > > > > > >> explicitly-specified nested values, with no multi-value paths
> > > > > > whatsoever,
> > > > > > >> knowing that we have an easy way to introduce new syntax and
> > > > features
> > > > > in
> > > > > > >> the future. (We could probably leave the "a...b" case for that
> > > next
> > > > > > >> version
> > > > > > >> too.)
> > > > > > >>
> > > > > > >> I was a huge fan of this KIP before we started trying to
> address
> > > > more
> > > > > > >> complex use cases, and although I don't want to write those
> > off, I
> > > > > think
> > > > > > >> we
> > > > > > >> may have bitten off more than we can chew in time for the
> 3.3.0
> > > > > release
> > > > > > >> and
> > > > > > >> would hate to see this KIP get delayed as a result.
> > > > > > >>
> > > > > > >> I'd be really curious to hear from Joshua and Tom on this
> front,
> > > > > though.
> > > > > > >> Is
> > > > > > >> it acceptable to move more incrementally here and settle on
> the
> > > > syntax
> > > > > > >> version property as our means of introducing new features, or
> is
> > > it
> > > > > > >> preferable to implement things monolithically and try to get
> > > > > everything
> > > > > > >> (or
> > > > > > >> at least, as much as possible) right the first time?
> > > > > > >>
> > > > > > >> Thanks again for your continued effort on this KIP!
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >>
> > > > > > >> Chris
> > > > > > >>
> > > > > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate Otoya <
> > > > > > >> quilcate.jorge@gmail.com> wrote:
> > > > > > >>
> > > > > > >> > Thanks, Chris!
> > > > > > >> >
> > > > > > >> > Please, find my comments below:
> > > > > > >> >
> > > > > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> > > > fearthecellos@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Jorge,
> > > > > > >> > >
> > > > > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > > > > >> > >
> > > > > > >> > > 1. Under the "Accessing multiple values by deep-scan"
> header
> > > > it's
> > > > > > >> stated
> > > > > > >> > > that "If deep-scan is used, it must have only one field
> > after
> > > > the
> > > > > > >> > asterisk
> > > > > > >> > > level.". However, in example 3 for the Cast SMT and other
> > > > examples
> > > > > > for
> > > > > > >> > > other SMTs, the spec contains a field of "*.child.k2",
> which
> > > > > appears
> > > > > > >> to
> > > > > > >> > > have two fields after the asterisk level. I may be
> > > > > misunderstanding
> > > > > > >> the
> > > > > > >> > > proposal, but it seems like the two contradict each other.
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > Thanks for catching this. I have clarified it by removing
> this
> > > > > > >> restriction.
> > > > > > >> > Also, have extended the deep-scan scenarios.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > 2. I'm a little unclear on why we need the special
> handling
> > > for
> > > > > > arrays
> > > > > > >> > > where, for an array field "a", the field name "a" can be
> > > treated
> > > > > as
> > > > > > >> > either
> > > > > > >> > > the array itself, or every element in the array. Is there
> a
> > > > reason
> > > > > > we
> > > > > > >> > can't
> > > > > > >> > > use the field name "a.*" to handle the latter case, and
> "a"
> > to
> > > > > > handle
> > > > > > >> the
> > > > > > >> > > former?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > Agree, this is confusing. I like the `a.*` approach to
> access
> > > > array
> > > > > > >> items.
> > > > > > >> > I have added this to the proposal.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > 3. How would a user specify that they'd like to access a
> > field
> > > > > with
> > > > > > >> the
> > > > > > >> > > literal name "*"?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > Good one. I'm proposing an approach similar to how it's
> > proposed
> > > > to
> > > > > > >> escape
> > > > > > >> > dots, with a double-asterisk. Curious on your thoughts
> around
> > > > this.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > 4. For the Cast SMT, do you think it might bite some
> people
> > if
> > > > > > fields
> > > > > > >> > that
> > > > > > >> > > can't be cast correctly are silently ignored? I'm
> imagining
> > > the
> > > > > case
> > > > > > >> > where
> > > > > > >> > > none of the fields in a multi-path expression can be cast
> > > > > correctly
> > > > > > >> and
> > > > > > >> > it
> > > > > > >> > > ends up eating half of someone's day to track down why
> their
> > > SMT
> > > > > > isn't
> > > > > > >> > > doing anything.
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > If I understand correctly, this challenge could be relevant
> > > across
> > > > > > SMTs.
> > > > > > >> > At the moment, most/all? SMTs just silently ignore.
> > > > > > >> > Was thinking about adding a flag `field.on.path.not.found`
> to
> > > > either
> > > > > > >> ignore
> > > > > > >> > or fail when no paths are found. What do your think?
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > 5. For the ExtractField and ValueToKey SMTs, what happens
> > if a
> > > > > > >> deep-scan
> > > > > > >> > > field name is used, but only one field is found? Is the
> > > > resulting
> > > > > > >> field
> > > > > > >> > > still an array, or is it just the single field that was
> > found?
> > > > > (FWIW
> > > > > > >> I'm
> > > > > > >> > > leaning towards keeping it an array just to keep schemas
> > > > > consistent
> > > > > > >> in a
> > > > > > >> > > pipeline in case the number of fields found fluctuates
> > across
> > > > > > >> records.)
> > > > > > >> > >
> > > > > > >> > > Agree. Will clarify that an array is always produced even
> > for
> > > 1
> > > > > or 0
> > > > > > >> > fields are found.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that "As this
> > SMT
> > > > > > affects
> > > > > > >> > only
> > > > > > >> > > existing fields, additional configurations will not be
> > > > required.".
> > > > > > >> Given
> > > > > > >> > > the new "field.syntax.version" property, this part should
> > > > probably
> > > > > > be
> > > > > > >> > > removed.
> > > > > > >> > >
> > > > > > >> > Agree.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > 7. Is recursive descent intentionally excluded? That was
> an
> > > > > > important
> > > > > > >> > part
> > > > > > >> > > of Joshua's KIP and his feedback on this KIP; I think it's
> > > worth
> > > > > > >> pursuing
> > > > > > >> > > if we can.
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > My understanding from Joshua's feedback is that by including
> > > > support
> > > > > > for
> > > > > > >> > deep-scan, we are already covering the recursive
> > functionality.
> > > > > > Though,
> > > > > > >> I
> > > > > > >> > may be missing something.
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Cheers,
> > > > > > >> > >
> > > > > > >> > > Chris
> > > > > > >> > >
> > > > > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate
> > Otoya <
> > > > > > >> > > quilcate.jorge@gmail.com> wrote:
> > > > > > >> > >
> > > > > > >> > > > Thanks, Chris!
> > > > > > >> > > >
> > > > > > >> > > > I have updated the KIP with the rejected alternatives
> > > updated.
> > > > > > >> Also, I
> > > > > > >> > > have
> > > > > > >> > > > drafted the support for arrays and deep scans as part of
> > the
> > > > > > >> proposed
> > > > > > >> > > > notation to make it more complete, even though these can
> > be
> > > > > > >> implemented
> > > > > > >> > > in
> > > > > > >> > > > multiple PRs.
> > > > > > >> > > >
> > > > > > >> > > > Looking forward to your feedback.
> > > > > > >> > > >
> > > > > > >> > > > Cheers,
> > > > > > >> > > > Jorge.
> > > > > > >> > > >
> > > > > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > > > > >> fearthecellos@gmail.com>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Jorge,
> > > > > > >> > > > >
> > > > > > >> > > > > I really appreciate the effort you've made to simplify
> > the
> > > > > > syntax
> > > > > > >> and
> > > > > > >> > > > > feature set of a JSONPath-based approach as much as
> > > > possible.
> > > > > > I'm
> > > > > > >> > still
> > > > > > >> > > > > hesitant to continue with it, though.
> > > > > > >> > > > >
> > > > > > >> > > > > 1. The syntax is much less friendly. Just compare
> > > > > > >> "top.mid.bottom" to
> > > > > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses
> > JSONPath
> > > or
> > > > > > even
> > > > > > >> > JSON,
> > > > > > >> > > > and
> > > > > > >> > > > > the learning curve for the former is going to be
> > steeper.
> > > > The
> > > > > > >> > examples
> > > > > > >> > > in
> > > > > > >> > > > > the new KIP draft you published demonstrate this
> pretty
> > > > well,
> > > > > > and
> > > > > > >> > this
> > > > > > >> > > is
> > > > > > >> > > > > without diving into the details of what escape syntax
> > > would
> > > > > look
> > > > > > >> > like.
> > > > > > >> > > > >
> > > > > > >> > > > > 2. The three new major features that this syntax adds
> > > (array
> > > > > > >> > accesses,
> > > > > > >> > > > deep
> > > > > > >> > > > > scans, and multi-value paths) could all be added
> pretty
> > > > easily
> > > > > > to
> > > > > > >> the
> > > > > > >> > > dot
> > > > > > >> > > > > notation syntax without introducing brackets and
> dollar
> > > > signs.
> > > > > > >> Array
> > > > > > >> > > > > accesses can be described using the same syntax as
> > > > struct/map
> > > > > > >> field
> > > > > > >> > > > access,
> > > > > > >> > > > > deep scans can be described using '*', and multi-value
> > > paths
> > > > > can
> > > > > > >> be
> > > > > > >> > > > > described by referencing the name of a field that's
> > > expected
> > > > > to
> > > > > > >> have
> > > > > > >> > > > > children. These are all top-of-the-head ideas and can
> > > > probably
> > > > > > be
> > > > > > >> > > > refined,
> > > > > > >> > > > > but hopefully they demonstrate that we can keep the
> > syntax
> > > > > > simple
> > > > > > >> > > without
> > > > > > >> > > > > sacrificing features. Of course, the question of
> leaving
> > > > room
> > > > > > for
> > > > > > >> > > future
> > > > > > >> > > > > features might arise... given that these are the
> > > > > out-of-the-box
> > > > > > >> SMTs
> > > > > > >> > > that
> > > > > > >> > > > > are likely to be the first that many people encounter,
> > I'd
> > > > err
> > > > > > on
> > > > > > >> the
> > > > > > >> > > > side
> > > > > > >> > > > > of simplicity and a gentle learning curve; if people
> > need
> > > to
> > > > > get
> > > > > > >> more
> > > > > > >> > > out
> > > > > > >> > > > > of transforms, the option to implement their own is
> > still
> > > > > there.
> > > > > > >> If
> > > > > > >> > we
> > > > > > >> > > > can
> > > > > > >> > > > > address 95% of use cases with something easy to use,
> > it's
> > > > not
> > > > > > >> worth
> > > > > > >> > > > making
> > > > > > >> > > > > the feature harder for everyone to use just to
> > accommodate
> > > > the
> > > > > > >> > > remaining
> > > > > > >> > > > > 5%.
> > > > > > >> > > > >
> > > > > > >> > > > > 3. The advantage of leveraging an existing syntax is
> > > > twofold:
> > > > > > >> users
> > > > > > >> > who
> > > > > > >> > > > are
> > > > > > >> > > > > already familiar with that syntax don't need to learn
> a
> > > new
> > > > > > >> syntax,
> > > > > > >> > and
> > > > > > >> > > > > maintainers of the syntax get to leverage existing
> > > libraries
> > > > > and
> > > > > > >> > > > > documentation for the syntax. With the current
> proposal,
> > > > > reusing
> > > > > > >> > > > libraries
> > > > > > >> > > > > is off the table, which means that we're going to have
> > to
> > > > > parse
> > > > > > >> this
> > > > > > >> > > > > ourselves (including all the escape syntax edge
> cases),
> > > and
> > > > we
> > > > > > >> won't
> > > > > > >> > be
> > > > > > >> > > > > able to automatically leverage new features added to
> > that
> > > > > > syntax.
> > > > > > >> And
> > > > > > >> > > > given
> > > > > > >> > > > > how stripped-down the syntax is in comparison to full
> > > > > JSONPath,
> > > > > > >> > there's
> > > > > > >> > > > > still going to be a learning curve for users who are
> > > already
> > > > > > >> familiar
> > > > > > >> > > > with
> > > > > > >> > > > > it, and we'll still have to document how Connect's
> > variant
> > > > of
> > > > > > >> > JSONPath
> > > > > > >> > > > > works either instead of or in addition to just linking
> > to
> > > a
> > > > > > >> > > > well-maintained
> > > > > > >> > > > > third-party docs site.
> > > > > > >> > > > >
> > > > > > >> > > > > On the topic of "field.path" and "include.path" vs.
> > > > > > >> "field.style", I
> > > > > > >> > > > > actually think that a single property per SMT is
> cleaner
> > > and
> > > > > > >> simpler.
> > > > > > >> > > > > Allowing users to mix and match styles within the same
> > SMT
> > > > > > config
> > > > > > >> > seems
> > > > > > >> > > > > like a recipe for confusion, and with a single
> property
> > > that
> > > > > > >> dictates
> > > > > > >> > > > field
> > > > > > >> > > > > syntax behavior, we leave the door open to change the
> > > > default
> > > > > > at a
> > > > > > >> > > later
> > > > > > >> > > > > date. We could even fully remove support for plain
> field
> > > > > > notation
> > > > > > >> at
> > > > > > >> > > some
> > > > > > >> > > > > point and still be able to retain the simple property
> > > names
> > > > of
> > > > > > >> > "field",
> > > > > > >> > > > > "include", etc. instead of forcing people to use
> > > > "field.path"
> > > > > > and
> > > > > > >> the
> > > > > > >> > > > like.
> > > > > > >> > > > > That said, the term "field.style" and the permitted
> > values
> > > > > might
> > > > > > >> be a
> > > > > > >> > > > > little ambiguous. One alternative, though a little
> > > > > heavy-handed,
> > > > > > >> is
> > > > > > >> > to
> > > > > > >> > > > > change it to "field.syntax.version" with permitted
> > values
> > > of
> > > > > > "V1"
> > > > > > >> > > > (default,
> > > > > > >> > > > > equivalent to "field.style = plain") and "V2"
> > (equivalent
> > > to
> > > > > > >> > > > "field.style =
> > > > > > >> > > > > nested"). This would leave us room in the future to
> make
> > > > > further
> > > > > > >> > > changes
> > > > > > >> > > > to
> > > > > > >> > > > > the syntax without having to come up with new names,
> > > > although
> > > > > it
> > > > > > >> does
> > > > > > >> > > > > sacrifice a little bit in that the permitted values
> are
> > no
> > > > > > longer
> > > > > > >> > > > > self-describing.
> > > > > > >> > > > >
> > > > > > >> > > > > Cheers,
> > > > > > >> > > > >
> > > > > > >> > > > > Chris
> > > > > > >> > > > >
> > > > > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban Quilcate
> > > > Otoya <
> > > > > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Thank you all for your feedback, and sorry for the
> > long
> > > > wait
> > > > > > >> for a
> > > > > > >> > > > reply.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I would like to explore the idea of
> > > > JSONPath-inspired/subset
> > > > > > >> > > notation a
> > > > > > >> > > > > bit
> > > > > > >> > > > > > further:
> > > > > > >> > > > > >
> > > > > > >> > > > > > It will need to be a much-reduced version of
> JSONPath:
> > > > > > >> > > > > > - No full support for JsonPath therefore an
> additional
> > > > > > >> dependency.
> > > > > > >> > > > > > - All paths must start with '$'
> > > > > > >> > > > > > - No functions support or other operators allowed.
> > > > > > >> > > > > > - JsonPath dotted and square-bracket notations can
> be
> > > > > > supported
> > > > > > >> to
> > > > > > >> > > > avoid
> > > > > > >> > > > > > escaping dots or other characters: `$a.b.c` and
> > > > > > `$['a.b']['c']`
> > > > > > >> -
> > > > > > >> > or
> > > > > > >> > > we
> > > > > > >> > > > > > could only support the second one as it's more
> > complete.
> > > > > > >> > > > > > - Add support for arrays with `[<integer>]`, e.g.
> > > > `$a.[1].b`
> > > > > > >> > > > > > - Add support for multiple-value paths using array
> > > access
> > > > > > >> `$a.*.b`
> > > > > > >> > or
> > > > > > >> > > > > deep
> > > > > > >> > > > > > scan `$a..b`.
> > > > > > >> > > > > >     - Some SMTs that will benefit from this:
> > > `MaskField`,
> > > > > > >> `Cast`,
> > > > > > >> > > > > > `ReplaceField`.
> > > > > > >> > > > > > - We could introduce a `path[s]` config under the
> > > current
> > > > > > >> > > > configurations
> > > > > > >> > > > > to
> > > > > > >> > > > > > apply this feature so no compatibility issues are
> > > > > introduced:
> > > > > > >> e.g.
> > > > > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > > > > >> > > > > >
> > > > > > >> > > > > > With these,  100, 101, 102, and 103 will be
> > effectively
> > > > > > solved.
> > > > > > >> > > > > >
> > > > > > >> > > > > > The main challenge that I see at the moment is that
> by
> > > > being
> > > > > > >> JSON
> > > > > > >> > > > > > path-like, there may be some edge cases that I can't
> > > > foresee
> > > > > > at
> > > > > > >> the
> > > > > > >> > > > > moment,
> > > > > > >> > > > > > that could make this hard to implement, test, and
> > > maintain
> > > > > in
> > > > > > >> the
> > > > > > >> > > long
> > > > > > >> > > > > run.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I'll appreciate your feedback on how this
> > JSONPath-based
> > > > > > >> > alternative
> > > > > > >> > > > > > compares to the dotted notation initially proposed,
> > and
> > > if
> > > > > it
> > > > > > >> > matches
> > > > > > >> > > > > your
> > > > > > >> > > > > > feedback.
> > > > > > >> > > > > > I have drafted a copy of the KIP to change the
> > examples
> > > to
> > > > > > >> JSONPath
> > > > > > >> > > > > > approach and validate some differences:
> > > > > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > > > > >> > > > > >
> > > > > > >> > > > > > About 104. I agreed with Chris. We can handle this
> as
> > > part
> > > > > of
> > > > > > a
> > > > > > >> new
> > > > > > >> > > > KIP.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > Jorge.
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > > > > >> > fearthecellos@gmail.com
> > > > > > >> > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Joshua,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I have a few reservations about using JsonPath
> > > notation
> > > > > > here.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 1. There's likely to be a substantial performance
> > > > penalty
> > > > > > for
> > > > > > >> > > > > converting
> > > > > > >> > > > > > > between the Kafka Connect format and something
> that
> > a
> > > > > > JsonPath
> > > > > > >> > > > library
> > > > > > >> > > > > > > would understand.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 2. The complexity of the feature will be
> > significantly
> > > > > > >> higher. It
> > > > > > >> > > > will
> > > > > > >> > > > > be
> > > > > > >> > > > > > > harder to test, document, and implement. There
> will
> > be
> > > > > many
> > > > > > >> more
> > > > > > >> > > edge
> > > > > > >> > > > > > cases
> > > > > > >> > > > > > > to consider and support, and we'll be on the hook
> to
> > > > > handle
> > > > > > >> any
> > > > > > >> > > bugs
> > > > > > >> > > > or
> > > > > > >> > > > > > > inconsistencies that arise, either as a result of
> > our
> > > > use
> > > > > of
> > > > > > >> the
> > > > > > >> > > > > JsonPath
> > > > > > >> > > > > > > library we choose, or as a result of bugs in that
> > > > library.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 3. It's not clear that JsonPath is superior or
> even
> > > > > suitable
> > > > > > >> for
> > > > > > >> > > some
> > > > > > >> > > > > of
> > > > > > >> > > > > > > the SMTs proposed here. What would be the
> advantages
> > > of
> > > > > > >> JsonPath
> > > > > > >> > > with
> > > > > > >> > > > > the
> > > > > > >> > > > > > > InsertField or HoistField SMTs?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I also don't think that adding dot notation is
> > > > unfriendly
> > > > > to
> > > > > > >> > users;
> > > > > > >> > > > > many
> > > > > > >> > > > > > > have proposed this type of syntax in the past, and
> > > it's
> > > > > > >> > frequently
> > > > > > >> > > > used
> > > > > > >> > > > > > in
> > > > > > >> > > > > > > informal discussions to refer to nested fields. If
> > the
> > > > > > >> proposed
> > > > > > >> > > > syntax
> > > > > > >> > > > > > was
> > > > > > >> > > > > > > not already intuitive then a case against deciding
> > on
> > > > our
> > > > > > own
> > > > > > >> > might
> > > > > > >> > > > be
> > > > > > >> > > > > > more
> > > > > > >> > > > > > > convincing, but as things stand, simple dot
> notation
> > > is
> > > > > > likely
> > > > > > >> > > going
> > > > > > >> > > > to
> > > > > > >> > > > > > be
> > > > > > >> > > > > > > easier for users to understand than JsonPath
> syntax.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Cheers,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Chris
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua Grisham <
> > > > > > >> > > > grishamjosh@gmail.com>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hello all!
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Sorry that I come a bit later to the party here,
> > > but I
> > > > > am
> > > > > > >> the
> > > > > > >> > one
> > > > > > >> > > > who
> > > > > > >> > > > > > > wrote
> > > > > > >> > > > > > > > KIP-683 [1] for recursive support (just simply
> > > looping
> > > > > > >> through
> > > > > > >> > > all
> > > > > > >> > > > > > child
> > > > > > >> > > > > > > > non-primitive structures for the same matching
> > > > name(s))
> > > > > > >> which
> > > > > > >> > is
> > > > > > >> > > a
> > > > > > >> > > > > > > slightly
> > > > > > >> > > > > > > > different way to try and solve a similar
> > requirement
> > > > --
> > > > > > >> > > > unfortunately
> > > > > > >> > > > > > at
> > > > > > >> > > > > > > > the time the dev community was not quite as
> active
> > > and
> > > > > > then
> > > > > > >> I
> > > > > > >> > > also
> > > > > > >> > > > > got
> > > > > > >> > > > > > > busy
> > > > > > >> > > > > > > > with work and just life in general so wasn't
> able
> > to
> > > > > > follow
> > > > > > >> up
> > > > > > >> > or
> > > > > > >> > > > > push
> > > > > > >> > > > > > > it.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I do think it is a very good idea to have some
> > kind
> > > of
> > > > > > >> > path-like
> > > > > > >> > > > > > > expression
> > > > > > >> > > > > > > > to be able to specifically address a nested
> field,
> > > as
> > > > I
> > > > > > can
> > > > > > >> see
> > > > > > >> > > > that
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > simple "recursive" case could potentially result
> > in
> > > > > > >> unwanted or
> > > > > > >> > > > > > > unexpected
> > > > > > >> > > > > > > > behavior, plus there is the potential to
> > introduce a
> > > > bit
> > > > > > of
> > > > > > >> a
> > > > > > >> > > > > > performance
> > > > > > >> > > > > > > > hit to always loop through everything in cases
> > where
> > > > the
> > > > > > >> > > > > schemas/values
> > > > > > >> > > > > > > > might be quite large.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > One thing I wanted to ask: instead of creating a
> > new
> > > > > "path
> > > > > > >> > > parser"
> > > > > > >> > > > > > > > including some bespoke or "borrowed" syntax, why
> > not
> > > > > just
> > > > > > >> use
> > > > > > >> > > > > something
> > > > > > >> > > > > > > > that already exists? Specifically here I am
> > thinking
> > > > > about
> > > > > > >> > > > JsonPath (
> > > > > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > There is already quite nice support in JsonPath
> > for
> > > > > > handling
> > > > > > >> > > > special
> > > > > > >> > > > > > > > characters in field names, for handling
> different
> > > > > > >> non-primitive
> > > > > > >> > > > types
> > > > > > >> > > > > > > > (arrays etc), for handling multiple levels of
> > > nesting,
> > > > > etc
> > > > > > >> etc.
> > > > > > >> > > > > Would
> > > > > > >> > > > > > it
> > > > > > >> > > > > > > > be possible to instead to re-think this and
> maybe
> > > have
> > > > > > some
> > > > > > >> > kind
> > > > > > >> > > of
> > > > > > >> > > > > > > > JsonPath-based Schema selector / updater and/or
> > > > > > >> JsonPath-based
> > > > > > >> > > > Value
> > > > > > >> > > > > > > > selector / updater? Conceptually this feels like
> > it
> > > > > makes
> > > > > > >> sense
> > > > > > >> > > to
> > > > > > >> > > > > me,
> > > > > > >> > > > > > as
> > > > > > >> > > > > > > > from the top of my head it would be quite a
> > natural
> > > > fit
> > > > > to
> > > > > > >> map
> > > > > > >> > a
> > > > > > >> > > > Json
> > > > > > >> > > > > > > data
> > > > > > >> > > > > > > > structure to the Connect API data structure (and
> > you
> > > > > could
> > > > > > >> > > > > potentially
> > > > > > >> > > > > > > even
> > > > > > >> > > > > > > > try to leverage the existing Json-to-Connect
> > > > > > >> > > > serializer/deserializer
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > help out with this even in a more "out of the
> > > > > box"-feeling
> > > > > > >> kind
> > > > > > >> > > of
> > > > > > >> > > > > > way).
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Maybe also as Tom mentioned, this part (in my
> > > example,
> > > > > > this
> > > > > > >> > > > > > > JsonPath-based
> > > > > > >> > > > > > > > "thing") could even be a generic API that could
> be
> > > > used
> > > > > by
> > > > > > >> any
> > > > > > >> > > SMT,
> > > > > > >> > > > > > > > including used in custom ones built by the
> > > community.
> > > > > > Then
> > > > > > >> I
> > > > > > >> > > think
> > > > > > >> > > > > to
> > > > > > >> > > > > > > use
> > > > > > >> > > > > > > > a completely separate config property somehow
> > > related
> > > > to
> > > > > > >> "path"
> > > > > > >> > > (as
> > > > > > >> > > > > Tom
> > > > > > >> > > > > > > > also mentioned) would also make a lot of sense
> > here
> > > as
> > > > > > well.
> > > > > > >> > This
> > > > > > >> > > > > way,
> > > > > > >> > > > > > if
> > > > > > >> > > > > > > > you select based on "path" then this
> > JsonPath-based
> > > > API
> > > > > > >> would
> > > > > > >> > be
> > > > > > >> > > > > used,
> > > > > > >> > > > > > > > otherwise it could use something similar to the
> > > > existing
> > > > > > >> > > get-field
> > > > > > >> > > > > > based
> > > > > > >> > > > > > > > approach (which I guess could also be refactored
> > > into
> > > > > some
> > > > > > >> kind
> > > > > > >> > > of
> > > > > > >> > > > > > > utility
> > > > > > >> > > > > > > > / API as well if it made sense?)
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > And with that in mind, if this was the kind of
> > > > direction
> > > > > > to
> > > > > > >> go,
> > > > > > >> > > > then
> > > > > > >> > > > > a
> > > > > > >> > > > > > > > "recursive" capability like I pitched in KIP-683
> > > would
> > > > > > also
> > > > > > >> > > become
> > > > > > >> > > > > > > > unnecessary because you could easily write a
> > > JsonPath
> > > > > > >> > expression
> > > > > > >> > > > like
> > > > > > >> > > > > > > > "$..someRecuriveField" and it would do the same
> > > thing
> > > > > (on
> > > > > > >> top
> > > > > > >> > of
> > > > > > >> > > > > > anything
> > > > > > >> > > > > > > > else you would want to do that is already
> > supported
> > > by
> > > > > > >> > JsonPath).
> > > > > > >> > > > > Then
> > > > > > >> > > > > > we
> > > > > > >> > > > > > > > could also kill that older KIP and do a bit of
> > > > clean-up
> > > > > :)
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > [1] -
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Just some extra food for thought. All-in-all I
> > think
> > > > > this
> > > > > > >> is a
> > > > > > >> > > > super
> > > > > > >> > > > > > > great
> > > > > > >> > > > > > > > initiative!
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Best,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Joshua
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris
> Egerton
> > <
> > > > > > >> > > > > > > fearthecellos@gmail.com
> > > > > > >> > > > > > > > >:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Tom,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks for taking a look at this, and for your
> > > > > > thoughtful
> > > > > > >> > > > comments.
> > > > > > >> > > > > > > I'll
> > > > > > >> > > > > > > > > leave it up to Jorge to address most of your
> > > > comments
> > > > > > but
> > > > > > >> I
> > > > > > >> > > > wanted
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > share
> > > > > > >> > > > > > > > > a couple quick thoughts I had regarding 103
> and
> > > 104.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 103. Like you, I was envisioning a possible
> > syntax
> > > > for
> > > > > > >> array
> > > > > > >> > > > access
> > > > > > >> > > > > > > that
> > > > > > >> > > > > > > > > used classic C-style brackets; e.g.,
> > `arr[index]`.
> > > > > > >> However, I
> > > > > > >> > > > > wonder
> > > > > > >> > > > > > if
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > could keep things simple and use the same
> syntax
> > > > that
> > > > > > >> we're
> > > > > > >> > > > > proposing
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > nested field access? In other words, instead
> of
> > > > > > >> `arr[index]`,
> > > > > > >> > > > you'd
> > > > > > >> > > > > > > write
> > > > > > >> > > > > > > > > `arr.index`. It'd save us and users the
> headache
> > > of
> > > > > > >> reserving
> > > > > > >> > > > > > > characters
> > > > > > >> > > > > > > > > now that would need to be escaped even if
> their
> > > > > > unescaped
> > > > > > >> > > > brethren
> > > > > > >> > > > > > > aren't
> > > > > > >> > > > > > > > > used for anything, and also avoid the question
> > of
> > > > what
> > > > > > >> > exactly
> > > > > > >> > > we
> > > > > > >> > > > > > > should
> > > > > > >> > > > > > > > do
> > > > > > >> > > > > > > > > when we see a config that uses reserved
> > characters
> > > > > that
> > > > > > >> > aren't
> > > > > > >> > > > yet
> > > > > > >> > > > > > > > > supported (throwing an exception seems pretty
> > > > > unfriendly
> > > > > > >> for
> > > > > > >> > > new
> > > > > > >> > > > > > > users).
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 104. This would probably be useful, but it
> would
> > > > come
> > > > > > with
> > > > > > >> > some
> > > > > > >> > > > > nasty
> > > > > > >> > > > > > > > > compatibility questions that would need to be
> > > > > addressed
> > > > > > if
> > > > > > >> > we'd
> > > > > > >> > > > > want
> > > > > > >> > > > > > > SMTs
> > > > > > >> > > > > > > > > that leverage this new API to be viable for
> > older
> > > > > > >> versions of
> > > > > > >> > > > > > Connect.
> > > > > > >> > > > > > > If
> > > > > > >> > > > > > > > > we package and distribute this feature as a
> > > library
> > > > > > >> (either
> > > > > > >> > via
> > > > > > >> > > > an
> > > > > > >> > > > > > > > entirely
> > > > > > >> > > > > > > > > new artifact, or as part of the existing
> > > > > > >> connect-transforms
> > > > > > >> > or
> > > > > > >> > > > > > > > connect-api
> > > > > > >> > > > > > > > > artifacts), then we'd have to either sidestep
> > the
> > > > > > existing
> > > > > > >> > > plugin
> > > > > > >> > > > > > > > isolation
> > > > > > >> > > > > > > > > logic [1] that basically makes it impossible
> for
> > > > > Connect
> > > > > > >> > > plugins
> > > > > > >> > > > to
> > > > > > >> > > > > > > ship
> > > > > > >> > > > > > > > > their own versions of Connect artifacts, or
> > issue
> > > a
> > > > > big
> > > > > > >> > warning
> > > > > > >> > > > to
> > > > > > >> > > > > > > people
> > > > > > >> > > > > > > > > that any SMT that uses this API won't work
> with
> > > any
> > > > > > older
> > > > > > >> > > > versions
> > > > > > >> > > > > of
> > > > > > >> > > > > > > > > Connect. There's also some other features we
> > might
> > > > > want
> > > > > > to
> > > > > > >> > add
> > > > > > >> > > in
> > > > > > >> > > > > an
> > > > > > >> > > > > > > > > SMT-utils library such as the existing,
> > internal,
> > > > > utils
> > > > > > >> that
> > > > > > >> > > > > Connect
> > > > > > >> > > > > > > uses
> > > > > > >> > > > > > > > > right now [2]. It may be worth exploring this
> > in a
> > > > > > >> separate
> > > > > > >> > KIP
> > > > > > >> > > > of
> > > > > > >> > > > > > its
> > > > > > >> > > > > > > > own.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > [1] -
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > [2] -
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Cheers,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Chris
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom Bentley <
> > > > > > >> > > tbentley@redhat.com
> > > > > > >> > > > >
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks for the KIP, especially for the
> > examples
> > > > > which
> > > > > > >> are
> > > > > > >> > > > > > > super-clear.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 100. The name `field.style` isn't so clear
> for
> > > > > > something
> > > > > > >> > like
> > > > > > >> > > > > > > > > ReplaceField:
> > > > > > >> > > > > > > > > > it's not so obvious that field.style applies
> > to
> > > > > > >> `include`
> > > > > > >> > and
> > > > > > >> > > > > > > > `exclude`.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 101. The permitted values for `field.style`
> > > don't
> > > > > seem
> > > > > > >> > > terribly
> > > > > > >> > > > > > > > intuitive
> > > > > > >> > > > > > > > > > (to me anyway): the meaning of `plain` isn't
> > > very
> > > > > > >> > guessable.
> > > > > > >> > > > Why
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > > > > `top-level` or `root` instead? Also `nested`
> > > could
> > > > > be
> > > > > > >> > > > > misconstrued
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > mean
> > > > > > >> > > > > > > > > > nested-but-not-top-level, so perhaps
> > `recursive`
> > > > or
> > > > > > >> > > `cascading`
> > > > > > >> > > > > > might
> > > > > > >> > > > > > > > be
> > > > > > >> > > > > > > > > > better?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 102. I'm torn on whether making the
> > > interpretation
> > > > > of
> > > > > > >> > > existing
> > > > > > >> > > > > > > configs
> > > > > > >> > > > > > > > > like
> > > > > > >> > > > > > > > > > `field` be dependent on `field.style` is a
> > good
> > > > > idea.
> > > > > > I
> > > > > > >> can
> > > > > > >> > > see
> > > > > > >> > > > > > that
> > > > > > >> > > > > > > > it's
> > > > > > >> > > > > > > > > > the simplest thing to do, but it just feels
> a
> > > bit
> > > > > odd
> > > > > > >> that
> > > > > > >> > > > > > sometimes
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > `field` would actually be a path and have
> > > > different
> > > > > > >> > escaping
> > > > > > >> > > > > rules.
> > > > > > >> > > > > > > An
> > > > > > >> > > > > > > > > > alternative would be to come up with a
> > parallel
> > > > set
> > > > > of
> > > > > > >> > config
> > > > > > >> > > > > names
> > > > > > >> > > > > > > > (e.g.
> > > > > > >> > > > > > > > > > as well as "field" an SMT might support
> > "path")
> > > > > which
> > > > > > >> were
> > > > > > >> > > > > defined
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > > always take paths, thus avoiding the
> > changeable
> > > > > > >> > > interpretation
> > > > > > >> > > > of
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > > existing configs. The SMT's #configure()
> would
> > > > need
> > > > > to
> > > > > > >> > throw
> > > > > > >> > > in
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > case
> > > > > > >> > > > > > > > > > that both configs were given. I can see that
> > > that
> > > > > > would
> > > > > > >> be
> > > > > > >> > > more
> > > > > > >> > > > > > work
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 103. I think in order to allow for
> supporting
> > > > arrays
> > > > > > in
> > > > > > >> a
> > > > > > >> > > later
> > > > > > >> > > > > KIP
> > > > > > >> > > > > > > > > (which
> > > > > > >> > > > > > > > > > certainly seems like it could be useful),
> we'd
> > > > want
> > > > > to
> > > > > > >> > > specify
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > syntax
> > > > > > >> > > > > > > > > > now, even if it wasn't implemented under
> this
> > > KIP.
> > > > > > >> That's
> > > > > > >> > > > > because I
> > > > > > >> > > > > > > > don't
> > > > > > >> > > > > > > > > > think you can't exclude the possibility that
> > > > > > characters
> > > > > > >> > such
> > > > > > >> > > as
> > > > > > >> > > > > `[`
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > `]`
> > > > > > >> > > > > > > > > > appear in field names. So you'd have a
> > > > compatibility
> > > > > > >> > problem
> > > > > > >> > > if
> > > > > > >> > > > > > > people
> > > > > > >> > > > > > > > > > started using the features of this KIP to
> > access
> > > > > such
> > > > > > >> > fields,
> > > > > > >> > > > > only
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > > those characters to change their meaning
> > under a
> > > > > later
> > > > > > >> KIP.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 104. I also wonder whether making paths
> into a
> > > > > public
> > > > > > >> Java
> > > > > > >> > > API,
> > > > > > >> > > > > for
> > > > > > >> > > > > > > use
> > > > > > >> > > > > > > > > by
> > > > > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks again,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Tom
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris Egerton
> <
> > > > > > >> > > > > > fearthecellos@gmail.com
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge Esteban
> > > > Quilcate
> > > > > > >> Otoya
> > > > > > >> > <
> > > > > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Thank you, Chris! Not possible without
> > your
> > > > > > >> feedback.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04, Chris
> > Egerton
> > > <
> > > > > > >> > > > > > > > fearthecellos@gmail.com
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thank you for sticking through this. I
> > > have
> > > > > one
> > > > > > >> small
> > > > > > >> > > > > remark
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > one
> > > > > > >> > > > > > > > > > > > small
> > > > > > >> > > > > > > > > > > > > clarification; assuming you agree with
> > me
> > > on
> > > > > > them
> > > > > > >> > then
> > > > > > >> > > > I'm
> > > > > > >> > > > > > > ready
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > vote
> > > > > > >> > > > > > > > > > > > on
> > > > > > >> > > > > > > > > > > > > the KIP.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > 1. InsertField: The
> > > > "field.on.missing.parent"
> > > > > > and
> > > > > > >> > > > > > > > > > > > "field.on.existing.field"
> > > > > > >> > > > > > > > > > > > > docs both mention a permitted value of
> > > > > "ingore";
> > > > > > >> this
> > > > > > >> > > > > should
> > > > > > >> > > > > > be
> > > > > > >> > > > > > > > > > > "ignore",
> > > > > > >> > > > > > > > > > > > > right?
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Of course, one more typo :)
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > 2. InsertField: The examples are still
> > > > missing
> > > > > > the
> > > > > > >> > > > > > > "field.style"
> > > > > > >> > > > > > > > > > > property
> > > > > > >> > > > > > > > > > > > > from the configurations. They should
> all
> > > > > include
> > > > > > >> the
> > > > > > >> > > > > property
> > > > > > >> > > > > > > > > > > > > "transforms.smt1.field.style":
> "nested",
> > > > > > correct?
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Yes, it is there. I think I know what
> you
> > > mean
> > > > > > now,
> > > > > > >> > seems
> > > > > > >> > > > > that
> > > > > > >> > > > > > > > > > Confluence
> > > > > > >> > > > > > > > > > > > is putting everything in one line when
> > it's
> > > in
> > > > > > >> separate
> > > > > > >> > > > lines
> > > > > > >> > > > > > in
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > editor.
> > > > > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thanks again for working through this,
> > and
> > > > > > >> > > > congratulations
> > > > > > >> > > > > > on a
> > > > > > >> > > > > > > > > > > > > well-written KIP!
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Chris
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM Jorge
> > > > Esteban
> > > > > > >> > Quilcate
> > > > > > >> > > > > Otoya
> > > > > > >> > > > > > <
> > > > > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Thank you, Chris! I apply these
> > > > improvements
> > > > > > to
> > > > > > >> the
> > > > > > >> > > > KIP,
> > > > > > >> > > > > > let
> > > > > > >> > > > > > > me
> > > > > > >> > > > > > > > > > know
> > > > > > >> > > > > > > > > > > > how
> > > > > > >> > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > looks now.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43, Chris
> > > > Egerton
> > > > > <
> > > > > > >> > > > > > > > > > fearthecellos@gmail.com
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > Wow, those examples are great! A
> few
> > > > more
> > > > > > >> > remarks,
> > > > > > >> > > > but
> > > > > > >> > > > > I
> > > > > > >> > > > > > > > think
> > > > > > >> > > > > > > > > > > we're
> > > > > > >> > > > > > > > > > > > > > > getting close:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > 1. The examples differ across SMTs
> > > with
> > > > > the
> > > > > > >> name
> > > > > > >> > of
> > > > > > >> > > > the
> > > > > > >> > > > > > > > > > > > > newly-introduced
> > > > > > >> > > > > > > > > > > > > > > style property; some of them use
> > > > > > >> "field.style",
> > > > > > >> > and
> > > > > > >> > > > > some
> > > > > > >> > > > > > > use
> > > > > > >> > > > > > > > > > > > > > > "fields.style". I think for
> > > > consistency's
> > > > > > >> sake we
> > > > > > >> > > > > should
> > > > > > >> > > > > > > > stick
> > > > > > >> > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > > just
> > > > > > >> > > > > > > > > > > > > > > "field.style"; otherwise it could
> be
> > > > > painful
> > > > > > >> for
> > > > > > >> > > > users
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > have
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > remember
> > > > > > >> > > > > > > > > > > > > > > which to use.
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed the
> config
> > > > names
> > > > > > to
> > > > > > >> > > > > > > `field.style`.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > 2. Some of the examples are off:
> > > > > > >> > > > > > > > > > > > > > > - TimestampConverter: the input in
> > the
> > > > > > second
> > > > > > >> > > example
> > > > > > >> > > > > > > ("when
> > > > > > >> > > > > > > > > > field
> > > > > > >> > > > > > > > > > > > > names
> > > > > > >> > > > > > > > > > > > > > > include dots") doesn't contain a
> > field
> > > > > with
> > > > > > a
> > > > > > >> > > dotted
> > > > > > >> > > > > name
> > > > > > >> > > > > > > > > > > > > > > - ValueToKey: the config in the
> > third
> > > > > > example
> > > > > > >> > > ("when
> > > > > > >> > > > > > field
> > > > > > >> > > > > > > > > names
> > > > > > >> > > > > > > > > > > > > include
> > > > > > >> > > > > > > > > > > > > > > dots") should probably use
> > > > > > "parent..child.k2"
> > > > > > >> as
> > > > > > >> > > the
> > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.fields" property
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > 3. RE changes to InsertField:
> > > > > > >> > > > > > > > > > > > > > > - The InsertField SMT should also
> > come
> > > > > with
> > > > > > >> the
> > > > > > >> > new
> > > > > > >> > > > > > > > > "field.style"
> > > > > > >> > > > > > > > > > > > > > property
> > > > > > >> > > > > > > > > > > > > > > in order to preserve backwards
> > > > > > compatibility,
> > > > > > >> > > right?
> > > > > > >> > > > I
> > > > > > >> > > > > > > don't
> > > > > > >> > > > > > > > > see
> > > > > > >> > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > included in the example configs
> for
> > > that
> > > > > > one,
> > > > > > >> > just
> > > > > > >> > > > want
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > make
> > > > > > >> > > > > > > > > > > sure
> > > > > > >> > > > > > > > > > > > > > > - I don't know of any cases where
> we
> > > use
> > > > > > >> > snake_case
> > > > > > >> > > > for
> > > > > > >> > > > > > > > > property
> > > > > > >> > > > > > > > > > > > names
> > > > > > >> > > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > > Kafka; we should probably use
> > > > > > >> "on.missing.parent"
> > > > > > >> > > and
> > > > > > >> > > > > > > > > > > > > "on.existing.field"
> > > > > > >> > > > > > > > > > > > > > > as the new property names for
> > > > InsertField.
> > > > > > >> > > > > > > > > > > > > > > - Why is the "on_existing_field"
> (or
> > > > > > >> > > > > "on.existing.field")
> > > > > > >> > > > > > > > > > property
> > > > > > >> > > > > > > > > > > > only
> > > > > > >> > > > > > > > > > > > > > > applied when the field style is
> > > nested?
> > > > > > >> Couldn't
> > > > > > >> > it
> > > > > > >> > > > be
> > > > > > >> > > > > > > useful
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Great points! I have applied these
> > > > > suggestions
> > > > > > >> to
> > > > > > >> > the
> > > > > > >> > > > > KIP.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > Chris
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40 PM
> > Jorge
> > > > > > Esteban
> > > > > > >> > > > Quilcate
> > > > > > >> > > > > > > Otoya
> > > > > > >> > > > > > > > <
> > > > > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Again, great feedback Chris.
> Much
> > > > > > >> appreciated.
> > > > > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at 20:22,
> Chris
> > > > > > Egerton <
> > > > > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > Looking good! I have a few
> > > comments
> > > > > left
> > > > > > >> but
> > > > > > >> > > all
> > > > > > >> > > > > but
> > > > > > >> > > > > > > one
> > > > > > >> > > > > > > > or
> > > > > > >> > > > > > > > > > two
> > > > > > >> > > > > > > > > > > > are
> > > > > > >> > > > > > > > > > > > > > > > minor.
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > 1. The motivation section
> states
> > > > "This
> > > > > > >> KIP is
> > > > > > >> > > > aimed
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > > include
> > > > > > >> > > > > > > > > > > > > > support
> > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > nested structures on the
> > existing
> > > > > > SMTs...
> > > > > > >> and
> > > > > > >> > > to
> > > > > > >> > > > > > > include
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > abstractions
> > > > > > >> > > > > > > > > > > > > > > > > to reuse this in future
> SMTs". A
> > > > good
> > > > > > >> > > > > implementation
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > > this
> > > > > > >> > > > > > > > > > > KIP
> > > > > > >> > > > > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > > > > > > definitely isolate reusable
> > logic
> > > > > into a
> > > > > > >> > > separate
> > > > > > >> > > > > > > > > abstraction
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > can
> > > > > > >> > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > > easily pulled in to the SMTs
> we
> > > want
> > > > > to
> > > > > > >> add
> > > > > > >> > > > nested
> > > > > > >> > > > > > > field
> > > > > > >> > > > > > > > > > > support
> > > > > > >> > > > > > > > > > > > > to,
> > > > > > >> > > > > > > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > > > > unless we plan on making this
> > kind
> > > > of
> > > > > > >> > > abstraction
> > > > > > >> > > > > > > > publicly
> > > > > > >> > > > > > > > > > > > > available
> > > > > > >> > > > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > > > > > > some kind of utility method or
> > > class
> > > > > > that
> > > > > > >> > > > external
> > > > > > >> > > > > > SMT
> > > > > > >> > > > > > > > > > > developers
> > > > > > >> > > > > > > > > > > > > can
> > > > > > >> > > > > > > > > > > > > > > > > leverage, we can probably
> leave
> > > this
> > > > > > part
> > > > > > >> out
> > > > > > >> > > as
> > > > > > >> > > > > it's
> > > > > > >> > > > > > > > more
> > > > > > >> > > > > > > > > of
> > > > > > >> > > > > > > > > > > an
> > > > > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Make sense, will leave this out
> of
> > > the
> > > > > > KIP.
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a
> little
> > > > > > >> misleading,
> > > > > > >> > > isn't
> > > > > > >> > > > > it?
> > > > > > >> > > > > > > It
> > > > > > >> > > > > > > > > > > > > demonstrates
> > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > escape syntax for fields with
> > dot
> > > > > > >> literals in
> > > > > > >> > > > their
> > > > > > >> > > > > > > > names,
> > > > > > >> > > > > > > > > > but
> > > > > > >> > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > doesn't
> > > > > > >> > > > > > > > > > > > > > > > > demonstrate a way to actually
> > use
> > > > the
> > > > > > Cast
> > > > > > >> > (or
> > > > > > >> > > > any
> > > > > > >> > > > > > > other)
> > > > > > >> > > > > > > > > SMT
> > > > > > >> > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > access a
> > > > > > >> > > > > > > > > > > > > > > > > nested field in a record,
> which
> > is
> > > > the
> > > > > > >> whole
> > > > > > >> > > > point
> > > > > > >> > > > > of
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > KIP.
> > > > > > >> > > > > > > > > > > I
> > > > > > >> > > > > > > > > > > > > like
> > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > example of escape syntax but
> we
> > > > should
> > > > > > >> > probably
> > > > > > >> > > > > also
> > > > > > >> > > > > > > add
> > > > > > >> > > > > > > > > one
> > > > > > >> > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > > > field access.
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Agree. I have added examples to
> > each
> > > > SMT
> > > > > > to
> > > > > > >> be
> > > > > > >> > > more
> > > > > > >> > > > > > clear
> > > > > > >> > > > > > > > > about
> > > > > > >> > > > > > > > > > > how
> > > > > > >> > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > > affects each function
> > > > > > >> > > > > > > > > > > > > > > > .
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > 3. With the InsertField SMT,
> I'm
> > > > > > wondering
> > > > > > >> > what
> > > > > > >> > > > the
> > > > > > >> > > > > > > > > specific
> > > > > > >> > > > > > > > > > > > > behavior
> > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > > > > > > be when some or all of the
> > "middle
> > > > > > layer"
> > > > > > >> > > nested
> > > > > > >> > > > > > fields
> > > > > > >> > > > > > > > are
> > > > > > >> > > > > > > > > > > > > missing.
> > > > > > >> > > > > > > > > > > > > > > For
> > > > > > >> > > > > > > > > > > > > > > > > example, if I have a record
> > with a
> > > > > value
> > > > > > >> of {
> > > > > > >> > > > "k1":
> > > > > > >> > > > > > > "v1 }
> > > > > > >> > > > > > > > > > and I
> > > > > > >> > > > > > > > > > > > > apply
> > > > > > >> > > > > > > > > > > > > > > > > InsertField with topic.field =
> > > > > > >> > n1.n2.n3.topic,
> > > > > > >> > > > what
> > > > > > >> > > > > > > will
> > > > > > >> > > > > > > > > > > happen?
> > > > > > >> > > > > > > > > > > > > Will
> > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > updated value become { "k1":
> > "v1",
> > > > > > "n1": {
> > > > > > >> > > "n2":
> > > > > > >> > > > {
> > > > > > >> > > > > > > "n3":
> > > > > > >> > > > > > > > > > > "topic"
> > > > > > >> > > > > > > > > > > > > }}},
> > > > > > >> > > > > > > > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > > > > > > an exception be thrown, or
> > > something
> > > > > > else?
> > > > > > >> > This
> > > > > > >> > > > > seems
> > > > > > >> > > > > > > > > > analogous
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > command line mkdir command,
> > which
> > > > (at
> > > > > > >> least
> > > > > > >> > on
> > > > > > >> > > > some
> > > > > > >> > > > > > > > > operating
> > > > > > >> > > > > > > > > > > > > > systems)
> > > > > > >> > > > > > > > > > > > > > > > > fails by default if you try to
> > > > create
> > > > > a
> > > > > > >> new
> > > > > > >> > > > nested
> > > > > > >> > > > > > > > > directory
> > > > > > >> > > > > > > > > > > > where
> > > > > > >> > > > > > > > > > > > > > > > anything
> > > > > > >> > > > > > > > > > > > > > > > > but the last element in the
> path
> > > > > doesn't
> > > > > > >> > exist,
> > > > > > >> > > > but
> > > > > > >> > > > > > can
> > > > > > >> > > > > > > > be
> > > > > > >> > > > > > > > > > > > invoked
> > > > > > >> > > > > > > > > > > > > > > with a
> > > > > > >> > > > > > > > > > > > > > > > > flag that instructs it to go
> > ahead
> > > > and
> > > > > > >> create
> > > > > > >> > > all
> > > > > > >> > > > > > > levels
> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > > > directory instead. I'm leaning
> > on
> > > > the
> > > > > > >> side of
> > > > > > >> > > > "just
> > > > > > >> > > > > > > > create
> > > > > > >> > > > > > > > > > > > > > everything"
> > > > > > >> > > > > > > > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > > > > would be interested in your
> > > > thoughts,
> > > > > > and
> > > > > > >> > > either
> > > > > > >> > > > > way,
> > > > > > >> > > > > > > we
> > > > > > >> > > > > > > > > > should
> > > > > > >> > > > > > > > > > > > > > > probably
> > > > > > >> > > > > > > > > > > > > > > > > make sure the intended
> behavior
> > is
> > > > > > >> > well-defined
> > > > > > >> > > > > > before
> > > > > > >> > > > > > > > > > voting.
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > This is an interesting case,
> > thanks
> > > > for
> > > > > > >> > catching
> > > > > > >> > > > > this!
> > > > > > >> > > > > > > > > > > > > > > > The default behavior I see is to
> > > > create
> > > > > > >> parents
> > > > > > >> > > if
> > > > > > >> > > > > they
> > > > > > >> > > > > > > are
> > > > > > >> > > > > > > > > > > missing
> > > > > > >> > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > overwrite fields
> > > > > > >> > > > > > > > > > > > > > > > if they already exist.
> > > > > > >> > > > > > > > > > > > > > > > I'm planning to include the
> > > following
> > > > > two
> > > > > > >> flags
> > > > > > >> > > if
> > > > > > >> > > > > > there
> > > > > > >> > > > > > > > is a
> > > > > > >> > > > > > > > > > > need
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > > > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> > > > (CREATE|IGNORE),
> > > > > > >> > > > > default=CREATE
> > > > > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > > > > (OVERWRITE|IGNORE),
> > > > > > >> > > > > > > > default=OVERWRITE
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will the
> > > behavior
> > > > > be
> > > > > > if
> > > > > > >> > any
> > > > > > >> > > of
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > field
> > > > > > >> > > > > > > > > > > > > elements
> > > > > > >> > > > > > > > > > > > > > > > > specified with InsertField
> > already
> > > > > exist
> > > > > > >> in
> > > > > > >> > the
> > > > > > >> > > > > > record
> > > > > > >> > > > > > > > > value?
> > > > > > >> > > > > > > > > > > > Will
> > > > > > >> > > > > > > > > > > > > we
> > > > > > >> > > > > > > > > > > > > > > > just
> > > > > > >> > > > > > > > > > > > > > > > > overwrite them? What's the
> > > behavior
> > > > of
> > > > > > >> > > > InsertField
> > > > > > >> > > > > > > today
> > > > > > >> > > > > > > > > > under
> > > > > > >> > > > > > > > > > > > > > similar
> > > > > > >> > > > > > > > > > > > > > > > > circumstances?
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > The current behavior is to
> > overwrite
> > > > the
> > > > > > >> value.
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > Chris
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at 4:15
> PM
> > > > Jorge
> > > > > > >> Esteban
> > > > > > >> > > > > > Quilcate
> > > > > > >> > > > > > > > > Otoya
> > > > > > >> > > > > > > > > > <
> > > > > > >> > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > wrote:
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much
> > appreciated
> > > > all
> > > > > > the
> > > > > > >> > > > feedback
> > > > > > >> > > > > > > here.
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > 1. You nailed it setting the
> > > > design
> > > > > > goal
> > > > > > >> > > here:
> > > > > > >> > > > > "it
> > > > > > >> > > > > > > > > > shouldn't
> > > > > > >> > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > > impossible
> > > > > > >> > > > > > > > > > > > > > > > > > to use this new feature for
> > any
> > > > > field
> > > > > > >> name,
> > > > > > >> > > no
> > > > > > >> > > > > > matter
> > > > > > >> > > > > > > > how
> > > > > > >> > > > > > > > > > > > > > convoluted.
> > > > > > >> > > > > > > > > > > > > > > > > It's
> > > > > > >> > > > > > > > > > > > > > > > > > fine if edge cases introduce
> > > > > > difficulty
> > > > > > >> > (such
> > > > > > >> > > > as
> > > > > > >> > > > > > > > > > > less-readable
> > > > > > >> > > > > > > > > > > > > > > > > > configurations), but it's
> not
> > > fine
> > > > > if
> > > > > > >> they
> > > > > > >> > > > can't
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > > addressed
> > > > > > >> > > > > > > > > > > > at
> > > > > > >> > > > > > > > > > > > > > > all."
> > > > > > >> > > > > > > > > > > > > > > > > > Back to the previous
> proposals
> > > > > (using
> > > > > > >> only
> > > > > > >> > > dots
> > > > > > >> > > > > as
> > > > > > >> > > > > > > > > > > separators)
> > > > > > >> > > > > > > > > > > > we
> > > > > > >> > > > > > > > > > > > > > > have
> > > > > > >> > > > > > > > > > > > > > > > 2
> > > > > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > > > > >> > > > > > > > > > > > > > > > > > 1. escaping with backslashes
> > > > > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots itself
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > I'll lean for alternative 2,
> > as
> > > > you
> > > > > > >> > proposed
> > > > > > >> > > > > > before.
> > > > > > >> > > > > > > > > Feels
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > me
> > > > > > >> > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > backslashes have more
> > potential
> > > to
> > > > > > lead
> > > > > > >> to
> > > > > > >> > > > > > confusion
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > > JSON
> > > > > > >> > > > > > > > > > > > > > configs,
> > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > CSV seems like a good
> > precedent
> > > to
> > > > > use
> > > > > > >> the
> > > > > > >> > > same
> > > > > > >> > > > > > > > character
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > escape
> > > > > > >> > > > > > > > > > > > > > > > > itself.
> > > > > > >> > > > > > > > > > > > > > > > > > KIP is updated to reflect
> > this.
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an
> > example,
> > > > and
> > > > > > >> stick
> > > > > > >> > > with
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > current
> > > > > > >> > > > > > > > > > > > > approach
> > > > > > >> > > > > > > > > > > > > > > > > > defining the style per
> > > individual
> > > > > > >> transform
> > > > > > >> > > > > > > > > configuration.
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP updated.
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP updated.
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at
> 21:59,
> > > > Chris
> > > > > > >> > Egerton <
> > > > > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for addressing my
> > > > comments;
> > > > > > the
> > > > > > >> > KIP
> > > > > > >> > > > > looks
> > > > > > >> > > > > > > > > > up-to-date
> > > > > > >> > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > pretty
> > > > > > >> > > > > > > > > > > > > > > > > > > readable now, and the
> > rejected
> > > > > > >> > alternatives
> > > > > > >> > > > > > section
> > > > > > >> > > > > > > > > does
> > > > > > >> > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > great
> > > > > > >> > > > > > > > > > > > > > > job
> > > > > > >> > > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > outlining the discussion
> so
> > > far
> > > > > and
> > > > > > >> > > providing
> > > > > > >> > > > > > > context
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > anyone
> > > > > > >> > > > > > > > > > > > > > > else
> > > > > > >> > > > > > > > > > > > > > > > > who
> > > > > > >> > > > > > > > > > > > > > > > > > > might want to join in.
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on choice of
> > > > > delimiter:
> > > > > > >> > > > > > > > > > > > > > > > > > > - I like the optimization
> > for
> > > > > simple
> > > > > > >> > cases,
> > > > > > >> > > > > but I
> > > > > > >> > > > > > > > think
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > new
> > > > > > >> > > > > > > > > > > > > > > > > proposal
> > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > > > > a little too restrictive.
> > What
> > > > if
> > > > > > >> > there's a
> > > > > > >> > > > > field
> > > > > > >> > > > > > > > whose
> > > > > > >> > > > > > > > > > > name
> > > > > > >> > > > > > > > > > > > > > > contains
> > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > >> > > > > > > > > > > > > > > > > > > of the permitted options
> > > > > (currently
> > > > > > >> just
> > > > > > >> > > ".",
> > > > > > >> > > > > > ",",
> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > > > > > > "/")?
> > > > > > >> > > > > > > > > > > > > > > > > > > - If we expand the set of
> > > > > permitted
> > > > > > >> > > > delimiters
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > allow
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > any
> > > > > > >> > > > > > > > > > > > > > > > > > > single-character string,
> > > > > > configuration
> > > > > > >> > > > > complexity
> > > > > > >> > > > > > > > will
> > > > > > >> > > > > > > > > > > > increase
> > > > > > >> > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > readability may decrease
> > > > > > >> > > > > > > > > > > > > > > > > > > - Also worth pointing out
> > that
> > > > > there
> > > > > > >> is
> > > > > > >> > > some
> > > > > > >> > > > > > > > convention
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > doubling
> > > > > > >> > > > > > > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > > > > > delimiter character as an
> > > escape
> > > > > > >> > mechanism
> > > > > > >> > > > with
> > > > > > >> > > > > > > > formats
> > > > > > >> > > > > > > > > > > like
> > > > > > >> > > > > > > > > > > > > CSV
> > > > > > >> > > > > > > > > > > > > > > [1]
> > > > > > >> > > > > > > > > > > > > > > > > > > - Overall I think we may
> be
> > > > > > >> approaching
> > > > > > >> > the
> > > > > > >> > > > > > > > saturation
> > > > > > >> > > > > > > > > > > point
> > > > > > >> > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > productive
> > > > > > >> > > > > > > > > > > > > > > > > > > discussion on delimiter
> > syntax
> > > > so
> > > > > I
> > > > > > >> don't
> > > > > > >> > > > want
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > spend
> > > > > > >> > > > > > > > > > too
> > > > > > >> > > > > > > > > > > > > much
> > > > > > >> > > > > > > > > > > > > > > more
> > > > > > >> > > > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > your time on it. I think
> the
> > > one
> > > > > > point
> > > > > > >> > I'd
> > > > > > >> > > > like
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > leave
> > > > > > >> > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > now
> > > > > > >> > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > > > > > shouldn't be impossible to
> > use
> > > > > this
> > > > > > >> new
> > > > > > >> > > > feature
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > any
> > > > > > >> > > > > > > > > > > field
> > > > > > >> > > > > > > > > > > > > > name,
> > > > > > >> > > > > > > > > > > > > > > > no
> > > > > > >> > > > > > > > > > > > > > > > > > > matter how convoluted.
> It's
> > > fine
> > > > > if
> > > > > > >> edge
> > > > > > >> > > > cases
> > > > > > >> > > > > > > > > introduce
> > > > > > >> > > > > > > > > > > > > > difficulty
> > > > > > >> > > > > > > > > > > > > > > > > (such
> > > > > > >> > > > > > > > > > > > > > > > > > > as less-readable
> > > > configurations),
> > > > > > but
> > > > > > >> > it's
> > > > > > >> > > > not
> > > > > > >> > > > > > fine
> > > > > > >> > > > > > > > if
> > > > > > >> > > > > > > > > > they
> > > > > > >> > > > > > > > > > > > > can't
> > > > > > >> > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > 2.
> > > > > > >> > > > > > > > > > > > > > > > > > > The configuration style
> > where
> > > > you
> > > > > > >> define
> > > > > > >> > > > > > > > > > > > > "transforms.field.style"
> > > > > > >> > > > > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > connector config, and then
> > > this
> > > > > > >> applies
> > > > > > >> > to
> > > > > > >> > > > all
> > > > > > >> > > > > > SMTs
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > connector,
> > > > > > >> > > > > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > > > > very interesting. However,
> > it
> > > > > > doesn't
> > > > > > >> > > follow
> > > > > > >> > > > > > > > convention
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > existing
> > > > > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > > > > >> > > > > > > > > > > > > > > > > > > Right now, if you want to
> > > > > configure
> > > > > > an
> > > > > > >> > SMT,
> > > > > > >> > > > you
> > > > > > >> > > > > > > > define
> > > > > > >> > > > > > > > > > its
> > > > > > >> > > > > > > > > > > > name
> > > > > > >> > > > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > connector config (for
> > example,
> > > > > > >> > > "transforms":
> > > > > > >> > > > > > > "smt1"),
> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > > > > > > > then
> > > > > > >> > > > > > > > > > > > > > > define
> > > > > > >> > > > > > > > > > > > > > > > > all
> > > > > > >> > > > > > > > > > > > > > > > > > > of the properties for that
> > SMT
> > > > in
> > > > > > the
> > > > > > >> > > > connector
> > > > > > >> > > > > > > > config
> > > > > > >> > > > > > > > > > > using
> > > > > > >> > > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > > > > namespacing
> > > > > > >> > > > > > > > > > > > > > > > > > > mechanism specific to that
> > SMT
> > > > > (for
> > > > > > >> > > example,
> > > > > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > > > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT then
> sees
> > > only
> > > > > the
> > > > > > >> > > > properties
> > > > > > >> > > > > > > > defined
> > > > > > >> > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > namespace,
> > > > > > >> > > > > > > > > > > > > > > > > > > with the prefix stripped
> > (for
> > > > > > example,
> > > > > > >> > > > "prop1":
> > > > > > >> > > > > > > > "val1")
> > > > > > >> > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > its
> > > > > > >> > > > > > > > > > > > > > > > > configure
> > > > > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > > > > >> > > > > > > > > > > > > > > > > > > If we want to continue to
> > > follow
> > > > > > this
> > > > > > >> > > > > convention,
> > > > > > >> > > > > > > > then
> > > > > > >> > > > > > > > > > > > instead
> > > > > > >> > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > specifying
> > > > > "transforms.field.style"
> > > > > > >> in a
> > > > > > >> > > > > > connector
> > > > > > >> > > > > > > > > > config,
> > > > > > >> > > > > > > > > > > we
> > > > > > >> > > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > > > > > expect
> > > > > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > > > > >> > > > > > "transforms.<name>.field.style",
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > > each
> > > > > > >> > > > > > > > > > > > > SMT
> > > > > > >> > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > they
> > > > > > >> > > > > > > > > > > > > > > > > > > want to configure a field
> > > style
> > > > > for.
> > > > > > >> This
> > > > > > >> > > > would
> > > > > > >> > > > > > > > require
> > > > > > >> > > > > > > > > > > more
> > > > > > >> > > > > > > > > > > > > work
> > > > > > >> > > > > > > > > > > > > > > on
> > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > part of the user, but
> would
> > be
> > > > > > >> simpler to
> > > > > > >> > > > > reason
> > > > > > >> > > > > > > > about
> > > > > > >> > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > easier
> > > > > > >> > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > implement.
> > > > > > >> > > > > > > > > > > > > > > > > > > If we want to explore an
> > > > > alternative
> > > > > > >> > where
> > > > > > >> > > > > users
> > > > > > >> > > > > > > can
> > > > > > >> > > > > > > > > > > specify
> > > > > > >> > > > > > > > > > > > > > global
> > > > > > >> > > > > > > > > > > > > > > > > > > properties that apply to
> all
> > > > > > >> transforms
> > > > > > >> > in
> > > > > > >> > > a
> > > > > > >> > > > > > > > connector
> > > > > > >> > > > > > > > > > > > config,
> > > > > > >> > > > > > > > > > > > > > then
> > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > semantics for this need to
> > be
> > > > > > defined
> > > > > > >> in
> > > > > > >> > > the
> > > > > > >> > > > > KIP.
> > > > > > >> > > > > > > > This
> > > > > > >> > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > have
> > > > > > >> > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > include whether this will
> > > apply
> > > > > only
> > > > > > >> for
> > > > > > >> > > the
> > > > > > >> > > > > > > special
> > > > > > >> > > > > > > > > case
> > > > > > >> > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > "field.style" and possibly
> > > > > > >> > > "field.separator"
> > > > > > >> > > > > > > > properties
> > > > > > >> > > > > > > > > > or
> > > > > > >> > > > > > > > > > > if
> > > > > > >> > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > > > > available more generally
> for
> > > > other
> > > > > > >> > > > properties,
> > > > > > >> > > > > > > > whether
> > > > > > >> > > > > > > > > it
> > > > > > >> > > > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > > > > apply
> > > > > > >> > > > > > > > > > > > > > > > > only
> > > > > > >> > > > > > > > > > > > > > > > > > > for the SMTs outlined in
> the
> > > KIP
> > > > > or
> > > > > > if
> > > > > > >> > the
> > > > > > >> > > > > > > > > "field.style"
> > > > > > >> > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > possibly
> > > > > > >> > > > > > > > > > > > > > > > > > > "field.separator"
> properties
> > > > would
> > > > > > >> also
> > > > > > >> > be
> > > > > > >> > > > > passed
> > > > > > >> > > > > > > > into
> > > > > > >> > > > > > > > > > > custom
> > > > > > >> > > > > > > > > > > > > > SMTs
> > > > > > >> > > > > > > > > > > > > > > so
> > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > they could choose to act
> on
> > > them
> > > > > if
> > > > > > >> > > > applicable,
> > > > > > >> > > > > > how
> > > > > > >> > > > > > > > > edge
> > > > > > >> > > > > > > > > > > > cases
> > > > > > >> > > > > > > > > > > > > > like
> > > > > > >> > > > > > > > > > > > > > > > > > having
> > > > > > >> > > > > > > > > > > > > > > > > > > an SMT named "field" in
> your
> > > > > > connector
> > > > > > >> > > config
> > > > > > >> > > > > > would
> > > > > > >> > > > > > > > be
> > > > > > >> > > > > > > > > > > > handled,
> > > > > > >> > > > > > > > > > > > > > > etc.
> > > > > > >> > > > > > > > > > > > > > > > > > > Either way, it might help
> to
> > > > have
> > > > > an
> > > > > > >> > > example
> > > > > > >> > > > in
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > KIP
> > > > > > >> > > > > > > > > > > > > outlining
> > > > > > >> > > > > > > > > > > > > > > how
> > > > > > >> > > > > > > > > > > > > > > > > one
> > > > > > >> > > > > > > > > > > > > > > > > > > of the to-be-augmented
> SMTs
> > > can
> > > > be
> > > > > > >> > > configured
> > > > > > >> > > > > > with
> > > > > > >> > > > > > > > this
> > > > > > >> > > > > > > > > > new
> > > > > > >> > > > > > > > > > > > > > feature
> > > > > > >> > > > > > > > > > > > > > > > > and a
> > > > > > >> > > > > > > > > > > > > > > > > > > before/after of how a
> record
> > > > value
> > > > > > >> would
> > > > > > >> > be
> > > > > > >> > > > > > > > transformed
> > > > > > >> > > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for the
> > > > > > >> > > > > "transforms.field.style"
> > > > > > >> > > > > > > > > > property
> > > > > > >> > > > > > > > > > > > > > mentions
> > > > > > >> > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > the permitted values are
> > > "plain"
> > > > > and
> > > > > > >> > > > "nested",
> > > > > > >> > > > > > but
> > > > > > >> > > > > > > > then
> > > > > > >> > > > > > > > > > > > > describes
> > > > > > >> > > > > > > > > > > > > > > > > > behavior
> > > > > > >> > > > > > > > > > > > > > > > > > > with a value of "root".
> > Should
> > > > > that
> > > > > > be
> > > > > > >> > > > "plain"
> > > > > > >> > > > > > > > instead?
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for the
> > > > > > >> > > > > > > "transforms.field.separator"
> > > > > > >> > > > > > > > > > > > property
> > > > > > >> > > > > > > > > > > > > > > > > > exclusively
> > > > > > >> > > > > > > > > > > > > > > > > > > mentions structs, but the
> > > > feature
> > > > > is
> > > > > > >> > > intended
> > > > > > >> > > > > to
> > > > > > >> > > > > > > work
> > > > > > >> > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > maps
> > > > > > >> > > > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > > > > > > well.
> > > > > > >> > > > > > > > > > > > > > > > > > > Can we update it to
> reflect
> > > > this?
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > References:
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > > > > >> > https://stackoverflow.com/a/17808731
> > > > > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > > > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > Chris
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022 at
> 1:32
> > > PM
> > > > > > Jorge
> > > > > > >> > > Esteban
> > > > > > >> > > > > > > > Quilcate
> > > > > > >> > > > > > > > > > > Otoya
> > > > > > >> > > > > > > > > > > > <
> > > > > > >> > > > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > > wrote:
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> > > > "this..field.child"
> > > > > > >> could
> > > > > > >> > be
> > > > > > >> > > > > > harder
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > grasp
> > > > > > >> > > > > > > > > > > > > than
> > > > > > >> > > > > > > > > > > > > > > > > > > > "this.field/child" +
> > > > separator:
> > > > > > "/".
> > > > > > >> > > > > > > > > > > > > > > > > > > > Even though this
> > represents
> > > > > > >> additional
> > > > > > >> > > > > > > information,
> > > > > > >> > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > follows
> > > > > > >> > > > > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > > > > similar
> > > > > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > > > > "Flatten#delimeter"
> > > > > > >> > > > > > > configuration.
> > > > > > >> > > > > > > > > > > > > > > > > > > > I want to give the
> > separator
> > > > > > >> approach
> > > > > > >> > > > another
> > > > > > >> > > > > > > try,
> > > > > > >> > > > > > > > > so I
> > > > > > >> > > > > > > > > > > > have
> > > > > > >> > > > > > > > > > > > > > > > updated
> > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > KIP with the separator
> > > > proposal,
> > > > > > >> > sticking
> > > > > > >> > > > to
> > > > > > >> > > > > > > only 2
> > > > > > >> > > > > > > > > > > > > > alternatives
> > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > > should hopefully cover
> > most
> > > > > > >> scenarios.
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has been
> > > updated
> > > > > > with
> > > > > > >> > this
> > > > > > >> > > > > > > > improvement.
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I have
> > > > updated
> > > > > > this
> > > > > > >> > > > section
> > > > > > >> > > > > > > > > > accordingly.
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch! I've
> > replaced
> > > > it
> > > > > > with
> > > > > > >> > > > > > > `DropHeaders`.
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > Looking forward to your
> > > > > feedback.
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022 at
> > 21:33,
> > > > > Chris
> > > > > > >> > > Egerton
> > > > > > >> > > > <
> > > > > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > Looking good! Got a
> few
> > > more
> > > > > > >> > thoughts.
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to revisit
> > this,
> > > > but
> > > > > I
> > > > > > >> think
> > > > > > >> > > we
> > > > > > >> > > > > may
> > > > > > >> > > > > > > want
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > adopt
> > > > > > >> > > > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > > > slightly
> > > > > > >> > > > > > > > > > > > > > > > > > > > > different escape
> syntax
> > > > style.
> > > > > > >> > > > Backslashes
> > > > > > >> > > > > > are
> > > > > > >> > > > > > > > > great,
> > > > > > >> > > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > since
> > > > > > >> > > > > > > > > > > > > > > > > > they're
> > > > > > >> > > > > > > > > > > > > > > > > > > > > already used by JSON,
> > > using
> > > > > them
> > > > > > >> as
> > > > > > >> > an
> > > > > > >> > > > > escape
> > > > > > >> > > > > > > > > > sequence
> > > > > > >> > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > field
> > > > > > >> > > > > > > > > > > > > > > > > > > notation
> > > > > > >> > > > > > > > > > > > > > > > > > > > > would also lead to
> some
> > > > pretty
> > > > > > >> ugly
> > > > > > >> > > > > connector
> > > > > > >> > > > > > > > > > configs.
> > > > > > >> > > > > > > > > > > > > Anyone
> > > > > > >> > > > > > > > > > > > > > > > who's
> > > > > > >> > > > > > > > > > > > > > > > > > had
> > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > write regular
> > expressions
> > > > with
> > > > > > >> > > > backslashes
> > > > > > >> > > > > in
> > > > > > >> > > > > > > > Java
> > > > > > >> > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > probably
> > > > > > >> > > > > > > > > > > > > > > > > > already
> > > > > > >> > > > > > > > > > > > > > > > > > > > > familiar with this:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > > > > >> > > > > > > > > > > > > > > > > What
> > > > > > >> > > > > > > > > > > > > > > > > > > do
> > > > > > >> > > > > > > > > > > > > > > > > > > > > you think about using
> > the
> > > > dot
> > > > > > >> > character
> > > > > > >> > > > to
> > > > > > >> > > > > > > escape
> > > > > > >> > > > > > > > > > > itself?
> > > > > > >> > > > > > > > > > > > > In
> > > > > > >> > > > > > > > > > > > > > > > other
> > > > > > >> > > > > > > > > > > > > > > > > > > words,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > to access a single
> field
> > > > named
> > > > > > >> > > > > "this.field",
> > > > > > >> > > > > > > > > instead
> > > > > > >> > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > using
> > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > syntax
> > > > > > >> > > > > > > > > > > > > > > > > > > > > "this\.field" (which
> in
> > > JSON
> > > > > > would
> > > > > > >> > have
> > > > > > >> > > > to
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > > expressed
> > > > > > >> > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > > "this..field",
> > > > > and
> > > > > > >> for a
> > > > > > >> > > > > single
> > > > > > >> > > > > > > > field
> > > > > > >> > > > > > > > > > > named
> > > > > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > > > > >> > > > > > > > > > > > > > > > > > > > > instead of using the
> > > syntax
> > > > > > >> > > "this\\field"
> > > > > > >> > > > > > (or,
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > > JSON,
> > > > > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > "this\field"
> > > > (or,
> > > > > > in
> > > > > > >> > JSON,
> > > > > > >> > > > > > > > > > "this\\field").
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you flesh out
> > the
> > > > > > >> details on
> > > > > > >> > > the
> > > > > > >> > > > > new
> > > > > > >> > > > > > > > > > > > "field.style"
> > > > > > >> > > > > > > > > > > > > > > > > property,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > including the type,
> > > default
> > > > > > value,
> > > > > > >> > > > > > importance,
> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > preliminary
> > > > > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > > > > >> > > > > > > > > > > > > > > > > > > > > See
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > > > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the
> > "Compatibility,
> > > > > > >> > Deprecation,
> > > > > > >> > > > and
> > > > > > >> > > > > > > > > Migration
> > > > > > >> > > > > > > > > > > > Plan"
> > > > > > >> > > > > > > > > > > > > > > > section
> > > > > > >> > > > > > > > > > > > > > > > > > > still
> > > > > > >> > > > > > > > > > > > > > > > > > > > > accurate after the
> > latest
> > > > > > update?
> > > > > > >> > Seems
> > > > > > >> > > > > like
> > > > > > >> > > > > > > it's
> > > > > > >> > > > > > > > > > still
> > > > > > >> > > > > > > > > > > > > > written
> > > > > > >> > > > > > > > > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > assumption that nested
> > > field
> > > > > > >> syntax
> > > > > > >> > > will
> > > > > > >> > > > be
> > > > > > >> > > > > > > > > hardcoded
> > > > > > >> > > > > > > > > > > or
> > > > > > >> > > > > > > > > > > > > > > opt-in,
> > > > > > >> > > > > > > > > > > > > > > > > > which
> > > > > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > > > > >> > > > > > > > > > > > > > > > > > > > > isn't the case
> anymore.
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The "These
> SMTs
> > do
> > > > not
> > > > > > >> > require
> > > > > > >> > > > > nested
> > > > > > >> > > > > > > > > > structure
> > > > > > >> > > > > > > > > > > > > > > support"
> > > > > > >> > > > > > > > > > > > > > > > > > > section
> > > > > > >> > > > > > > > > > > > > > > > > > > > > mentions a "Drop"
> SMT. I
> > > > think
> > > > > > >> this
> > > > > > >> > may
> > > > > > >> > > > be
> > > > > > >> > > > > > > > > referring
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > Confluent
> > > > > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > > > > >> > > > > > > > > > > > > > > > > > > > > SMT, which isn't a
> part
> > of
> > > > > > Apache
> > > > > > >> > > Kafka.
> > > > > > >> > > > > > Should
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > > drop
> > > > > > >> > > > > > > > > > > > > (heh)
> > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > SMT
> > > > > > >> > > > > > > > > > > > > > > > > > > > from
> > > > > > >> > > > > > > > > > > > > > > > > > > > > the list? Or perhaps
> > just
> > > > > > replace
> > > > > > >> it
> > > > > > >> > > with
> > > > > > >> > > > > > > > > > > "DropHeaders",
> > > > > > >> > > > > > > > > > > > > > which
> > > > > > >> > > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > > > > > currently
> > > > > > >> > > > > > > > > > > > > > > > > > > > > missing from the list
> > and
> > > > > > >> shouldn't
> > > > > > >> > > > require
> > > > > > >> > > > > > any
> > > > > > >> > > > > > > > > > > > > nested-field
> > > > > > >> > > > > > > > > > > > > > > > > related
> > > > > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28, 2022
> at
> > > 2:12
> > > > > PM
> > > > > > >> Jorge
> > > > > > >> > > > > Esteban
> > > > > > >> > > > > > > > > > Quilcate
> > > > > > >> > > > > > > > > > > > > Otoya
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > quilcate.jorge@gmail.com>
> > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thank you, Chris!
> and
> > > > sorry
> > > > > > for
> > > > > > >> the
> > > > > > >> > > > > delayed
> > > > > > >> > > > > > > > > > response.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Please, find my
> > comments
> > > > > > below:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb 2022
> at
> > > > > 17:34,
> > > > > > >> Chris
> > > > > > >> > > > > Egerton
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > <chrise@confluent.io.invalid
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> KIP!
> > > I'd
> > > > > love
> > > > > > >> to
> > > > > > >> > see
> > > > > > >> > > > > > support
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > fields
> > > > > > >> > > > > > > > > > > > > > > > > > added
> > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > out-of-the-box
> SMTs
> > > > > provided
> > > > > > >> with
> > > > > > >> > > > > > Connect.
> > > > > > >> > > > > > > > Here
> > > > > > >> > > > > > > > > > are
> > > > > > >> > > > > > > > > > > > my
> > > > > > >> > > > > > > > > > > > > > > > initial
> > > > > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree that
> > > there's
> > > > a
> > > > > > >> case to
> > > > > > >> > > be
> > > > > > >> > > > > made
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > > > > expanding
> > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > new config
> property
> > > for
> > > > > > >> > > identifying a
> > > > > > >> > > > > > > nested,
> > > > > > >> > > > > > > > > > > > > > to-be-hoisted
> > > > > > >> > > > > > > > > > > > > > > > > > field,
> > > > > > >> > > > > > > > > > > > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example in the KIP
> > > > doesn't
> > > > > > >> really
> > > > > > >> > > > > > > demonstrate
> > > > > > >> > > > > > > > > why
> > > > > > >> > > > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd be
> > helpful
> > > to
> > > > > > >> expand
> > > > > > >> > the
> > > > > > >> > > > > > example
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > add
> > > > > > >> > > > > > > > > > > > > other
> > > > > > >> > > > > > > > > > > > > > > > fields
> > > > > > >> > > > > > > > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > > > > > > > > order
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > show how adding
> > nested
> > > > > field
> > > > > > >> > > support
> > > > > > >> > > > > > > enables
> > > > > > >> > > > > > > > > > users
> > > > > > >> > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > hoist a
> > > > > > >> > > > > > > > > > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > without dropping
> > other
> > > > > > fields
> > > > > > >> > from
> > > > > > >> > > > the
> > > > > > >> > > > > > > value.
> > > > > > >> > > > > > > > > > Maybe
> > > > > > >> > > > > > > > > > > > > > > something
> > > > > > >> > > > > > > > > > > > > > > > > > like
> > > > > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     source =
> > > nested.val
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     field = line
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value
> (before):
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "nested":
> > > {
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "val":
> > > > 42,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "other
> > > > > val":
> > > > > > >> 96
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value (after):
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "nested":
> > > {
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "line":
> > > > {
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >  "val":
> > > > > > 42,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >                 }
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "other
> > > > > val":
> > > > > > >> 96
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I think
> > > "source"
> > > > > is
> > > > > > a
> > > > > > >> > > little
> > > > > > >> > > > > > > strange
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > new
> > > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property name.
> Maybe
> > > > > > >> "hoisted" or
> > > > > > >> > > > > > > > > "hoisted.field"
> > > > > > >> > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and 2.:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The example
> for
> > > > this
> > > > > > SMT
> > > > > > >> is
> > > > > > >> > > > > updated
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > have
> > > > > > >> > > > > > > > > > > > added
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there a
> > > reasonable
> > > > > use
> > > > > > >> case
> > > > > > >> > > for
> > > > > > >> > > > > > > > expanding
> > > > > > >> > > > > > > > > > > > Flatten
> > > > > > >> > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten specific
> > > fields?
> > > > > My
> > > > > > >> > > > > understanding
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > that
> > > > > > >> > > > > > > > > > > > it's
> > > > > > >> > > > > > > > > > > > > > > mostly
> > > > > > >> > > > > > > > > > > > > > > > > > > useful
> > > > > > >> > > > > > > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > writing to systems
> > > like
> > > > > > >> databases
> > > > > > >> > > > that
> > > > > > >> > > > > > > don't
> > > > > > >> > > > > > > > > > > support
> > > > > > >> > > > > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > > > > values
> > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > require everything
> > to
> > > > be a
> > > > > > >> flat
> > > > > > >> > > list
> > > > > > >> > > > of
> > > > > > >> > > > > > > > > key-value
> > > > > > >> > > > > > > > > > > > > pairs.
> > > > > > >> > > > > > > > > > > > > > > > Being
> > > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a nested
> > field
> > > > > > >> wouldn't
> > > > > > >> > > > provide
> > > > > > >> > > > > > any
> > > > > > >> > > > > > > > > > > advantage
> > > > > > >> > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > use
> > > > > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Are there other
> > cases
> > > > > where
> > > > > > it
> > > > > > >> > > would?
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't think
> we
> > > > should
> > > > > > >> > > > > > unconditionally
> > > > > > >> > > > > > > > > change
> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > default
> > > > > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten. It's a
> > > > > > >> > > > backwards-incompatible,
> > > > > > >> > > > > > > > > breaking
> > > > > > >> > > > > > > > > > > > change
> > > > > > >> > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > could
> > > > > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > headaches for
> users.
> > > It
> > > > > > might
> > > > > > >> be
> > > > > > >> > > > > > reasonable
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > change
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > >> > > > > > > > > > > > > > > > > > > > value
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically based
> on
> > > > > whether
> > > > > > >> the
> > > > > > >> > > user
> > > > > > >> > > > > has
> > > > > > >> > > > > > > > > > > specified a
> > > > > > >> > > > > > > > > > > > > > value
> > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > property, but
> > > > considering
> > > > > > the
> > > > > > >> > > > > motivation
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > > > changing
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > default
> > > > > > >> > > > > > > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it creates
> conflicts
> > > > with
> > > > > > the
> > > > > > >> > > > > > > > to-be-introduced
> > > > > > >> > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > field
> > > > > > >> > > > > > > > > > > > > > > > > > syntax
> > > > > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > could arise with
> > > > > downstream
> > > > > > >> SMTs
> > > > > > >> > > > > > regardless
> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > > > > > > > whether
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > user
> > > > > > >> > > > > > > > > > > > > > > > > > has
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly
> > configured
> > > > > > Flatten
> > > > > > >> > with
> > > > > > >> > > > the
> > > > > > >> > > > > > > > "field"
> > > > > > >> > > > > > > > > > > > > > property), I
> > > > > > >> > > > > > > > > > > > > > > > > don't
> > > > > > >> > > > > > > > > > > > > > > > > > > > know
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this would be too
> > > useful
> > > > > > >> either.
> > > > > > >> > I
> > > > > > >> > > > have
> > > > > > >> > > > > > > some
> > > > > > >> > > > > > > > > > > thoughts
> > > > > > >> > > > > > > > > > > > > > below
> > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > >> > > > > > > > > > > > > > > > > > how
> > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > handle possible
> > > > conflicts
> > > > > > >> between
> > > > > > >> > > > names
> > > > > > >> > > > > > > with
> > > > > > >> > > > > > > > > dots
> > > > > > >> > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > their
> > > > > > >> > > > > > > > > > > > > > > > > names
> > > > > > >> > > > > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for nested
> > > field
> > > > > > >> > references
> > > > > > >> > > > that
> > > > > > >> > > > > > > > should
> > > > > > >> > > > > > > > > > > > > hopefully
> > > > > > >> > > > > > > > > > > > > > > make
> > > > > > >> > > > > > > > > > > > > > > > > > > either
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough. With
> the
> > > > > support
> > > > > > >> for
> > > > > > >> > > > nested
> > > > > > >> > > > > > > fields
> > > > > > >> > > > > > > > > in
> > > > > > >> > > > > > > > > > > > other
> > > > > > >> > > > > > > > > > > > > > > SMTs,
> > > > > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > could stay as it is.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > This removes the
> need
> > > for
> > > > > (4)
> > > > > > >> > > changing
> > > > > > >> > > > > > > Flatten
> > > > > > >> > > > > > > > > > config
> > > > > > >> > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > > > > well.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think it's
> fine
> > > to
> > > > > > expand
> > > > > > >> > > > > > ExtractField
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > > support
> > > > > > >> > > > > > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > it might be worth
> > > noting
> > > > > in
> > > > > > >> the
> > > > > > >> > > > > rejected
> > > > > > >> > > > > > > > > > > alternatives
> > > > > > >> > > > > > > > > > > > > > > section
> > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > isn't strictly
> > > necessary
> > > > > > since
> > > > > > >> > you
> > > > > > >> > > > can
> > > > > > >> > > > > > > > replace
> > > > > > >> > > > > > > > > > any
> > > > > > >> > > > > > > > > > > > > single
> > > > > > >> > > > > > > > > > > > > > > > > > > invocation
> > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT that uses
> > > > nested
> > > > > > >> field
> > > > > > >> > > > > notation
> > > > > > >> > > > > > > with
> > > > > > >> > > > > > > > > > > > multiple
> > > > > > >> > > > > > > > > > > > > > > > > > invocations
> > > > > > >> > > > > > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that use
> non-nested
> > > > > > notation.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding it.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit:
> > "RegerRouter"
> > > > > should
> > > > > > >> be
> > > > > > >> > > > > > > "RegexRouter"
> > > > > > >> > > > > > > > > in
> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > list
> > > > > > >> > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > SMTs
> > > > > > >> > > > > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > > > do
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > not require nested
> > > > > structure
> > > > > > >> > > support.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing it.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be rare
> > for
> > > > dots
> > > > > > in
> > > > > > >> > field
> > > > > > >> > > > > names
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > occur
> > > > > > >> > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > wild
> > > > > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be so
> > certain
> > > > of
> > > > > > >> this),
> > > > > > >> > > but
> > > > > > >> > > > > > unless
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > > want
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > inflict
> > > > > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > users of Flatten,
> I
> > > > think
> > > > > > >> we're
> > > > > > >> > > going
> > > > > > >> > > > > to
> > > > > > >> > > > > > > have
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > think
> > > > > > >> > > > > > > > > > > > > > > about
> > > > > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between dotted
> > > notation
> > > > > and
> > > > > > >> > > > non-nested
> > > > > > >> > > > > > > fields
> > > > > > >> > > > > > > > > > whose
> > > > > > >> > > > > > > > > > > > > names
> > > > > > >> > > > > > > > > > > > > > > > > contain
> > > > > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > don't think this
> is
> > > > > actually
> > > > > > >> > such a
> > > > > > >> > > > bad
> > > > > > >> > > > > > > > thing,
> > > > > > >> > > > > > > > > > > > though.
> > > > > > >> > > > > > > > > > > > > I
> > > > > > >> > > > > > > > > > > > > > > > agree
> > > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > notation is
> > intuitive
> > > > and
> > > > > > >> pretty
> > > > > > >> > > > > > > commonplace
> > > > > > >> > > > > > > > > (in
> > > > > > >> > > > > > > > > > > > tools
> > > > > > >> > > > > > > > > > > > > > like
> > > > > > >> > > > > > > > > > > > > > > > jq,
> > > > > > >> > > > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > example), so I'd
> > like
> > > it
> > > > > if
> > > > > > we
> > > > > > >> > > could
> > > > > > >> > > > > > stick
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > it.
> > > > > > >> > > > > > > > > > > > What
> > > > > > >> > > > > > > > > > > > > > > about
> > > > > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > escape syntax,
> > using a
> > > > > > >> backslash?
> > > > > > >> > > > That
> > > > > > >> > > > > > way,
> > > > > > >> > > > > > > > > users
> > > > > > >> > > > > > > > > > > > could
> > > > > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > between
> "this.field"
> > > > > (which
> > > > > > >> would
> > > > > > >> > > > refer
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > field
> > > > > > >> > > > > > > > > > > > > > > > > > "field"
> > > > > > >> > > > > > > > > > > > > > > > > > > > > under
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the top-level
> "this"
> > > > > field),
> > > > > > >> and
> > > > > > >> > > > > > > > "this\.field"
> > > > > > >> > > > > > > > > > > (which
> > > > > > >> > > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > > > > refer
> > > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> > > > "this.field").
> > > > > > >> Like
> > > > > > >> > > with
> > > > > > >> > > > > most
> > > > > > >> > > > > > > > > > languages
> > > > > > >> > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > > use
> > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > for escape
> > sequences,
> > > it
> > > > > > could
> > > > > > >> > also
> > > > > > >> > > > be
> > > > > > >> > > > > > used
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > escape
> > > > > > >> > > > > > > > > > > > > > > itself,
> > > > > > >> > > > > > > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that a field name
> > > > > contains a
> > > > > > >> > > > > backslash. I
> > > > > > >> > > > > > > > think
> > > > > > >> > > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > more
> > > > > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > simpler than,
> e.g.,
> > > > > adding a
> > > > > > >> new
> > > > > > >> > > > config
> > > > > > >> > > > > > > > > property
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > toggle
> > > > > > >> > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > be used when
> parsing
> > > > > nested
> > > > > > >> field
> > > > > > >> > > > > > > references.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like this
> approach.
> > > > Adding
> > > > > > to
> > > > > > >> the
> > > > > > >> > > > KIP.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't think
> we
> > > can
> > > > > > >> > > > unconditionally
> > > > > > >> > > > > > > turn
> > > > > > >> > > > > > > > > this
> > > > > > >> > > > > > > > > > > > > feature
> > > > > > >> > > > > > > > > > > > > > > on.
> > > > > > >> > > > > > > > > > > > > > > > > The
> > > > > > >> > > > > > > > > > > > > > > > > > > > risk
> > > > > > >> > > > > > > > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > breaking existing
> > > > > pipelines
> > > > > > >> > > > (especially
> > > > > > >> > > > > > > ones
> > > > > > >> > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > involve,
> > > > > > >> > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > combination of the
> > > > Flatten
> > > > > > and
> > > > > > >> > Cast
> > > > > > >> > > > > SMTs)
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > > pretty
> > > > > > >> > > > > > > > > > > > > > high. I
> > > > > > >> > > > > > > > > > > > > > > > > think
> > > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > should be an
> opt-in
> > > > > feature,
> > > > > > >> at
> > > > > > >> > > least
> > > > > > >> > > > > > until
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > next
> > > > > > >> > > > > > > > > > > > > > major
> > > > > > >> > > > > > > > > > > > > > > > > > release.
> > > > > > >> > > > > > > > > > > > > > > > > > > > One
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > we could
> accomplish
> > > this
> > > > > is
> > > > > > by
> > > > > > >> > > > > > introducing
> > > > > > >> > > > > > > a
> > > > > > >> > > > > > > > > new
> > > > > > >> > > > > > > > > > > > > > > > "field.style"
> > > > > > >> > > > > > > > > > > > > > > > > > > (name
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > obviously subject
> to
> > > > > change)
> > > > > > >> > > property
> > > > > > >> > > > > > with
> > > > > > >> > > > > > > > > values
> > > > > > >> > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > "plain"
> > > > > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If set
> to
> > > > > "plain"
> > > > > > >> then
> > > > > > >> > > the
> > > > > > >> > > > > > > current
> > > > > > >> > > > > > > > > > > > non-nested
> > > > > > >> > > > > > > > > > > > > > > > > behavior
> > > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > and if set to
> > > "nested",
> > > > > then
> > > > > > >> the
> > > > > > >> > > > > proposed
> > > > > > >> > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > behavior
> > > > > > >> > > > > > > > > > > > > > > is.
> > > > > > >> > > > > > > > > > > > > > > > > We
> > > > > > >> > > > > > > > > > > > > > > > > > > can
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > consider updating
> > the
> > > > > > default
> > > > > > >> > value
> > > > > > >> > > > to
> > > > > > >> > > > > > > > "nested"
> > > > > > >> > > > > > > > > > in
> > > > > > >> > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > future
> > > > > > >> > > > > > > > > > > > > > > > > major
> > > > > > >> > > > > > > > > > > > > > > > > > > > > release
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > (or even codify
> that
> > > > plan
> > > > > in
> > > > > > >> this
> > > > > > >> > > > KIP,
> > > > > > >> > > > > if
> > > > > > >> > > > > > > > > there's
> > > > > > >> > > > > > > > > > > > > enough
> > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > >> > > > > > > > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > This would also
> > leave
> > > > the
> > > > > > door
> > > > > > >> > open
> > > > > > >> > > > for
> > > > > > >> > > > > > > > adding
> > > > > > >> > > > > > > > > > > > > recursive
> > > > > > >> > > > > > > > > > > > > > > > > support
> > > > > > >> > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > the future by
> > adding a
> > > > > > >> permitted
> > > > > > >> > > > value
> > > > > > >> > > > > of
> > > > > > >> > > > > > > > > > > > "recursive".
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of the
> linked
> > > > > tickets
> > > > > > >> in
> > > > > > >> > the
> > > > > > >> > > > > > > > > "Motivation"
> > > > > > >> > > > > > > > > > > > > section,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > https://issues.apache.org/jira/browse/KAFKA-10640
> > > > ,
> > > > > > >> > > > > > > > > > > > has
> > > > > > >> > > > > > > > > > > > > > an
> > > > > > >> > > > > > > > > > > > > > > > open
> > > > > > >> > > > > > > > > > > > > > > > > > PR
> > > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > that propose
> adding
> > > > > > recursive
> > > > > > >> > > support
> > > > > > >> > > > > to
> > > > > > >> > > > > > > some
> > > > > > >> > > > > > > > > > SMTs.
> > > > > > >> > > > > > > > > > > > > Have
> > > > > > >> > > > > > > > > > > > > > > you
> > > > > > >> > > > > > > > > > > > > > > > > > > > considered
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> > > > functionality
> > > > > > for
> > > > > > >> > your
> > > > > > >> > > > > KIP?
> > > > > > >> > > > > > Or
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > > your
> > > > > > >> > > > > > > > > > > > aim
> > > > > > >> > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > stick
> > > > > > >> > > > > > > > > > > > > > > > > > > > solely
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > nested field
> > support?
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > I like the
> > `field.style`
> > > > > > >> > > configuration
> > > > > > >> > > > > flag
> > > > > > >> > > > > > > > > > approach.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for pointing
> > out
> > > > the
> > > > > > >> > > `recursive`
> > > > > > >> > > > > > > > approach.
> > > > > > >> > > > > > > > > > > Will
> > > > > > >> > > > > > > > > > > > > add
> > > > > > >> > > > > > > > > > > > > > > > > `nested`
> > > > > > >> > > > > > > > > > > > > > > > > > > at
> > > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's check
> > the
> > > > > demand
> > > > > > >> for
> > > > > > >> > > > > > > `recursive`
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > > > consider
> > > > > > >> > > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > > > > > > > part
> > > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > this or another KIP.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > I have added the
> > > following
> > > > > on
> > > > > > >> the
> > > > > > >> > > KIP:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs could
> > extend
> > > > > this
> > > > > > >> > support
> > > > > > >> > > > > for:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive
> notation:
> > > > name a
> > > > > > >> field
> > > > > > >> > > and
> > > > > > >> > > > > > apply
> > > > > > >> > > > > > > it
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > all
> > > > > > >> > > > > > > > > > > > > > fields
> > > > > > >> > > > > > > > > > > > > > > > > > across
> > > > > > >> > > > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > schema matching that
> > > name.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > - Access to arrays:
> > > Adding
> > > > > > `[]`
> > > > > > >> > > > notation
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > > > represent
> > > > > > >> > > > > > > > > > > > > > access
> > > > > > >> > > > > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > > > > > > arrays
> > > > > > >> > > > > > > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs to
> > fields
> > > > > within
> > > > > > >> an
> > > > > > >> > > > array.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 8,
> 2022
> > at
> > > > > 1:23
> > > > > > PM
> > > > > > >> > > Jorge
> > > > > > >> > > > > > > Esteban
> > > > > > >> > > > > > > > > > > Quilcate
> > > > > > >> > > > > > > > > > > > > > > Otoya <
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > quilcate.jorge@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev team,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like to
> start
> > a
> > > > new
> > > > > > >> > > discussion
> > > > > > >> > > > > > thread
> > > > > > >> > > > > > > > on
> > > > > > >> > > > > > > > > > > Kafka
> > > > > > >> > > > > > > > > > > > > > > Connect
> > > > > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP is
> aimed
> > to
> > > > > > include
> > > > > > >> > > > support
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > > nested
> > > > > > >> > > > > > > > > > > > > > > structures
> > > > > > >> > > > > > > > > > > > > > > > on
> > > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs — where
> this
> > > make
> > > > > > >> sense.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward
> to
> > > > your
> > > > > > >> > feedback.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Robert Yokota <ra...@gmail.com>]

Hi Jorge,

Yes, to escape a backtick I would recommend a backslash.

For example, MySQL allows identifiers to be surrounded with single quotes,
double quotes, or backticks, and they use backslash to escape.

Jsonata went with primarily backticks as they are less common in strings
that appear in JSON, and it seems to have worked nicely for them.

Regards,
Robert

On Fri, Aug 12, 2022 at 10:01 AM Jorge Esteban Quilcate Otoya <
quilcate.jorge@gmail.com> wrote:

> Thanks Robert! And sorry for the late reply.
>
> That's a great catch and will require the current proposal to be updated.
>
> Using back-ticks is a good proposal. Though, the challenge we got is that
> JSON allows really any character into attribute names, including
> back-ticks.
> I wonder what's the best approach to escape back-ticks then. Given that
> those use-cases should be minimal we could fallback to backslash to escape
> back-ticks?
>
> What do you think?
>
> Many thanks,
> Jorge.
>
>
> On Wed, 20 Jul 2022 at 23:03, Robert Yokota <ra...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm late to this thread, but would like to comment on the double
> dot/double
> > asterisk syntax.
> >
> > Unfortunately double dot is often used in JSON Path as a descendant
> > selector, see https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html
> >
> > I think a better notation would be to use backticks to quote any names
> with
> > a dot, such as  root.`child.with.dot`.  See Jsonata syntax for example:
> > http://docs.jsonata.org/simple
> >
> > Robert
> >
> > On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
> > quilcate.jorge@gmail.com> wrote:
> >
> > > Thanks Chris! I have updated the KIP to include this fix.
> > >
> > > I will keep the array as a potential improvement at the moment, and out
> > of
> > > the scope of this KIP.
> > >
> > > Thanks,
> > > Jorge.
> > >
> > > On Tue, 28 Jun 2022 at 23:19, Chris Egerton <fe...@gmail.com>
> > > wrote:
> > >
> > > > Hi Jorge,
> > > >
> > > > Apologies for the long delay, had my own KIP-related work to focus
> on.
> > > >
> > > > I think it's fine to include array accesses but it's not a blocker.
> I'm
> > > +1
> > > > either way. On that front though, I think the MaskField section might
> > > need
> > > > to be updated as it still mentions arrays and deep-scan?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> > > > quilcate.jorge@gmail.com> wrote:
> > > >
> > > > > Hi there,
> > > > >
> > > > > I have update the KIP to the previous state voted, including the
> > > > > configuration change from `field.style` to `field.syntax.version`.
> > > > >
> > > > > I'll bump the vote thread as well to see if there's agreement on
> > adding
> > > > > this feature to Connect.
> > > > >
> > > > > Cheers,
> > > > > Jorge.
> > > > >
> > > > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jorge@gmail.com> wrote:
> > > > >
> > > > > > Thanks, Chris. Your feedback is much appreciated!
> > > > > >
> > > > > > I see how the current proposal might be underestimating some edge
> > > > cases.
> > > > > > I'm happy to move the design for deep-scan and multi-values to
> > future
> > > > > > developments related with this KIP and reduce its scope, though
> > open
> > > > for
> > > > > > more feedback.
> > > > > >
> > > > > > Also, just to be sure, are you proposing also to not include
> array
> > > > access
> > > > > > at this stage?
> > > > > >
> > > > > > Thanks,
> > > > > > Jorge.
> > > > > >
> > > > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <
> > fearthecellos@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Jorge,
> > > > > >>
> > > > > >> I've done some more thinking and I hate to say it, but I think
> the
> > > > > syntax
> > > > > >> does need to be expanded. Right now it's clear what "a.b" refers
> > to
> > > > and
> > > > > >> what "a..b" refers to, but what about "a...b"? Is that referring
> > to
> > > > > >> subfield ".b" of field "a", or subfield "b" of field "a."? This
> > gets
> > > > > even
> > > > > >> more complicated when thinking about fields whose names are
> > > > exclusively
> > > > > >> made up of dots.
> > > > > >>
> > > > > >> I'm also a little hesitant to mix the cases of multi-value paths
> > and
> > > > > deep
> > > > > >> scans. What if you only want to access one subfield deep for an
> > SMT,
> > > > > >> instead of recursing through all the children of a given field?
> > It's
> > > > > akin
> > > > > >> to the distinction between * and ** with file globbing patterns,
> > and
> > > > > there
> > > > > >> could be a substantial performance difference if you have
> > > > heavily-nested
> > > > > >> fields.
> > > > > >>
> > > > > >> Ultimately, I think that if the proposed "field.syntax.version"
> > > > property
> > > > > >> sits well with people, it might be better to reduce the scope of
> > the
> > > > KIP
> > > > > >> back to the original proposal and just focus on adding support
> for
> > > > > >> explicitly-specified nested values, with no multi-value paths
> > > > > whatsoever,
> > > > > >> knowing that we have an easy way to introduce new syntax and
> > > features
> > > > in
> > > > > >> the future. (We could probably leave the "a...b" case for that
> > next
> > > > > >> version
> > > > > >> too.)
> > > > > >>
> > > > > >> I was a huge fan of this KIP before we started trying to address
> > > more
> > > > > >> complex use cases, and although I don't want to write those
> off, I
> > > > think
> > > > > >> we
> > > > > >> may have bitten off more than we can chew in time for the 3.3.0
> > > > release
> > > > > >> and
> > > > > >> would hate to see this KIP get delayed as a result.
> > > > > >>
> > > > > >> I'd be really curious to hear from Joshua and Tom on this front,
> > > > though.
> > > > > >> Is
> > > > > >> it acceptable to move more incrementally here and settle on the
> > > syntax
> > > > > >> version property as our means of introducing new features, or is
> > it
> > > > > >> preferable to implement things monolithically and try to get
> > > > everything
> > > > > >> (or
> > > > > >> at least, as much as possible) right the first time?
> > > > > >>
> > > > > >> Thanks again for your continued effort on this KIP!
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> Chris
> > > > > >>
> > > > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate Otoya <
> > > > > >> quilcate.jorge@gmail.com> wrote:
> > > > > >>
> > > > > >> > Thanks, Chris!
> > > > > >> >
> > > > > >> > Please, find my comments below:
> > > > > >> >
> > > > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> > > fearthecellos@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Jorge,
> > > > > >> > >
> > > > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > > > >> > >
> > > > > >> > > 1. Under the "Accessing multiple values by deep-scan" header
> > > it's
> > > > > >> stated
> > > > > >> > > that "If deep-scan is used, it must have only one field
> after
> > > the
> > > > > >> > asterisk
> > > > > >> > > level.". However, in example 3 for the Cast SMT and other
> > > examples
> > > > > for
> > > > > >> > > other SMTs, the spec contains a field of "*.child.k2", which
> > > > appears
> > > > > >> to
> > > > > >> > > have two fields after the asterisk level. I may be
> > > > misunderstanding
> > > > > >> the
> > > > > >> > > proposal, but it seems like the two contradict each other.
> > > > > >> > >
> > > > > >> >
> > > > > >> > Thanks for catching this. I have clarified it by removing this
> > > > > >> restriction.
> > > > > >> > Also, have extended the deep-scan scenarios.
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 2. I'm a little unclear on why we need the special handling
> > for
> > > > > arrays
> > > > > >> > > where, for an array field "a", the field name "a" can be
> > treated
> > > > as
> > > > > >> > either
> > > > > >> > > the array itself, or every element in the array. Is there a
> > > reason
> > > > > we
> > > > > >> > can't
> > > > > >> > > use the field name "a.*" to handle the latter case, and "a"
> to
> > > > > handle
> > > > > >> the
> > > > > >> > > former?
> > > > > >> > >
> > > > > >> >
> > > > > >> > Agree, this is confusing. I like the `a.*` approach to access
> > > array
> > > > > >> items.
> > > > > >> > I have added this to the proposal.
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 3. How would a user specify that they'd like to access a
> field
> > > > with
> > > > > >> the
> > > > > >> > > literal name "*"?
> > > > > >> > >
> > > > > >> >
> > > > > >> > Good one. I'm proposing an approach similar to how it's
> proposed
> > > to
> > > > > >> escape
> > > > > >> > dots, with a double-asterisk. Curious on your thoughts around
> > > this.
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 4. For the Cast SMT, do you think it might bite some people
> if
> > > > > fields
> > > > > >> > that
> > > > > >> > > can't be cast correctly are silently ignored? I'm imagining
> > the
> > > > case
> > > > > >> > where
> > > > > >> > > none of the fields in a multi-path expression can be cast
> > > > correctly
> > > > > >> and
> > > > > >> > it
> > > > > >> > > ends up eating half of someone's day to track down why their
> > SMT
> > > > > isn't
> > > > > >> > > doing anything.
> > > > > >> > >
> > > > > >> >
> > > > > >> > If I understand correctly, this challenge could be relevant
> > across
> > > > > SMTs.
> > > > > >> > At the moment, most/all? SMTs just silently ignore.
> > > > > >> > Was thinking about adding a flag `field.on.path.not.found` to
> > > either
> > > > > >> ignore
> > > > > >> > or fail when no paths are found. What do your think?
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 5. For the ExtractField and ValueToKey SMTs, what happens
> if a
> > > > > >> deep-scan
> > > > > >> > > field name is used, but only one field is found? Is the
> > > resulting
> > > > > >> field
> > > > > >> > > still an array, or is it just the single field that was
> found?
> > > > (FWIW
> > > > > >> I'm
> > > > > >> > > leaning towards keeping it an array just to keep schemas
> > > > consistent
> > > > > >> in a
> > > > > >> > > pipeline in case the number of fields found fluctuates
> across
> > > > > >> records.)
> > > > > >> > >
> > > > > >> > > Agree. Will clarify that an array is always produced even
> for
> > 1
> > > > or 0
> > > > > >> > fields are found.
> > > > > >> >
> > > > > >> >
> > > > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that "As this
> SMT
> > > > > affects
> > > > > >> > only
> > > > > >> > > existing fields, additional configurations will not be
> > > required.".
> > > > > >> Given
> > > > > >> > > the new "field.syntax.version" property, this part should
> > > probably
> > > > > be
> > > > > >> > > removed.
> > > > > >> > >
> > > > > >> > Agree.
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 7. Is recursive descent intentionally excluded? That was an
> > > > > important
> > > > > >> > part
> > > > > >> > > of Joshua's KIP and his feedback on this KIP; I think it's
> > worth
> > > > > >> pursuing
> > > > > >> > > if we can.
> > > > > >> > >
> > > > > >> >
> > > > > >> > My understanding from Joshua's feedback is that by including
> > > support
> > > > > for
> > > > > >> > deep-scan, we are already covering the recursive
> functionality.
> > > > > Though,
> > > > > >> I
> > > > > >> > may be missing something.
> > > > > >> >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Cheers,
> > > > > >> > >
> > > > > >> > > Chris
> > > > > >> > >
> > > > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate
> Otoya <
> > > > > >> > > quilcate.jorge@gmail.com> wrote:
> > > > > >> > >
> > > > > >> > > > Thanks, Chris!
> > > > > >> > > >
> > > > > >> > > > I have updated the KIP with the rejected alternatives
> > updated.
> > > > > >> Also, I
> > > > > >> > > have
> > > > > >> > > > drafted the support for arrays and deep scans as part of
> the
> > > > > >> proposed
> > > > > >> > > > notation to make it more complete, even though these can
> be
> > > > > >> implemented
> > > > > >> > > in
> > > > > >> > > > multiple PRs.
> > > > > >> > > >
> > > > > >> > > > Looking forward to your feedback.
> > > > > >> > > >
> > > > > >> > > > Cheers,
> > > > > >> > > > Jorge.
> > > > > >> > > >
> > > > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > > > >> fearthecellos@gmail.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Jorge,
> > > > > >> > > > >
> > > > > >> > > > > I really appreciate the effort you've made to simplify
> the
> > > > > syntax
> > > > > >> and
> > > > > >> > > > > feature set of a JSONPath-based approach as much as
> > > possible.
> > > > > I'm
> > > > > >> > still
> > > > > >> > > > > hesitant to continue with it, though.
> > > > > >> > > > >
> > > > > >> > > > > 1. The syntax is much less friendly. Just compare
> > > > > >> "top.mid.bottom" to
> > > > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses
> JSONPath
> > or
> > > > > even
> > > > > >> > JSON,
> > > > > >> > > > and
> > > > > >> > > > > the learning curve for the former is going to be
> steeper.
> > > The
> > > > > >> > examples
> > > > > >> > > in
> > > > > >> > > > > the new KIP draft you published demonstrate this pretty
> > > well,
> > > > > and
> > > > > >> > this
> > > > > >> > > is
> > > > > >> > > > > without diving into the details of what escape syntax
> > would
> > > > look
> > > > > >> > like.
> > > > > >> > > > >
> > > > > >> > > > > 2. The three new major features that this syntax adds
> > (array
> > > > > >> > accesses,
> > > > > >> > > > deep
> > > > > >> > > > > scans, and multi-value paths) could all be added pretty
> > > easily
> > > > > to
> > > > > >> the
> > > > > >> > > dot
> > > > > >> > > > > notation syntax without introducing brackets and dollar
> > > signs.
> > > > > >> Array
> > > > > >> > > > > accesses can be described using the same syntax as
> > > struct/map
> > > > > >> field
> > > > > >> > > > access,
> > > > > >> > > > > deep scans can be described using '*', and multi-value
> > paths
> > > > can
> > > > > >> be
> > > > > >> > > > > described by referencing the name of a field that's
> > expected
> > > > to
> > > > > >> have
> > > > > >> > > > > children. These are all top-of-the-head ideas and can
> > > probably
> > > > > be
> > > > > >> > > > refined,
> > > > > >> > > > > but hopefully they demonstrate that we can keep the
> syntax
> > > > > simple
> > > > > >> > > without
> > > > > >> > > > > sacrificing features. Of course, the question of leaving
> > > room
> > > > > for
> > > > > >> > > future
> > > > > >> > > > > features might arise... given that these are the
> > > > out-of-the-box
> > > > > >> SMTs
> > > > > >> > > that
> > > > > >> > > > > are likely to be the first that many people encounter,
> I'd
> > > err
> > > > > on
> > > > > >> the
> > > > > >> > > > side
> > > > > >> > > > > of simplicity and a gentle learning curve; if people
> need
> > to
> > > > get
> > > > > >> more
> > > > > >> > > out
> > > > > >> > > > > of transforms, the option to implement their own is
> still
> > > > there.
> > > > > >> If
> > > > > >> > we
> > > > > >> > > > can
> > > > > >> > > > > address 95% of use cases with something easy to use,
> it's
> > > not
> > > > > >> worth
> > > > > >> > > > making
> > > > > >> > > > > the feature harder for everyone to use just to
> accommodate
> > > the
> > > > > >> > > remaining
> > > > > >> > > > > 5%.
> > > > > >> > > > >
> > > > > >> > > > > 3. The advantage of leveraging an existing syntax is
> > > twofold:
> > > > > >> users
> > > > > >> > who
> > > > > >> > > > are
> > > > > >> > > > > already familiar with that syntax don't need to learn a
> > new
> > > > > >> syntax,
> > > > > >> > and
> > > > > >> > > > > maintainers of the syntax get to leverage existing
> > libraries
> > > > and
> > > > > >> > > > > documentation for the syntax. With the current proposal,
> > > > reusing
> > > > > >> > > > libraries
> > > > > >> > > > > is off the table, which means that we're going to have
> to
> > > > parse
> > > > > >> this
> > > > > >> > > > > ourselves (including all the escape syntax edge cases),
> > and
> > > we
> > > > > >> won't
> > > > > >> > be
> > > > > >> > > > > able to automatically leverage new features added to
> that
> > > > > syntax.
> > > > > >> And
> > > > > >> > > > given
> > > > > >> > > > > how stripped-down the syntax is in comparison to full
> > > > JSONPath,
> > > > > >> > there's
> > > > > >> > > > > still going to be a learning curve for users who are
> > already
> > > > > >> familiar
> > > > > >> > > > with
> > > > > >> > > > > it, and we'll still have to document how Connect's
> variant
> > > of
> > > > > >> > JSONPath
> > > > > >> > > > > works either instead of or in addition to just linking
> to
> > a
> > > > > >> > > > well-maintained
> > > > > >> > > > > third-party docs site.
> > > > > >> > > > >
> > > > > >> > > > > On the topic of "field.path" and "include.path" vs.
> > > > > >> "field.style", I
> > > > > >> > > > > actually think that a single property per SMT is cleaner
> > and
> > > > > >> simpler.
> > > > > >> > > > > Allowing users to mix and match styles within the same
> SMT
> > > > > config
> > > > > >> > seems
> > > > > >> > > > > like a recipe for confusion, and with a single property
> > that
> > > > > >> dictates
> > > > > >> > > > field
> > > > > >> > > > > syntax behavior, we leave the door open to change the
> > > default
> > > > > at a
> > > > > >> > > later
> > > > > >> > > > > date. We could even fully remove support for plain field
> > > > > notation
> > > > > >> at
> > > > > >> > > some
> > > > > >> > > > > point and still be able to retain the simple property
> > names
> > > of
> > > > > >> > "field",
> > > > > >> > > > > "include", etc. instead of forcing people to use
> > > "field.path"
> > > > > and
> > > > > >> the
> > > > > >> > > > like.
> > > > > >> > > > > That said, the term "field.style" and the permitted
> values
> > > > might
> > > > > >> be a
> > > > > >> > > > > little ambiguous. One alternative, though a little
> > > > heavy-handed,
> > > > > >> is
> > > > > >> > to
> > > > > >> > > > > change it to "field.syntax.version" with permitted
> values
> > of
> > > > > "V1"
> > > > > >> > > > (default,
> > > > > >> > > > > equivalent to "field.style = plain") and "V2"
> (equivalent
> > to
> > > > > >> > > > "field.style =
> > > > > >> > > > > nested"). This would leave us room in the future to make
> > > > further
> > > > > >> > > changes
> > > > > >> > > > to
> > > > > >> > > > > the syntax without having to come up with new names,
> > > although
> > > > it
> > > > > >> does
> > > > > >> > > > > sacrifice a little bit in that the permitted values are
> no
> > > > > longer
> > > > > >> > > > > self-describing.
> > > > > >> > > > >
> > > > > >> > > > > Cheers,
> > > > > >> > > > >
> > > > > >> > > > > Chris
> > > > > >> > > > >
> > > > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban Quilcate
> > > Otoya <
> > > > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Thank you all for your feedback, and sorry for the
> long
> > > wait
> > > > > >> for a
> > > > > >> > > > reply.
> > > > > >> > > > > >
> > > > > >> > > > > > I would like to explore the idea of
> > > JSONPath-inspired/subset
> > > > > >> > > notation a
> > > > > >> > > > > bit
> > > > > >> > > > > > further:
> > > > > >> > > > > >
> > > > > >> > > > > > It will need to be a much-reduced version of JSONPath:
> > > > > >> > > > > > - No full support for JsonPath therefore an additional
> > > > > >> dependency.
> > > > > >> > > > > > - All paths must start with '$'
> > > > > >> > > > > > - No functions support or other operators allowed.
> > > > > >> > > > > > - JsonPath dotted and square-bracket notations can be
> > > > > supported
> > > > > >> to
> > > > > >> > > > avoid
> > > > > >> > > > > > escaping dots or other characters: `$a.b.c` and
> > > > > `$['a.b']['c']`
> > > > > >> -
> > > > > >> > or
> > > > > >> > > we
> > > > > >> > > > > > could only support the second one as it's more
> complete.
> > > > > >> > > > > > - Add support for arrays with `[<integer>]`, e.g.
> > > `$a.[1].b`
> > > > > >> > > > > > - Add support for multiple-value paths using array
> > access
> > > > > >> `$a.*.b`
> > > > > >> > or
> > > > > >> > > > > deep
> > > > > >> > > > > > scan `$a..b`.
> > > > > >> > > > > >     - Some SMTs that will benefit from this:
> > `MaskField`,
> > > > > >> `Cast`,
> > > > > >> > > > > > `ReplaceField`.
> > > > > >> > > > > > - We could introduce a `path[s]` config under the
> > current
> > > > > >> > > > configurations
> > > > > >> > > > > to
> > > > > >> > > > > > apply this feature so no compatibility issues are
> > > > introduced:
> > > > > >> e.g.
> > > > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > > > >> > > > > >
> > > > > >> > > > > > With these,  100, 101, 102, and 103 will be
> effectively
> > > > > solved.
> > > > > >> > > > > >
> > > > > >> > > > > > The main challenge that I see at the moment is that by
> > > being
> > > > > >> JSON
> > > > > >> > > > > > path-like, there may be some edge cases that I can't
> > > foresee
> > > > > at
> > > > > >> the
> > > > > >> > > > > moment,
> > > > > >> > > > > > that could make this hard to implement, test, and
> > maintain
> > > > in
> > > > > >> the
> > > > > >> > > long
> > > > > >> > > > > run.
> > > > > >> > > > > >
> > > > > >> > > > > > I'll appreciate your feedback on how this
> JSONPath-based
> > > > > >> > alternative
> > > > > >> > > > > > compares to the dotted notation initially proposed,
> and
> > if
> > > > it
> > > > > >> > matches
> > > > > >> > > > > your
> > > > > >> > > > > > feedback.
> > > > > >> > > > > > I have drafted a copy of the KIP to change the
> examples
> > to
> > > > > >> JSONPath
> > > > > >> > > > > > approach and validate some differences:
> > > > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > > > >> > > > > >
> > > > > >> > > > > > About 104. I agreed with Chris. We can handle this as
> > part
> > > > of
> > > > > a
> > > > > >> new
> > > > > >> > > > KIP.
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > > Jorge.
> > > > > >> > > > > >
> > > > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > > > >> > fearthecellos@gmail.com
> > > > > >> > > >
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Joshua,
> > > > > >> > > > > > >
> > > > > >> > > > > > > I have a few reservations about using JsonPath
> > notation
> > > > > here.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 1. There's likely to be a substantial performance
> > > penalty
> > > > > for
> > > > > >> > > > > converting
> > > > > >> > > > > > > between the Kafka Connect format and something that
> a
> > > > > JsonPath
> > > > > >> > > > library
> > > > > >> > > > > > > would understand.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 2. The complexity of the feature will be
> significantly
> > > > > >> higher. It
> > > > > >> > > > will
> > > > > >> > > > > be
> > > > > >> > > > > > > harder to test, document, and implement. There will
> be
> > > > many
> > > > > >> more
> > > > > >> > > edge
> > > > > >> > > > > > cases
> > > > > >> > > > > > > to consider and support, and we'll be on the hook to
> > > > handle
> > > > > >> any
> > > > > >> > > bugs
> > > > > >> > > > or
> > > > > >> > > > > > > inconsistencies that arise, either as a result of
> our
> > > use
> > > > of
> > > > > >> the
> > > > > >> > > > > JsonPath
> > > > > >> > > > > > > library we choose, or as a result of bugs in that
> > > library.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 3. It's not clear that JsonPath is superior or even
> > > > suitable
> > > > > >> for
> > > > > >> > > some
> > > > > >> > > > > of
> > > > > >> > > > > > > the SMTs proposed here. What would be the advantages
> > of
> > > > > >> JsonPath
> > > > > >> > > with
> > > > > >> > > > > the
> > > > > >> > > > > > > InsertField or HoistField SMTs?
> > > > > >> > > > > > >
> > > > > >> > > > > > > I also don't think that adding dot notation is
> > > unfriendly
> > > > to
> > > > > >> > users;
> > > > > >> > > > > many
> > > > > >> > > > > > > have proposed this type of syntax in the past, and
> > it's
> > > > > >> > frequently
> > > > > >> > > > used
> > > > > >> > > > > > in
> > > > > >> > > > > > > informal discussions to refer to nested fields. If
> the
> > > > > >> proposed
> > > > > >> > > > syntax
> > > > > >> > > > > > was
> > > > > >> > > > > > > not already intuitive then a case against deciding
> on
> > > our
> > > > > own
> > > > > >> > might
> > > > > >> > > > be
> > > > > >> > > > > > more
> > > > > >> > > > > > > convincing, but as things stand, simple dot notation
> > is
> > > > > likely
> > > > > >> > > going
> > > > > >> > > > to
> > > > > >> > > > > > be
> > > > > >> > > > > > > easier for users to understand than JsonPath syntax.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Cheers,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Chris
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua Grisham <
> > > > > >> > > > grishamjosh@gmail.com>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hello all!
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Sorry that I come a bit later to the party here,
> > but I
> > > > am
> > > > > >> the
> > > > > >> > one
> > > > > >> > > > who
> > > > > >> > > > > > > wrote
> > > > > >> > > > > > > > KIP-683 [1] for recursive support (just simply
> > looping
> > > > > >> through
> > > > > >> > > all
> > > > > >> > > > > > child
> > > > > >> > > > > > > > non-primitive structures for the same matching
> > > name(s))
> > > > > >> which
> > > > > >> > is
> > > > > >> > > a
> > > > > >> > > > > > > slightly
> > > > > >> > > > > > > > different way to try and solve a similar
> requirement
> > > --
> > > > > >> > > > unfortunately
> > > > > >> > > > > > at
> > > > > >> > > > > > > > the time the dev community was not quite as active
> > and
> > > > > then
> > > > > >> I
> > > > > >> > > also
> > > > > >> > > > > got
> > > > > >> > > > > > > busy
> > > > > >> > > > > > > > with work and just life in general so wasn't able
> to
> > > > > follow
> > > > > >> up
> > > > > >> > or
> > > > > >> > > > > push
> > > > > >> > > > > > > it.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I do think it is a very good idea to have some
> kind
> > of
> > > > > >> > path-like
> > > > > >> > > > > > > expression
> > > > > >> > > > > > > > to be able to specifically address a nested field,
> > as
> > > I
> > > > > can
> > > > > >> see
> > > > > >> > > > that
> > > > > >> > > > > > the
> > > > > >> > > > > > > > simple "recursive" case could potentially result
> in
> > > > > >> unwanted or
> > > > > >> > > > > > > unexpected
> > > > > >> > > > > > > > behavior, plus there is the potential to
> introduce a
> > > bit
> > > > > of
> > > > > >> a
> > > > > >> > > > > > performance
> > > > > >> > > > > > > > hit to always loop through everything in cases
> where
> > > the
> > > > > >> > > > > schemas/values
> > > > > >> > > > > > > > might be quite large.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > One thing I wanted to ask: instead of creating a
> new
> > > > "path
> > > > > >> > > parser"
> > > > > >> > > > > > > > including some bespoke or "borrowed" syntax, why
> not
> > > > just
> > > > > >> use
> > > > > >> > > > > something
> > > > > >> > > > > > > > that already exists? Specifically here I am
> thinking
> > > > about
> > > > > >> > > > JsonPath (
> > > > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > There is already quite nice support in JsonPath
> for
> > > > > handling
> > > > > >> > > > special
> > > > > >> > > > > > > > characters in field names, for handling different
> > > > > >> non-primitive
> > > > > >> > > > types
> > > > > >> > > > > > > > (arrays etc), for handling multiple levels of
> > nesting,
> > > > etc
> > > > > >> etc.
> > > > > >> > > > > Would
> > > > > >> > > > > > it
> > > > > >> > > > > > > > be possible to instead to re-think this and maybe
> > have
> > > > > some
> > > > > >> > kind
> > > > > >> > > of
> > > > > >> > > > > > > > JsonPath-based Schema selector / updater and/or
> > > > > >> JsonPath-based
> > > > > >> > > > Value
> > > > > >> > > > > > > > selector / updater? Conceptually this feels like
> it
> > > > makes
> > > > > >> sense
> > > > > >> > > to
> > > > > >> > > > > me,
> > > > > >> > > > > > as
> > > > > >> > > > > > > > from the top of my head it would be quite a
> natural
> > > fit
> > > > to
> > > > > >> map
> > > > > >> > a
> > > > > >> > > > Json
> > > > > >> > > > > > > data
> > > > > >> > > > > > > > structure to the Connect API data structure (and
> you
> > > > could
> > > > > >> > > > > potentially
> > > > > >> > > > > > > even
> > > > > >> > > > > > > > try to leverage the existing Json-to-Connect
> > > > > >> > > > serializer/deserializer
> > > > > >> > > > > to
> > > > > >> > > > > > > > help out with this even in a more "out of the
> > > > box"-feeling
> > > > > >> kind
> > > > > >> > > of
> > > > > >> > > > > > way).
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Maybe also as Tom mentioned, this part (in my
> > example,
> > > > > this
> > > > > >> > > > > > > JsonPath-based
> > > > > >> > > > > > > > "thing") could even be a generic API that could be
> > > used
> > > > by
> > > > > >> any
> > > > > >> > > SMT,
> > > > > >> > > > > > > > including used in custom ones built by the
> > community.
> > > > > Then
> > > > > >> I
> > > > > >> > > think
> > > > > >> > > > > to
> > > > > >> > > > > > > use
> > > > > >> > > > > > > > a completely separate config property somehow
> > related
> > > to
> > > > > >> "path"
> > > > > >> > > (as
> > > > > >> > > > > Tom
> > > > > >> > > > > > > > also mentioned) would also make a lot of sense
> here
> > as
> > > > > well.
> > > > > >> > This
> > > > > >> > > > > way,
> > > > > >> > > > > > if
> > > > > >> > > > > > > > you select based on "path" then this
> JsonPath-based
> > > API
> > > > > >> would
> > > > > >> > be
> > > > > >> > > > > used,
> > > > > >> > > > > > > > otherwise it could use something similar to the
> > > existing
> > > > > >> > > get-field
> > > > > >> > > > > > based
> > > > > >> > > > > > > > approach (which I guess could also be refactored
> > into
> > > > some
> > > > > >> kind
> > > > > >> > > of
> > > > > >> > > > > > > utility
> > > > > >> > > > > > > > / API as well if it made sense?)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > And with that in mind, if this was the kind of
> > > direction
> > > > > to
> > > > > >> go,
> > > > > >> > > > then
> > > > > >> > > > > a
> > > > > >> > > > > > > > "recursive" capability like I pitched in KIP-683
> > would
> > > > > also
> > > > > >> > > become
> > > > > >> > > > > > > > unnecessary because you could easily write a
> > JsonPath
> > > > > >> > expression
> > > > > >> > > > like
> > > > > >> > > > > > > > "$..someRecuriveField" and it would do the same
> > thing
> > > > (on
> > > > > >> top
> > > > > >> > of
> > > > > >> > > > > > anything
> > > > > >> > > > > > > > else you would want to do that is already
> supported
> > by
> > > > > >> > JsonPath).
> > > > > >> > > > > Then
> > > > > >> > > > > > we
> > > > > >> > > > > > > > could also kill that older KIP and do a bit of
> > > clean-up
> > > > :)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > [1] -
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Just some extra food for thought. All-in-all I
> think
> > > > this
> > > > > >> is a
> > > > > >> > > > super
> > > > > >> > > > > > > great
> > > > > >> > > > > > > > initiative!
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Best,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Joshua
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris Egerton
> <
> > > > > >> > > > > > > fearthecellos@gmail.com
> > > > > >> > > > > > > > >:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Tom,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks for taking a look at this, and for your
> > > > > thoughtful
> > > > > >> > > > comments.
> > > > > >> > > > > > > I'll
> > > > > >> > > > > > > > > leave it up to Jorge to address most of your
> > > comments
> > > > > but
> > > > > >> I
> > > > > >> > > > wanted
> > > > > >> > > > > to
> > > > > >> > > > > > > > share
> > > > > >> > > > > > > > > a couple quick thoughts I had regarding 103 and
> > 104.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 103. Like you, I was envisioning a possible
> syntax
> > > for
> > > > > >> array
> > > > > >> > > > access
> > > > > >> > > > > > > that
> > > > > >> > > > > > > > > used classic C-style brackets; e.g.,
> `arr[index]`.
> > > > > >> However, I
> > > > > >> > > > > wonder
> > > > > >> > > > > > if
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > could keep things simple and use the same syntax
> > > that
> > > > > >> we're
> > > > > >> > > > > proposing
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > nested field access? In other words, instead of
> > > > > >> `arr[index]`,
> > > > > >> > > > you'd
> > > > > >> > > > > > > write
> > > > > >> > > > > > > > > `arr.index`. It'd save us and users the headache
> > of
> > > > > >> reserving
> > > > > >> > > > > > > characters
> > > > > >> > > > > > > > > now that would need to be escaped even if their
> > > > > unescaped
> > > > > >> > > > brethren
> > > > > >> > > > > > > aren't
> > > > > >> > > > > > > > > used for anything, and also avoid the question
> of
> > > what
> > > > > >> > exactly
> > > > > >> > > we
> > > > > >> > > > > > > should
> > > > > >> > > > > > > > do
> > > > > >> > > > > > > > > when we see a config that uses reserved
> characters
> > > > that
> > > > > >> > aren't
> > > > > >> > > > yet
> > > > > >> > > > > > > > > supported (throwing an exception seems pretty
> > > > unfriendly
> > > > > >> for
> > > > > >> > > new
> > > > > >> > > > > > > users).
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 104. This would probably be useful, but it would
> > > come
> > > > > with
> > > > > >> > some
> > > > > >> > > > > nasty
> > > > > >> > > > > > > > > compatibility questions that would need to be
> > > > addressed
> > > > > if
> > > > > >> > we'd
> > > > > >> > > > > want
> > > > > >> > > > > > > SMTs
> > > > > >> > > > > > > > > that leverage this new API to be viable for
> older
> > > > > >> versions of
> > > > > >> > > > > > Connect.
> > > > > >> > > > > > > If
> > > > > >> > > > > > > > > we package and distribute this feature as a
> > library
> > > > > >> (either
> > > > > >> > via
> > > > > >> > > > an
> > > > > >> > > > > > > > entirely
> > > > > >> > > > > > > > > new artifact, or as part of the existing
> > > > > >> connect-transforms
> > > > > >> > or
> > > > > >> > > > > > > > connect-api
> > > > > >> > > > > > > > > artifacts), then we'd have to either sidestep
> the
> > > > > existing
> > > > > >> > > plugin
> > > > > >> > > > > > > > isolation
> > > > > >> > > > > > > > > logic [1] that basically makes it impossible for
> > > > Connect
> > > > > >> > > plugins
> > > > > >> > > > to
> > > > > >> > > > > > > ship
> > > > > >> > > > > > > > > their own versions of Connect artifacts, or
> issue
> > a
> > > > big
> > > > > >> > warning
> > > > > >> > > > to
> > > > > >> > > > > > > people
> > > > > >> > > > > > > > > that any SMT that uses this API won't work with
> > any
> > > > > older
> > > > > >> > > > versions
> > > > > >> > > > > of
> > > > > >> > > > > > > > > Connect. There's also some other features we
> might
> > > > want
> > > > > to
> > > > > >> > add
> > > > > >> > > in
> > > > > >> > > > > an
> > > > > >> > > > > > > > > SMT-utils library such as the existing,
> internal,
> > > > utils
> > > > > >> that
> > > > > >> > > > > Connect
> > > > > >> > > > > > > uses
> > > > > >> > > > > > > > > right now [2]. It may be worth exploring this
> in a
> > > > > >> separate
> > > > > >> > KIP
> > > > > >> > > > of
> > > > > >> > > > > > its
> > > > > >> > > > > > > > own.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > [1] -
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > [2] -
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Cheers,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Chris
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom Bentley <
> > > > > >> > > tbentley@redhat.com
> > > > > >> > > > >
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks for the KIP, especially for the
> examples
> > > > which
> > > > > >> are
> > > > > >> > > > > > > super-clear.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 100. The name `field.style` isn't so clear for
> > > > > something
> > > > > >> > like
> > > > > >> > > > > > > > > ReplaceField:
> > > > > >> > > > > > > > > > it's not so obvious that field.style applies
> to
> > > > > >> `include`
> > > > > >> > and
> > > > > >> > > > > > > > `exclude`.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 101. The permitted values for `field.style`
> > don't
> > > > seem
> > > > > >> > > terribly
> > > > > >> > > > > > > > intuitive
> > > > > >> > > > > > > > > > (to me anyway): the meaning of `plain` isn't
> > very
> > > > > >> > guessable.
> > > > > >> > > > Why
> > > > > >> > > > > > not
> > > > > >> > > > > > > > > > `top-level` or `root` instead? Also `nested`
> > could
> > > > be
> > > > > >> > > > > misconstrued
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > mean
> > > > > >> > > > > > > > > > nested-but-not-top-level, so perhaps
> `recursive`
> > > or
> > > > > >> > > `cascading`
> > > > > >> > > > > > might
> > > > > >> > > > > > > > be
> > > > > >> > > > > > > > > > better?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 102. I'm torn on whether making the
> > interpretation
> > > > of
> > > > > >> > > existing
> > > > > >> > > > > > > configs
> > > > > >> > > > > > > > > like
> > > > > >> > > > > > > > > > `field` be dependent on `field.style` is a
> good
> > > > idea.
> > > > > I
> > > > > >> can
> > > > > >> > > see
> > > > > >> > > > > > that
> > > > > >> > > > > > > > it's
> > > > > >> > > > > > > > > > the simplest thing to do, but it just feels a
> > bit
> > > > odd
> > > > > >> that
> > > > > >> > > > > > sometimes
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > `field` would actually be a path and have
> > > different
> > > > > >> > escaping
> > > > > >> > > > > rules.
> > > > > >> > > > > > > An
> > > > > >> > > > > > > > > > alternative would be to come up with a
> parallel
> > > set
> > > > of
> > > > > >> > config
> > > > > >> > > > > names
> > > > > >> > > > > > > > (e.g.
> > > > > >> > > > > > > > > > as well as "field" an SMT might support
> "path")
> > > > which
> > > > > >> were
> > > > > >> > > > > defined
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > > always take paths, thus avoiding the
> changeable
> > > > > >> > > interpretation
> > > > > >> > > > of
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > > existing configs. The SMT's #configure() would
> > > need
> > > > to
> > > > > >> > throw
> > > > > >> > > in
> > > > > >> > > > > the
> > > > > >> > > > > > > > case
> > > > > >> > > > > > > > > > that both configs were given. I can see that
> > that
> > > > > would
> > > > > >> be
> > > > > >> > > more
> > > > > >> > > > > > work
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 103. I think in order to allow for supporting
> > > arrays
> > > > > in
> > > > > >> a
> > > > > >> > > later
> > > > > >> > > > > KIP
> > > > > >> > > > > > > > > (which
> > > > > >> > > > > > > > > > certainly seems like it could be useful), we'd
> > > want
> > > > to
> > > > > >> > > specify
> > > > > >> > > > > the
> > > > > >> > > > > > > > syntax
> > > > > >> > > > > > > > > > now, even if it wasn't implemented under this
> > KIP.
> > > > > >> That's
> > > > > >> > > > > because I
> > > > > >> > > > > > > > don't
> > > > > >> > > > > > > > > > think you can't exclude the possibility that
> > > > > characters
> > > > > >> > such
> > > > > >> > > as
> > > > > >> > > > > `[`
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > `]`
> > > > > >> > > > > > > > > > appear in field names. So you'd have a
> > > compatibility
> > > > > >> > problem
> > > > > >> > > if
> > > > > >> > > > > > > people
> > > > > >> > > > > > > > > > started using the features of this KIP to
> access
> > > > such
> > > > > >> > fields,
> > > > > >> > > > > only
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > > those characters to change their meaning
> under a
> > > > later
> > > > > >> KIP.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 104. I also wonder whether making paths into a
> > > > public
> > > > > >> Java
> > > > > >> > > API,
> > > > > >> > > > > for
> > > > > >> > > > > > > use
> > > > > >> > > > > > > > > by
> > > > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks again,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Tom
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris Egerton <
> > > > > >> > > > > > fearthecellos@gmail.com
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge Esteban
> > > Quilcate
> > > > > >> Otoya
> > > > > >> > <
> > > > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Thank you, Chris! Not possible without
> your
> > > > > >> feedback.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04, Chris
> Egerton
> > <
> > > > > >> > > > > > > > fearthecellos@gmail.com
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thank you for sticking through this. I
> > have
> > > > one
> > > > > >> small
> > > > > >> > > > > remark
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > one
> > > > > >> > > > > > > > > > > > small
> > > > > >> > > > > > > > > > > > > clarification; assuming you agree with
> me
> > on
> > > > > them
> > > > > >> > then
> > > > > >> > > > I'm
> > > > > >> > > > > > > ready
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > vote
> > > > > >> > > > > > > > > > > > on
> > > > > >> > > > > > > > > > > > > the KIP.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > 1. InsertField: The
> > > "field.on.missing.parent"
> > > > > and
> > > > > >> > > > > > > > > > > > "field.on.existing.field"
> > > > > >> > > > > > > > > > > > > docs both mention a permitted value of
> > > > "ingore";
> > > > > >> this
> > > > > >> > > > > should
> > > > > >> > > > > > be
> > > > > >> > > > > > > > > > > "ignore",
> > > > > >> > > > > > > > > > > > > right?
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Of course, one more typo :)
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > 2. InsertField: The examples are still
> > > missing
> > > > > the
> > > > > >> > > > > > > "field.style"
> > > > > >> > > > > > > > > > > property
> > > > > >> > > > > > > > > > > > > from the configurations. They should all
> > > > include
> > > > > >> the
> > > > > >> > > > > property
> > > > > >> > > > > > > > > > > > > "transforms.smt1.field.style": "nested",
> > > > > correct?
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Yes, it is there. I think I know what you
> > mean
> > > > > now,
> > > > > >> > seems
> > > > > >> > > > > that
> > > > > >> > > > > > > > > > Confluence
> > > > > >> > > > > > > > > > > > is putting everything in one line when
> it's
> > in
> > > > > >> separate
> > > > > >> > > > lines
> > > > > >> > > > > > in
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > editor.
> > > > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks again for working through this,
> and
> > > > > >> > > > congratulations
> > > > > >> > > > > > on a
> > > > > >> > > > > > > > > > > > > well-written KIP!
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Chris
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM Jorge
> > > Esteban
> > > > > >> > Quilcate
> > > > > >> > > > > Otoya
> > > > > >> > > > > > <
> > > > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Thank you, Chris! I apply these
> > > improvements
> > > > > to
> > > > > >> the
> > > > > >> > > > KIP,
> > > > > >> > > > > > let
> > > > > >> > > > > > > me
> > > > > >> > > > > > > > > > know
> > > > > >> > > > > > > > > > > > how
> > > > > >> > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > looks now.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43, Chris
> > > Egerton
> > > > <
> > > > > >> > > > > > > > > > fearthecellos@gmail.com
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Wow, those examples are great! A few
> > > more
> > > > > >> > remarks,
> > > > > >> > > > but
> > > > > >> > > > > I
> > > > > >> > > > > > > > think
> > > > > >> > > > > > > > > > > we're
> > > > > >> > > > > > > > > > > > > > > getting close:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > 1. The examples differ across SMTs
> > with
> > > > the
> > > > > >> name
> > > > > >> > of
> > > > > >> > > > the
> > > > > >> > > > > > > > > > > > > newly-introduced
> > > > > >> > > > > > > > > > > > > > > style property; some of them use
> > > > > >> "field.style",
> > > > > >> > and
> > > > > >> > > > > some
> > > > > >> > > > > > > use
> > > > > >> > > > > > > > > > > > > > > "fields.style". I think for
> > > consistency's
> > > > > >> sake we
> > > > > >> > > > > should
> > > > > >> > > > > > > > stick
> > > > > >> > > > > > > > > > with
> > > > > >> > > > > > > > > > > > > just
> > > > > >> > > > > > > > > > > > > > > "field.style"; otherwise it could be
> > > > painful
> > > > > >> for
> > > > > >> > > > users
> > > > > >> > > > > to
> > > > > >> > > > > > > > have
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > remember
> > > > > >> > > > > > > > > > > > > > > which to use.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed the config
> > > names
> > > > > to
> > > > > >> > > > > > > `field.style`.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > 2. Some of the examples are off:
> > > > > >> > > > > > > > > > > > > > > - TimestampConverter: the input in
> the
> > > > > second
> > > > > >> > > example
> > > > > >> > > > > > > ("when
> > > > > >> > > > > > > > > > field
> > > > > >> > > > > > > > > > > > > names
> > > > > >> > > > > > > > > > > > > > > include dots") doesn't contain a
> field
> > > > with
> > > > > a
> > > > > >> > > dotted
> > > > > >> > > > > name
> > > > > >> > > > > > > > > > > > > > > - ValueToKey: the config in the
> third
> > > > > example
> > > > > >> > > ("when
> > > > > >> > > > > > field
> > > > > >> > > > > > > > > names
> > > > > >> > > > > > > > > > > > > include
> > > > > >> > > > > > > > > > > > > > > dots") should probably use
> > > > > "parent..child.k2"
> > > > > >> as
> > > > > >> > > the
> > > > > >> > > > > > > > > > > > > > > "transforms.smt1.fields" property
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > 3. RE changes to InsertField:
> > > > > >> > > > > > > > > > > > > > > - The InsertField SMT should also
> come
> > > > with
> > > > > >> the
> > > > > >> > new
> > > > > >> > > > > > > > > "field.style"
> > > > > >> > > > > > > > > > > > > > property
> > > > > >> > > > > > > > > > > > > > > in order to preserve backwards
> > > > > compatibility,
> > > > > >> > > right?
> > > > > >> > > > I
> > > > > >> > > > > > > don't
> > > > > >> > > > > > > > > see
> > > > > >> > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > included in the example configs for
> > that
> > > > > one,
> > > > > >> > just
> > > > > >> > > > want
> > > > > >> > > > > > to
> > > > > >> > > > > > > > make
> > > > > >> > > > > > > > > > > sure
> > > > > >> > > > > > > > > > > > > > > - I don't know of any cases where we
> > use
> > > > > >> > snake_case
> > > > > >> > > > for
> > > > > >> > > > > > > > > property
> > > > > >> > > > > > > > > > > > names
> > > > > >> > > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > > Kafka; we should probably use
> > > > > >> "on.missing.parent"
> > > > > >> > > and
> > > > > >> > > > > > > > > > > > > "on.existing.field"
> > > > > >> > > > > > > > > > > > > > > as the new property names for
> > > InsertField.
> > > > > >> > > > > > > > > > > > > > > - Why is the "on_existing_field" (or
> > > > > >> > > > > "on.existing.field")
> > > > > >> > > > > > > > > > property
> > > > > >> > > > > > > > > > > > only
> > > > > >> > > > > > > > > > > > > > > applied when the field style is
> > nested?
> > > > > >> Couldn't
> > > > > >> > it
> > > > > >> > > > be
> > > > > >> > > > > > > useful
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Great points! I have applied these
> > > > suggestions
> > > > > >> to
> > > > > >> > the
> > > > > >> > > > > KIP.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Chris
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40 PM
> Jorge
> > > > > Esteban
> > > > > >> > > > Quilcate
> > > > > >> > > > > > > Otoya
> > > > > >> > > > > > > > <
> > > > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Again, great feedback Chris. Much
> > > > > >> appreciated.
> > > > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at 20:22, Chris
> > > > > Egerton <
> > > > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Looking good! I have a few
> > comments
> > > > left
> > > > > >> but
> > > > > >> > > all
> > > > > >> > > > > but
> > > > > >> > > > > > > one
> > > > > >> > > > > > > > or
> > > > > >> > > > > > > > > > two
> > > > > >> > > > > > > > > > > > are
> > > > > >> > > > > > > > > > > > > > > > minor.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 1. The motivation section states
> > > "This
> > > > > >> KIP is
> > > > > >> > > > aimed
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > > include
> > > > > >> > > > > > > > > > > > > > support
> > > > > >> > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > nested structures on the
> existing
> > > > > SMTs...
> > > > > >> and
> > > > > >> > > to
> > > > > >> > > > > > > include
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > abstractions
> > > > > >> > > > > > > > > > > > > > > > > to reuse this in future SMTs". A
> > > good
> > > > > >> > > > > implementation
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > this
> > > > > >> > > > > > > > > > > KIP
> > > > > >> > > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > > > definitely isolate reusable
> logic
> > > > into a
> > > > > >> > > separate
> > > > > >> > > > > > > > > abstraction
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > can
> > > > > >> > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > > easily pulled in to the SMTs we
> > want
> > > > to
> > > > > >> add
> > > > > >> > > > nested
> > > > > >> > > > > > > field
> > > > > >> > > > > > > > > > > support
> > > > > >> > > > > > > > > > > > > to,
> > > > > >> > > > > > > > > > > > > > > but
> > > > > >> > > > > > > > > > > > > > > > > unless we plan on making this
> kind
> > > of
> > > > > >> > > abstraction
> > > > > >> > > > > > > > publicly
> > > > > >> > > > > > > > > > > > > available
> > > > > >> > > > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > > > > > > some kind of utility method or
> > class
> > > > > that
> > > > > >> > > > external
> > > > > >> > > > > > SMT
> > > > > >> > > > > > > > > > > developers
> > > > > >> > > > > > > > > > > > > can
> > > > > >> > > > > > > > > > > > > > > > > leverage, we can probably leave
> > this
> > > > > part
> > > > > >> out
> > > > > >> > > as
> > > > > >> > > > > it's
> > > > > >> > > > > > > > more
> > > > > >> > > > > > > > > of
> > > > > >> > > > > > > > > > > an
> > > > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Make sense, will leave this out of
> > the
> > > > > KIP.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a little
> > > > > >> misleading,
> > > > > >> > > isn't
> > > > > >> > > > > it?
> > > > > >> > > > > > > It
> > > > > >> > > > > > > > > > > > > demonstrates
> > > > > >> > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > escape syntax for fields with
> dot
> > > > > >> literals in
> > > > > >> > > > their
> > > > > >> > > > > > > > names,
> > > > > >> > > > > > > > > > but
> > > > > >> > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > doesn't
> > > > > >> > > > > > > > > > > > > > > > > demonstrate a way to actually
> use
> > > the
> > > > > Cast
> > > > > >> > (or
> > > > > >> > > > any
> > > > > >> > > > > > > other)
> > > > > >> > > > > > > > > SMT
> > > > > >> > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > access a
> > > > > >> > > > > > > > > > > > > > > > > nested field in a record, which
> is
> > > the
> > > > > >> whole
> > > > > >> > > > point
> > > > > >> > > > > of
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > KIP.
> > > > > >> > > > > > > > > > > I
> > > > > >> > > > > > > > > > > > > like
> > > > > >> > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > example of escape syntax but we
> > > should
> > > > > >> > probably
> > > > > >> > > > > also
> > > > > >> > > > > > > add
> > > > > >> > > > > > > > > one
> > > > > >> > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > > > field access.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Agree. I have added examples to
> each
> > > SMT
> > > > > to
> > > > > >> be
> > > > > >> > > more
> > > > > >> > > > > > clear
> > > > > >> > > > > > > > > about
> > > > > >> > > > > > > > > > > how
> > > > > >> > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > > affects each function
> > > > > >> > > > > > > > > > > > > > > > .
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 3. With the InsertField SMT, I'm
> > > > > wondering
> > > > > >> > what
> > > > > >> > > > the
> > > > > >> > > > > > > > > specific
> > > > > >> > > > > > > > > > > > > behavior
> > > > > >> > > > > > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > > > be when some or all of the
> "middle
> > > > > layer"
> > > > > >> > > nested
> > > > > >> > > > > > fields
> > > > > >> > > > > > > > are
> > > > > >> > > > > > > > > > > > > missing.
> > > > > >> > > > > > > > > > > > > > > For
> > > > > >> > > > > > > > > > > > > > > > > example, if I have a record
> with a
> > > > value
> > > > > >> of {
> > > > > >> > > > "k1":
> > > > > >> > > > > > > "v1 }
> > > > > >> > > > > > > > > > and I
> > > > > >> > > > > > > > > > > > > apply
> > > > > >> > > > > > > > > > > > > > > > > InsertField with topic.field =
> > > > > >> > n1.n2.n3.topic,
> > > > > >> > > > what
> > > > > >> > > > > > > will
> > > > > >> > > > > > > > > > > happen?
> > > > > >> > > > > > > > > > > > > Will
> > > > > >> > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > updated value become { "k1":
> "v1",
> > > > > "n1": {
> > > > > >> > > "n2":
> > > > > >> > > > {
> > > > > >> > > > > > > "n3":
> > > > > >> > > > > > > > > > > "topic"
> > > > > >> > > > > > > > > > > > > }}},
> > > > > >> > > > > > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > > > an exception be thrown, or
> > something
> > > > > else?
> > > > > >> > This
> > > > > >> > > > > seems
> > > > > >> > > > > > > > > > analogous
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > command line mkdir command,
> which
> > > (at
> > > > > >> least
> > > > > >> > on
> > > > > >> > > > some
> > > > > >> > > > > > > > > operating
> > > > > >> > > > > > > > > > > > > > systems)
> > > > > >> > > > > > > > > > > > > > > > > fails by default if you try to
> > > create
> > > > a
> > > > > >> new
> > > > > >> > > > nested
> > > > > >> > > > > > > > > directory
> > > > > >> > > > > > > > > > > > where
> > > > > >> > > > > > > > > > > > > > > > anything
> > > > > >> > > > > > > > > > > > > > > > > but the last element in the path
> > > > doesn't
> > > > > >> > exist,
> > > > > >> > > > but
> > > > > >> > > > > > can
> > > > > >> > > > > > > > be
> > > > > >> > > > > > > > > > > > invoked
> > > > > >> > > > > > > > > > > > > > > with a
> > > > > >> > > > > > > > > > > > > > > > > flag that instructs it to go
> ahead
> > > and
> > > > > >> create
> > > > > >> > > all
> > > > > >> > > > > > > levels
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > > > directory instead. I'm leaning
> on
> > > the
> > > > > >> side of
> > > > > >> > > > "just
> > > > > >> > > > > > > > create
> > > > > >> > > > > > > > > > > > > > everything"
> > > > > >> > > > > > > > > > > > > > > > but
> > > > > >> > > > > > > > > > > > > > > > > would be interested in your
> > > thoughts,
> > > > > and
> > > > > >> > > either
> > > > > >> > > > > way,
> > > > > >> > > > > > > we
> > > > > >> > > > > > > > > > should
> > > > > >> > > > > > > > > > > > > > > probably
> > > > > >> > > > > > > > > > > > > > > > > make sure the intended behavior
> is
> > > > > >> > well-defined
> > > > > >> > > > > > before
> > > > > >> > > > > > > > > > voting.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > This is an interesting case,
> thanks
> > > for
> > > > > >> > catching
> > > > > >> > > > > this!
> > > > > >> > > > > > > > > > > > > > > > The default behavior I see is to
> > > create
> > > > > >> parents
> > > > > >> > > if
> > > > > >> > > > > they
> > > > > >> > > > > > > are
> > > > > >> > > > > > > > > > > missing
> > > > > >> > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > overwrite fields
> > > > > >> > > > > > > > > > > > > > > > if they already exist.
> > > > > >> > > > > > > > > > > > > > > > I'm planning to include the
> > following
> > > > two
> > > > > >> flags
> > > > > >> > > if
> > > > > >> > > > > > there
> > > > > >> > > > > > > > is a
> > > > > >> > > > > > > > > > > need
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> > > (CREATE|IGNORE),
> > > > > >> > > > > default=CREATE
> > > > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > > > (OVERWRITE|IGNORE),
> > > > > >> > > > > > > > default=OVERWRITE
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will the
> > behavior
> > > > be
> > > > > if
> > > > > >> > any
> > > > > >> > > of
> > > > > >> > > > > the
> > > > > >> > > > > > > > field
> > > > > >> > > > > > > > > > > > > elements
> > > > > >> > > > > > > > > > > > > > > > > specified with InsertField
> already
> > > > exist
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > record
> > > > > >> > > > > > > > > value?
> > > > > >> > > > > > > > > > > > Will
> > > > > >> > > > > > > > > > > > > we
> > > > > >> > > > > > > > > > > > > > > > just
> > > > > >> > > > > > > > > > > > > > > > > overwrite them? What's the
> > behavior
> > > of
> > > > > >> > > > InsertField
> > > > > >> > > > > > > today
> > > > > >> > > > > > > > > > under
> > > > > >> > > > > > > > > > > > > > similar
> > > > > >> > > > > > > > > > > > > > > > > circumstances?
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > The current behavior is to
> overwrite
> > > the
> > > > > >> value.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Chris
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at 4:15 PM
> > > Jorge
> > > > > >> Esteban
> > > > > >> > > > > > Quilcate
> > > > > >> > > > > > > > > Otoya
> > > > > >> > > > > > > > > > <
> > > > > >> > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> wrote:
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much
> appreciated
> > > all
> > > > > the
> > > > > >> > > > feedback
> > > > > >> > > > > > > here.
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > 1. You nailed it setting the
> > > design
> > > > > goal
> > > > > >> > > here:
> > > > > >> > > > > "it
> > > > > >> > > > > > > > > > shouldn't
> > > > > >> > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > > impossible
> > > > > >> > > > > > > > > > > > > > > > > > to use this new feature for
> any
> > > > field
> > > > > >> name,
> > > > > >> > > no
> > > > > >> > > > > > matter
> > > > > >> > > > > > > > how
> > > > > >> > > > > > > > > > > > > > convoluted.
> > > > > >> > > > > > > > > > > > > > > > > It's
> > > > > >> > > > > > > > > > > > > > > > > > fine if edge cases introduce
> > > > > difficulty
> > > > > >> > (such
> > > > > >> > > > as
> > > > > >> > > > > > > > > > > less-readable
> > > > > >> > > > > > > > > > > > > > > > > > configurations), but it's not
> > fine
> > > > if
> > > > > >> they
> > > > > >> > > > can't
> > > > > >> > > > > be
> > > > > >> > > > > > > > > > addressed
> > > > > >> > > > > > > > > > > > at
> > > > > >> > > > > > > > > > > > > > > all."
> > > > > >> > > > > > > > > > > > > > > > > > Back to the previous proposals
> > > > (using
> > > > > >> only
> > > > > >> > > dots
> > > > > >> > > > > as
> > > > > >> > > > > > > > > > > separators)
> > > > > >> > > > > > > > > > > > we
> > > > > >> > > > > > > > > > > > > > > have
> > > > > >> > > > > > > > > > > > > > > > 2
> > > > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > > > >> > > > > > > > > > > > > > > > > > 1. escaping with backslashes
> > > > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots itself
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > I'll lean for alternative 2,
> as
> > > you
> > > > > >> > proposed
> > > > > >> > > > > > before.
> > > > > >> > > > > > > > > Feels
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > me
> > > > > >> > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > backslashes have more
> potential
> > to
> > > > > lead
> > > > > >> to
> > > > > >> > > > > > confusion
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > > JSON
> > > > > >> > > > > > > > > > > > > > configs,
> > > > > >> > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > CSV seems like a good
> precedent
> > to
> > > > use
> > > > > >> the
> > > > > >> > > same
> > > > > >> > > > > > > > character
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > escape
> > > > > >> > > > > > > > > > > > > > > > > itself.
> > > > > >> > > > > > > > > > > > > > > > > > KIP is updated to reflect
> this.
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an
> example,
> > > and
> > > > > >> stick
> > > > > >> > > with
> > > > > >> > > > > the
> > > > > >> > > > > > > > > current
> > > > > >> > > > > > > > > > > > > approach
> > > > > >> > > > > > > > > > > > > > > > > > defining the style per
> > individual
> > > > > >> transform
> > > > > >> > > > > > > > > configuration.
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP updated.
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP updated.
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at 21:59,
> > > Chris
> > > > > >> > Egerton <
> > > > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > Thanks for addressing my
> > > comments;
> > > > > the
> > > > > >> > KIP
> > > > > >> > > > > looks
> > > > > >> > > > > > > > > > up-to-date
> > > > > >> > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > pretty
> > > > > >> > > > > > > > > > > > > > > > > > > readable now, and the
> rejected
> > > > > >> > alternatives
> > > > > >> > > > > > section
> > > > > >> > > > > > > > > does
> > > > > >> > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > great
> > > > > >> > > > > > > > > > > > > > > job
> > > > > >> > > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > outlining the discussion so
> > far
> > > > and
> > > > > >> > > providing
> > > > > >> > > > > > > context
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > > > > anyone
> > > > > >> > > > > > > > > > > > > > > else
> > > > > >> > > > > > > > > > > > > > > > > who
> > > > > >> > > > > > > > > > > > > > > > > > > might want to join in.
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on choice of
> > > > delimiter:
> > > > > >> > > > > > > > > > > > > > > > > > > - I like the optimization
> for
> > > > simple
> > > > > >> > cases,
> > > > > >> > > > > but I
> > > > > >> > > > > > > > think
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > new
> > > > > >> > > > > > > > > > > > > > > > > proposal
> > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > > > > a little too restrictive.
> What
> > > if
> > > > > >> > there's a
> > > > > >> > > > > field
> > > > > >> > > > > > > > whose
> > > > > >> > > > > > > > > > > name
> > > > > >> > > > > > > > > > > > > > > contains
> > > > > >> > > > > > > > > > > > > > > > > all
> > > > > >> > > > > > > > > > > > > > > > > > > of the permitted options
> > > > (currently
> > > > > >> just
> > > > > >> > > ".",
> > > > > >> > > > > > ",",
> > > > > >> > > > > > > > and
> > > > > >> > > > > > > > > > > "/")?
> > > > > >> > > > > > > > > > > > > > > > > > > - If we expand the set of
> > > > permitted
> > > > > >> > > > delimiters
> > > > > >> > > > > to
> > > > > >> > > > > > > > allow
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > any
> > > > > >> > > > > > > > > > > > > > > > > > > single-character string,
> > > > > configuration
> > > > > >> > > > > complexity
> > > > > >> > > > > > > > will
> > > > > >> > > > > > > > > > > > increase
> > > > > >> > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > readability may decrease
> > > > > >> > > > > > > > > > > > > > > > > > > - Also worth pointing out
> that
> > > > there
> > > > > >> is
> > > > > >> > > some
> > > > > >> > > > > > > > convention
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > doubling
> > > > > >> > > > > > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > > > > delimiter character as an
> > escape
> > > > > >> > mechanism
> > > > > >> > > > with
> > > > > >> > > > > > > > formats
> > > > > >> > > > > > > > > > > like
> > > > > >> > > > > > > > > > > > > CSV
> > > > > >> > > > > > > > > > > > > > > [1]
> > > > > >> > > > > > > > > > > > > > > > > > > - Overall I think we may be
> > > > > >> approaching
> > > > > >> > the
> > > > > >> > > > > > > > saturation
> > > > > >> > > > > > > > > > > point
> > > > > >> > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > productive
> > > > > >> > > > > > > > > > > > > > > > > > > discussion on delimiter
> syntax
> > > so
> > > > I
> > > > > >> don't
> > > > > >> > > > want
> > > > > >> > > > > to
> > > > > >> > > > > > > > spend
> > > > > >> > > > > > > > > > too
> > > > > >> > > > > > > > > > > > > much
> > > > > >> > > > > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > your time on it. I think the
> > one
> > > > > point
> > > > > >> > I'd
> > > > > >> > > > like
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > leave
> > > > > >> > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > now
> > > > > >> > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > > > > > shouldn't be impossible to
> use
> > > > this
> > > > > >> new
> > > > > >> > > > feature
> > > > > >> > > > > > for
> > > > > >> > > > > > > > any
> > > > > >> > > > > > > > > > > field
> > > > > >> > > > > > > > > > > > > > name,
> > > > > >> > > > > > > > > > > > > > > > no
> > > > > >> > > > > > > > > > > > > > > > > > > matter how convoluted. It's
> > fine
> > > > if
> > > > > >> edge
> > > > > >> > > > cases
> > > > > >> > > > > > > > > introduce
> > > > > >> > > > > > > > > > > > > > difficulty
> > > > > >> > > > > > > > > > > > > > > > > (such
> > > > > >> > > > > > > > > > > > > > > > > > > as less-readable
> > > configurations),
> > > > > but
> > > > > >> > it's
> > > > > >> > > > not
> > > > > >> > > > > > fine
> > > > > >> > > > > > > > if
> > > > > >> > > > > > > > > > they
> > > > > >> > > > > > > > > > > > > can't
> > > > > >> > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > 2.
> > > > > >> > > > > > > > > > > > > > > > > > > The configuration style
> where
> > > you
> > > > > >> define
> > > > > >> > > > > > > > > > > > > "transforms.field.style"
> > > > > >> > > > > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > connector config, and then
> > this
> > > > > >> applies
> > > > > >> > to
> > > > > >> > > > all
> > > > > >> > > > > > SMTs
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > connector,
> > > > > >> > > > > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > > > > very interesting. However,
> it
> > > > > doesn't
> > > > > >> > > follow
> > > > > >> > > > > > > > convention
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > existing
> > > > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > > > >> > > > > > > > > > > > > > > > > > > Right now, if you want to
> > > > configure
> > > > > an
> > > > > >> > SMT,
> > > > > >> > > > you
> > > > > >> > > > > > > > define
> > > > > >> > > > > > > > > > its
> > > > > >> > > > > > > > > > > > name
> > > > > >> > > > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > connector config (for
> example,
> > > > > >> > > "transforms":
> > > > > >> > > > > > > "smt1"),
> > > > > >> > > > > > > > > and
> > > > > >> > > > > > > > > > > > then
> > > > > >> > > > > > > > > > > > > > > define
> > > > > >> > > > > > > > > > > > > > > > > all
> > > > > >> > > > > > > > > > > > > > > > > > > of the properties for that
> SMT
> > > in
> > > > > the
> > > > > >> > > > connector
> > > > > >> > > > > > > > config
> > > > > >> > > > > > > > > > > using
> > > > > >> > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > > > namespacing
> > > > > >> > > > > > > > > > > > > > > > > > > mechanism specific to that
> SMT
> > > > (for
> > > > > >> > > example,
> > > > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT then sees
> > only
> > > > the
> > > > > >> > > > properties
> > > > > >> > > > > > > > defined
> > > > > >> > > > > > > > > > in
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > namespace,
> > > > > >> > > > > > > > > > > > > > > > > > > with the prefix stripped
> (for
> > > > > example,
> > > > > >> > > > "prop1":
> > > > > >> > > > > > > > "val1")
> > > > > >> > > > > > > > > > in
> > > > > >> > > > > > > > > > > > its
> > > > > >> > > > > > > > > > > > > > > > > configure
> > > > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > > > >> > > > > > > > > > > > > > > > > > > If we want to continue to
> > follow
> > > > > this
> > > > > >> > > > > convention,
> > > > > >> > > > > > > > then
> > > > > >> > > > > > > > > > > > instead
> > > > > >> > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > specifying
> > > > "transforms.field.style"
> > > > > >> in a
> > > > > >> > > > > > connector
> > > > > >> > > > > > > > > > config,
> > > > > >> > > > > > > > > > > we
> > > > > >> > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > > > expect
> > > > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > > > >> > > > > > "transforms.<name>.field.style",
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > each
> > > > > >> > > > > > > > > > > > > SMT
> > > > > >> > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > they
> > > > > >> > > > > > > > > > > > > > > > > > > want to configure a field
> > style
> > > > for.
> > > > > >> This
> > > > > >> > > > would
> > > > > >> > > > > > > > require
> > > > > >> > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > work
> > > > > >> > > > > > > > > > > > > > > on
> > > > > >> > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > part of the user, but would
> be
> > > > > >> simpler to
> > > > > >> > > > > reason
> > > > > >> > > > > > > > about
> > > > > >> > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > easier
> > > > > >> > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > implement.
> > > > > >> > > > > > > > > > > > > > > > > > > If we want to explore an
> > > > alternative
> > > > > >> > where
> > > > > >> > > > > users
> > > > > >> > > > > > > can
> > > > > >> > > > > > > > > > > specify
> > > > > >> > > > > > > > > > > > > > global
> > > > > >> > > > > > > > > > > > > > > > > > > properties that apply to all
> > > > > >> transforms
> > > > > >> > in
> > > > > >> > > a
> > > > > >> > > > > > > > connector
> > > > > >> > > > > > > > > > > > config,
> > > > > >> > > > > > > > > > > > > > then
> > > > > >> > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > semantics for this need to
> be
> > > > > defined
> > > > > >> in
> > > > > >> > > the
> > > > > >> > > > > KIP.
> > > > > >> > > > > > > > This
> > > > > >> > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > have
> > > > > >> > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > include whether this will
> > apply
> > > > only
> > > > > >> for
> > > > > >> > > the
> > > > > >> > > > > > > special
> > > > > >> > > > > > > > > case
> > > > > >> > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > "field.style" and possibly
> > > > > >> > > "field.separator"
> > > > > >> > > > > > > > properties
> > > > > >> > > > > > > > > > or
> > > > > >> > > > > > > > > > > if
> > > > > >> > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > > > > available more generally for
> > > other
> > > > > >> > > > properties,
> > > > > >> > > > > > > > whether
> > > > > >> > > > > > > > > it
> > > > > >> > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > apply
> > > > > >> > > > > > > > > > > > > > > > > only
> > > > > >> > > > > > > > > > > > > > > > > > > for the SMTs outlined in the
> > KIP
> > > > or
> > > > > if
> > > > > >> > the
> > > > > >> > > > > > > > > "field.style"
> > > > > >> > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > possibly
> > > > > >> > > > > > > > > > > > > > > > > > > "field.separator" properties
> > > would
> > > > > >> also
> > > > > >> > be
> > > > > >> > > > > passed
> > > > > >> > > > > > > > into
> > > > > >> > > > > > > > > > > custom
> > > > > >> > > > > > > > > > > > > > SMTs
> > > > > >> > > > > > > > > > > > > > > so
> > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > they could choose to act on
> > them
> > > > if
> > > > > >> > > > applicable,
> > > > > >> > > > > > how
> > > > > >> > > > > > > > > edge
> > > > > >> > > > > > > > > > > > cases
> > > > > >> > > > > > > > > > > > > > like
> > > > > >> > > > > > > > > > > > > > > > > > having
> > > > > >> > > > > > > > > > > > > > > > > > > an SMT named "field" in your
> > > > > connector
> > > > > >> > > config
> > > > > >> > > > > > would
> > > > > >> > > > > > > > be
> > > > > >> > > > > > > > > > > > handled,
> > > > > >> > > > > > > > > > > > > > > etc.
> > > > > >> > > > > > > > > > > > > > > > > > > Either way, it might help to
> > > have
> > > > an
> > > > > >> > > example
> > > > > >> > > > in
> > > > > >> > > > > > the
> > > > > >> > > > > > > > KIP
> > > > > >> > > > > > > > > > > > > outlining
> > > > > >> > > > > > > > > > > > > > > how
> > > > > >> > > > > > > > > > > > > > > > > one
> > > > > >> > > > > > > > > > > > > > > > > > > of the to-be-augmented SMTs
> > can
> > > be
> > > > > >> > > configured
> > > > > >> > > > > > with
> > > > > >> > > > > > > > this
> > > > > >> > > > > > > > > > new
> > > > > >> > > > > > > > > > > > > > feature
> > > > > >> > > > > > > > > > > > > > > > > and a
> > > > > >> > > > > > > > > > > > > > > > > > > before/after of how a record
> > > value
> > > > > >> would
> > > > > >> > be
> > > > > >> > > > > > > > transformed
> > > > > >> > > > > > > > > > > with
> > > > > >> > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for the
> > > > > >> > > > > "transforms.field.style"
> > > > > >> > > > > > > > > > property
> > > > > >> > > > > > > > > > > > > > mentions
> > > > > >> > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > the permitted values are
> > "plain"
> > > > and
> > > > > >> > > > "nested",
> > > > > >> > > > > > but
> > > > > >> > > > > > > > then
> > > > > >> > > > > > > > > > > > > describes
> > > > > >> > > > > > > > > > > > > > > > > > behavior
> > > > > >> > > > > > > > > > > > > > > > > > > with a value of "root".
> Should
> > > > that
> > > > > be
> > > > > >> > > > "plain"
> > > > > >> > > > > > > > instead?
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for the
> > > > > >> > > > > > > "transforms.field.separator"
> > > > > >> > > > > > > > > > > > property
> > > > > >> > > > > > > > > > > > > > > > > > exclusively
> > > > > >> > > > > > > > > > > > > > > > > > > mentions structs, but the
> > > feature
> > > > is
> > > > > >> > > intended
> > > > > >> > > > > to
> > > > > >> > > > > > > work
> > > > > >> > > > > > > > > > with
> > > > > >> > > > > > > > > > > > maps
> > > > > >> > > > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > > > > > > well.
> > > > > >> > > > > > > > > > > > > > > > > > > Can we update it to reflect
> > > this?
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > References:
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > > > >> > https://stackoverflow.com/a/17808731
> > > > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > Chris
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022 at 1:32
> > PM
> > > > > Jorge
> > > > > >> > > Esteban
> > > > > >> > > > > > > > Quilcate
> > > > > >> > > > > > > > > > > Otoya
> > > > > >> > > > > > > > > > > > <
> > > > > >> > > > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > wrote:
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> > > "this..field.child"
> > > > > >> could
> > > > > >> > be
> > > > > >> > > > > > harder
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > grasp
> > > > > >> > > > > > > > > > > > > than
> > > > > >> > > > > > > > > > > > > > > > > > > > "this.field/child" +
> > > separator:
> > > > > "/".
> > > > > >> > > > > > > > > > > > > > > > > > > > Even though this
> represents
> > > > > >> additional
> > > > > >> > > > > > > information,
> > > > > >> > > > > > > > > it
> > > > > >> > > > > > > > > > > > > follows
> > > > > >> > > > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > > > similar
> > > > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > > > "Flatten#delimeter"
> > > > > >> > > > > > > configuration.
> > > > > >> > > > > > > > > > > > > > > > > > > > I want to give the
> separator
> > > > > >> approach
> > > > > >> > > > another
> > > > > >> > > > > > > try,
> > > > > >> > > > > > > > > so I
> > > > > >> > > > > > > > > > > > have
> > > > > >> > > > > > > > > > > > > > > > updated
> > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > KIP with the separator
> > > proposal,
> > > > > >> > sticking
> > > > > >> > > > to
> > > > > >> > > > > > > only 2
> > > > > >> > > > > > > > > > > > > > alternatives
> > > > > >> > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > > should hopefully cover
> most
> > > > > >> scenarios.
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has been
> > updated
> > > > > with
> > > > > >> > this
> > > > > >> > > > > > > > improvement.
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I have
> > > updated
> > > > > this
> > > > > >> > > > section
> > > > > >> > > > > > > > > > accordingly.
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch! I've
> replaced
> > > it
> > > > > with
> > > > > >> > > > > > > `DropHeaders`.
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > Looking forward to your
> > > > feedback.
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022 at
> 21:33,
> > > > Chris
> > > > > >> > > Egerton
> > > > > >> > > > <
> > > > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > Looking good! Got a few
> > more
> > > > > >> > thoughts.
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to revisit
> this,
> > > but
> > > > I
> > > > > >> think
> > > > > >> > > we
> > > > > >> > > > > may
> > > > > >> > > > > > > want
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > adopt
> > > > > >> > > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > > slightly
> > > > > >> > > > > > > > > > > > > > > > > > > > > different escape syntax
> > > style.
> > > > > >> > > > Backslashes
> > > > > >> > > > > > are
> > > > > >> > > > > > > > > great,
> > > > > >> > > > > > > > > > > but
> > > > > >> > > > > > > > > > > > > > since
> > > > > >> > > > > > > > > > > > > > > > > > they're
> > > > > >> > > > > > > > > > > > > > > > > > > > > already used by JSON,
> > using
> > > > them
> > > > > >> as
> > > > > >> > an
> > > > > >> > > > > escape
> > > > > >> > > > > > > > > > sequence
> > > > > >> > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > field
> > > > > >> > > > > > > > > > > > > > > > > > > notation
> > > > > >> > > > > > > > > > > > > > > > > > > > > would also lead to some
> > > pretty
> > > > > >> ugly
> > > > > >> > > > > connector
> > > > > >> > > > > > > > > > configs.
> > > > > >> > > > > > > > > > > > > Anyone
> > > > > >> > > > > > > > > > > > > > > > who's
> > > > > >> > > > > > > > > > > > > > > > > > had
> > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > write regular
> expressions
> > > with
> > > > > >> > > > backslashes
> > > > > >> > > > > in
> > > > > >> > > > > > > > Java
> > > > > >> > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > probably
> > > > > >> > > > > > > > > > > > > > > > > > already
> > > > > >> > > > > > > > > > > > > > > > > > > > > familiar with this:
> > > > > >> > > > > > > > > > > > > > >
> > > > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > > > >> > > > > > > > > > > > > > > > > What
> > > > > >> > > > > > > > > > > > > > > > > > > do
> > > > > >> > > > > > > > > > > > > > > > > > > > > you think about using
> the
> > > dot
> > > > > >> > character
> > > > > >> > > > to
> > > > > >> > > > > > > escape
> > > > > >> > > > > > > > > > > itself?
> > > > > >> > > > > > > > > > > > > In
> > > > > >> > > > > > > > > > > > > > > > other
> > > > > >> > > > > > > > > > > > > > > > > > > words,
> > > > > >> > > > > > > > > > > > > > > > > > > > > to access a single field
> > > named
> > > > > >> > > > > "this.field",
> > > > > >> > > > > > > > > instead
> > > > > >> > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > using
> > > > > >> > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > syntax
> > > > > >> > > > > > > > > > > > > > > > > > > > > "this\.field" (which in
> > JSON
> > > > > would
> > > > > >> > have
> > > > > >> > > > to
> > > > > >> > > > > be
> > > > > >> > > > > > > > > > expressed
> > > > > >> > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> > "this..field",
> > > > and
> > > > > >> for a
> > > > > >> > > > > single
> > > > > >> > > > > > > > field
> > > > > >> > > > > > > > > > > named
> > > > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > > > >> > > > > > > > > > > > > > > > > > > > > instead of using the
> > syntax
> > > > > >> > > "this\\field"
> > > > > >> > > > > > (or,
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > > JSON,
> > > > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> "this\field"
> > > (or,
> > > > > in
> > > > > >> > JSON,
> > > > > >> > > > > > > > > > "this\\field").
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you flesh out
> the
> > > > > >> details on
> > > > > >> > > the
> > > > > >> > > > > new
> > > > > >> > > > > > > > > > > > "field.style"
> > > > > >> > > > > > > > > > > > > > > > > property,
> > > > > >> > > > > > > > > > > > > > > > > > > > > including the type,
> > default
> > > > > value,
> > > > > >> > > > > > importance,
> > > > > >> > > > > > > > and
> > > > > >> > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > preliminary
> > > > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > > > >> > > > > > > > > > > > > > > > > > > > > See
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the
> "Compatibility,
> > > > > >> > Deprecation,
> > > > > >> > > > and
> > > > > >> > > > > > > > > Migration
> > > > > >> > > > > > > > > > > > Plan"
> > > > > >> > > > > > > > > > > > > > > > section
> > > > > >> > > > > > > > > > > > > > > > > > > still
> > > > > >> > > > > > > > > > > > > > > > > > > > > accurate after the
> latest
> > > > > update?
> > > > > >> > Seems
> > > > > >> > > > > like
> > > > > >> > > > > > > it's
> > > > > >> > > > > > > > > > still
> > > > > >> > > > > > > > > > > > > > written
> > > > > >> > > > > > > > > > > > > > > > > with
> > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > assumption that nested
> > field
> > > > > >> syntax
> > > > > >> > > will
> > > > > >> > > > be
> > > > > >> > > > > > > > > hardcoded
> > > > > >> > > > > > > > > > > or
> > > > > >> > > > > > > > > > > > > > > opt-in,
> > > > > >> > > > > > > > > > > > > > > > > > which
> > > > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > > > >> > > > > > > > > > > > > > > > > > > > > isn't the case anymore.
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The "These SMTs
> do
> > > not
> > > > > >> > require
> > > > > >> > > > > nested
> > > > > >> > > > > > > > > > structure
> > > > > >> > > > > > > > > > > > > > > support"
> > > > > >> > > > > > > > > > > > > > > > > > > section
> > > > > >> > > > > > > > > > > > > > > > > > > > > mentions a "Drop" SMT. I
> > > think
> > > > > >> this
> > > > > >> > may
> > > > > >> > > > be
> > > > > >> > > > > > > > > referring
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > Confluent
> > > > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > > > >> > > > > > > > > > > > > > > > > > > > > SMT, which isn't a part
> of
> > > > > Apache
> > > > > >> > > Kafka.
> > > > > >> > > > > > Should
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > > drop
> > > > > >> > > > > > > > > > > > > (heh)
> > > > > >> > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > SMT
> > > > > >> > > > > > > > > > > > > > > > > > > > from
> > > > > >> > > > > > > > > > > > > > > > > > > > > the list? Or perhaps
> just
> > > > > replace
> > > > > >> it
> > > > > >> > > with
> > > > > >> > > > > > > > > > > "DropHeaders",
> > > > > >> > > > > > > > > > > > > > which
> > > > > >> > > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > > > > > currently
> > > > > >> > > > > > > > > > > > > > > > > > > > > missing from the list
> and
> > > > > >> shouldn't
> > > > > >> > > > require
> > > > > >> > > > > > any
> > > > > >> > > > > > > > > > > > > nested-field
> > > > > >> > > > > > > > > > > > > > > > > related
> > > > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28, 2022 at
> > 2:12
> > > > PM
> > > > > >> Jorge
> > > > > >> > > > > Esteban
> > > > > >> > > > > > > > > > Quilcate
> > > > > >> > > > > > > > > > > > > Otoya
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> quilcate.jorge@gmail.com>
> > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Thank you, Chris! and
> > > sorry
> > > > > for
> > > > > >> the
> > > > > >> > > > > delayed
> > > > > >> > > > > > > > > > response.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Please, find my
> comments
> > > > > below:
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb 2022 at
> > > > 17:34,
> > > > > >> Chris
> > > > > >> > > > > Egerton
> > > > > >> > > > > > > > > > > > > > > > > > > >
> <chrise@confluent.io.invalid
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP!
> > I'd
> > > > love
> > > > > >> to
> > > > > >> > see
> > > > > >> > > > > > support
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > fields
> > > > > >> > > > > > > > > > > > > > > > > > added
> > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > out-of-the-box SMTs
> > > > provided
> > > > > >> with
> > > > > >> > > > > > Connect.
> > > > > >> > > > > > > > Here
> > > > > >> > > > > > > > > > are
> > > > > >> > > > > > > > > > > > my
> > > > > >> > > > > > > > > > > > > > > > initial
> > > > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree that
> > there's
> > > a
> > > > > >> case to
> > > > > >> > > be
> > > > > >> > > > > made
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > > > > expanding
> > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > new config property
> > for
> > > > > >> > > identifying a
> > > > > >> > > > > > > nested,
> > > > > >> > > > > > > > > > > > > > to-be-hoisted
> > > > > >> > > > > > > > > > > > > > > > > > field,
> > > > > >> > > > > > > > > > > > > > > > > > > > but
> > > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > example in the KIP
> > > doesn't
> > > > > >> really
> > > > > >> > > > > > > demonstrate
> > > > > >> > > > > > > > > why
> > > > > >> > > > > > > > > > > > this
> > > > > >> > > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd be
> helpful
> > to
> > > > > >> expand
> > > > > >> > the
> > > > > >> > > > > > example
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > add
> > > > > >> > > > > > > > > > > > > other
> > > > > >> > > > > > > > > > > > > > > > fields
> > > > > >> > > > > > > > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > > > > > > > > order
> > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > show how adding
> nested
> > > > field
> > > > > >> > > support
> > > > > >> > > > > > > enables
> > > > > >> > > > > > > > > > users
> > > > > >> > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > hoist a
> > > > > >> > > > > > > > > > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > without dropping
> other
> > > > > fields
> > > > > >> > from
> > > > > >> > > > the
> > > > > >> > > > > > > value.
> > > > > >> > > > > > > > > > Maybe
> > > > > >> > > > > > > > > > > > > > > something
> > > > > >> > > > > > > > > > > > > > > > > > like
> > > > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >     source =
> > nested.val
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >     field = line
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value (before):
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "nested":
> > {
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "val":
> > > 42,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "other
> > > > val":
> > > > > >> 96
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >     value (after):
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "nested":
> > {
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "line":
> > > {
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > >  "val":
> > > > > 42,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >                 }
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "other
> > > > val":
> > > > > >> 96
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I think
> > "source"
> > > > is
> > > > > a
> > > > > >> > > little
> > > > > >> > > > > > > strange
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > new
> > > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > property name. Maybe
> > > > > >> "hoisted" or
> > > > > >> > > > > > > > > "hoisted.field"
> > > > > >> > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and 2.:
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The example for
> > > this
> > > > > SMT
> > > > > >> is
> > > > > >> > > > > updated
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > have
> > > > > >> > > > > > > > > > > > added
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there a
> > reasonable
> > > > use
> > > > > >> case
> > > > > >> > > for
> > > > > >> > > > > > > > expanding
> > > > > >> > > > > > > > > > > > Flatten
> > > > > >> > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten specific
> > fields?
> > > > My
> > > > > >> > > > > understanding
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > that
> > > > > >> > > > > > > > > > > > it's
> > > > > >> > > > > > > > > > > > > > > mostly
> > > > > >> > > > > > > > > > > > > > > > > > > useful
> > > > > >> > > > > > > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > writing to systems
> > like
> > > > > >> databases
> > > > > >> > > > that
> > > > > >> > > > > > > don't
> > > > > >> > > > > > > > > > > support
> > > > > >> > > > > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > > > > values
> > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > require everything
> to
> > > be a
> > > > > >> flat
> > > > > >> > > list
> > > > > >> > > > of
> > > > > >> > > > > > > > > key-value
> > > > > >> > > > > > > > > > > > > pairs.
> > > > > >> > > > > > > > > > > > > > > > Being
> > > > > >> > > > > > > > > > > > > > > > > > able
> > > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a nested
> field
> > > > > >> wouldn't
> > > > > >> > > > provide
> > > > > >> > > > > > any
> > > > > >> > > > > > > > > > > advantage
> > > > > >> > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > use
> > > > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > Are there other
> cases
> > > > where
> > > > > it
> > > > > >> > > would?
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't think we
> > > should
> > > > > >> > > > > > unconditionally
> > > > > >> > > > > > > > > change
> > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > default
> > > > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten. It's a
> > > > > >> > > > backwards-incompatible,
> > > > > >> > > > > > > > > breaking
> > > > > >> > > > > > > > > > > > change
> > > > > >> > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > could
> > > > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > headaches for users.
> > It
> > > > > might
> > > > > >> be
> > > > > >> > > > > > reasonable
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > change
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > default
> > > > > >> > > > > > > > > > > > > > > > > > > > value
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically based on
> > > > whether
> > > > > >> the
> > > > > >> > > user
> > > > > >> > > > > has
> > > > > >> > > > > > > > > > > specified a
> > > > > >> > > > > > > > > > > > > > value
> > > > > >> > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > property, but
> > > considering
> > > > > the
> > > > > >> > > > > motivation
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > > > changing
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > default
> > > > > >> > > > > > > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > it creates conflicts
> > > with
> > > > > the
> > > > > >> > > > > > > > to-be-introduced
> > > > > >> > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > field
> > > > > >> > > > > > > > > > > > > > > > > > syntax
> > > > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > could arise with
> > > > downstream
> > > > > >> SMTs
> > > > > >> > > > > > regardless
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > > > > > whether
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > user
> > > > > >> > > > > > > > > > > > > > > > > > has
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly
> configured
> > > > > Flatten
> > > > > >> > with
> > > > > >> > > > the
> > > > > >> > > > > > > > "field"
> > > > > >> > > > > > > > > > > > > > property), I
> > > > > >> > > > > > > > > > > > > > > > > don't
> > > > > >> > > > > > > > > > > > > > > > > > > > know
> > > > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > this would be too
> > useful
> > > > > >> either.
> > > > > >> > I
> > > > > >> > > > have
> > > > > >> > > > > > > some
> > > > > >> > > > > > > > > > > thoughts
> > > > > >> > > > > > > > > > > > > > below
> > > > > >> > > > > > > > > > > > > > > > on
> > > > > >> > > > > > > > > > > > > > > > > > how
> > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > handle possible
> > > conflicts
> > > > > >> between
> > > > > >> > > > names
> > > > > >> > > > > > > with
> > > > > >> > > > > > > > > dots
> > > > > >> > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > their
> > > > > >> > > > > > > > > > > > > > > > > names
> > > > > >> > > > > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for nested
> > field
> > > > > >> > references
> > > > > >> > > > that
> > > > > >> > > > > > > > should
> > > > > >> > > > > > > > > > > > > hopefully
> > > > > >> > > > > > > > > > > > > > > make
> > > > > >> > > > > > > > > > > > > > > > > > > either
> > > > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough. With the
> > > > support
> > > > > >> for
> > > > > >> > > > nested
> > > > > >> > > > > > > fields
> > > > > >> > > > > > > > > in
> > > > > >> > > > > > > > > > > > other
> > > > > >> > > > > > > > > > > > > > > SMTs,
> > > > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > > > >> > > > > > > > > > > > > > > > > > > > > > could stay as it is.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > This removes the need
> > for
> > > > (4)
> > > > > >> > > changing
> > > > > >> > > > > > > Flatten
> > > > > >> > > > > > > > > > config
> > > > > >> > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > > > > well.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think it's fine
> > to
> > > > > expand
> > > > > >> > > > > > ExtractField
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > > support
> > > > > >> > > > > > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > it might be worth
> > noting
> > > > in
> > > > > >> the
> > > > > >> > > > > rejected
> > > > > >> > > > > > > > > > > alternatives
> > > > > >> > > > > > > > > > > > > > > section
> > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > isn't strictly
> > necessary
> > > > > since
> > > > > >> > you
> > > > > >> > > > can
> > > > > >> > > > > > > > replace
> > > > > >> > > > > > > > > > any
> > > > > >> > > > > > > > > > > > > single
> > > > > >> > > > > > > > > > > > > > > > > > > invocation
> > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT that uses
> > > nested
> > > > > >> field
> > > > > >> > > > > notation
> > > > > >> > > > > > > with
> > > > > >> > > > > > > > > > > > multiple
> > > > > >> > > > > > > > > > > > > > > > > > invocations
> > > > > >> > > > > > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > that use non-nested
> > > > > notation.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding it.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit:
> "RegerRouter"
> > > > should
> > > > > >> be
> > > > > >> > > > > > > "RegexRouter"
> > > > > >> > > > > > > > > in
> > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > list
> > > > > >> > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > SMTs
> > > > > >> > > > > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > > > do
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > not require nested
> > > > structure
> > > > > >> > > support.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing it.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be rare
> for
> > > dots
> > > > > in
> > > > > >> > field
> > > > > >> > > > > names
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > occur
> > > > > >> > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > wild
> > > > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be so
> certain
> > > of
> > > > > >> this),
> > > > > >> > > but
> > > > > >> > > > > > unless
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > > want
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > inflict
> > > > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > users of Flatten, I
> > > think
> > > > > >> we're
> > > > > >> > > going
> > > > > >> > > > > to
> > > > > >> > > > > > > have
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > think
> > > > > >> > > > > > > > > > > > > > > about
> > > > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > between dotted
> > notation
> > > > and
> > > > > >> > > > non-nested
> > > > > >> > > > > > > fields
> > > > > >> > > > > > > > > > whose
> > > > > >> > > > > > > > > > > > > names
> > > > > >> > > > > > > > > > > > > > > > > contain
> > > > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > don't think this is
> > > > actually
> > > > > >> > such a
> > > > > >> > > > bad
> > > > > >> > > > > > > > thing,
> > > > > >> > > > > > > > > > > > though.
> > > > > >> > > > > > > > > > > > > I
> > > > > >> > > > > > > > > > > > > > > > agree
> > > > > >> > > > > > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > notation is
> intuitive
> > > and
> > > > > >> pretty
> > > > > >> > > > > > > commonplace
> > > > > >> > > > > > > > > (in
> > > > > >> > > > > > > > > > > > tools
> > > > > >> > > > > > > > > > > > > > like
> > > > > >> > > > > > > > > > > > > > > > jq,
> > > > > >> > > > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > example), so I'd
> like
> > it
> > > > if
> > > > > we
> > > > > >> > > could
> > > > > >> > > > > > stick
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > it.
> > > > > >> > > > > > > > > > > > What
> > > > > >> > > > > > > > > > > > > > > about
> > > > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > escape syntax,
> using a
> > > > > >> backslash?
> > > > > >> > > > That
> > > > > >> > > > > > way,
> > > > > >> > > > > > > > > users
> > > > > >> > > > > > > > > > > > could
> > > > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > between "this.field"
> > > > (which
> > > > > >> would
> > > > > >> > > > refer
> > > > > >> > > > > > to
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > field
> > > > > >> > > > > > > > > > > > > > > > > > "field"
> > > > > >> > > > > > > > > > > > > > > > > > > > > under
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > the top-level "this"
> > > > field),
> > > > > >> and
> > > > > >> > > > > > > > "this\.field"
> > > > > >> > > > > > > > > > > (which
> > > > > >> > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > > refer
> > > > > >> > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> > > "this.field").
> > > > > >> Like
> > > > > >> > > with
> > > > > >> > > > > most
> > > > > >> > > > > > > > > > languages
> > > > > >> > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > > use
> > > > > >> > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > for escape
> sequences,
> > it
> > > > > could
> > > > > >> > also
> > > > > >> > > > be
> > > > > >> > > > > > used
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > escape
> > > > > >> > > > > > > > > > > > > > > itself,
> > > > > >> > > > > > > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > that a field name
> > > > contains a
> > > > > >> > > > > backslash. I
> > > > > >> > > > > > > > think
> > > > > >> > > > > > > > > > > this
> > > > > >> > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > simpler than, e.g.,
> > > > adding a
> > > > > >> new
> > > > > >> > > > config
> > > > > >> > > > > > > > > property
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > toggle
> > > > > >> > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > be used when parsing
> > > > nested
> > > > > >> field
> > > > > >> > > > > > > references.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > I like this approach.
> > > Adding
> > > > > to
> > > > > >> the
> > > > > >> > > > KIP.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't think we
> > can
> > > > > >> > > > unconditionally
> > > > > >> > > > > > > turn
> > > > > >> > > > > > > > > this
> > > > > >> > > > > > > > > > > > > feature
> > > > > >> > > > > > > > > > > > > > > on.
> > > > > >> > > > > > > > > > > > > > > > > The
> > > > > >> > > > > > > > > > > > > > > > > > > > risk
> > > > > >> > > > > > > > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > breaking existing
> > > > pipelines
> > > > > >> > > > (especially
> > > > > >> > > > > > > ones
> > > > > >> > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > involve,
> > > > > >> > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > combination of the
> > > Flatten
> > > > > and
> > > > > >> > Cast
> > > > > >> > > > > SMTs)
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > > pretty
> > > > > >> > > > > > > > > > > > > > high. I
> > > > > >> > > > > > > > > > > > > > > > > think
> > > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > should be an opt-in
> > > > feature,
> > > > > >> at
> > > > > >> > > least
> > > > > >> > > > > > until
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > next
> > > > > >> > > > > > > > > > > > > > major
> > > > > >> > > > > > > > > > > > > > > > > > release.
> > > > > >> > > > > > > > > > > > > > > > > > > > One
> > > > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > we could accomplish
> > this
> > > > is
> > > > > by
> > > > > >> > > > > > introducing
> > > > > >> > > > > > > a
> > > > > >> > > > > > > > > new
> > > > > >> > > > > > > > > > > > > > > > "field.style"
> > > > > >> > > > > > > > > > > > > > > > > > > (name
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > obviously subject to
> > > > change)
> > > > > >> > > property
> > > > > >> > > > > > with
> > > > > >> > > > > > > > > values
> > > > > >> > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > "plain"
> > > > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If set to
> > > > "plain"
> > > > > >> then
> > > > > >> > > the
> > > > > >> > > > > > > current
> > > > > >> > > > > > > > > > > > non-nested
> > > > > >> > > > > > > > > > > > > > > > > behavior
> > > > > >> > > > > > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > and if set to
> > "nested",
> > > > then
> > > > > >> the
> > > > > >> > > > > proposed
> > > > > >> > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > behavior
> > > > > >> > > > > > > > > > > > > > > is.
> > > > > >> > > > > > > > > > > > > > > > > We
> > > > > >> > > > > > > > > > > > > > > > > > > can
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > consider updating
> the
> > > > > default
> > > > > >> > value
> > > > > >> > > > to
> > > > > >> > > > > > > > "nested"
> > > > > >> > > > > > > > > > in
> > > > > >> > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > future
> > > > > >> > > > > > > > > > > > > > > > > major
> > > > > >> > > > > > > > > > > > > > > > > > > > > release
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > (or even codify that
> > > plan
> > > > in
> > > > > >> this
> > > > > >> > > > KIP,
> > > > > >> > > > > if
> > > > > >> > > > > > > > > there's
> > > > > >> > > > > > > > > > > > > enough
> > > > > >> > > > > > > > > > > > > > > > > support
> > > > > >> > > > > > > > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > This would also
> leave
> > > the
> > > > > door
> > > > > >> > open
> > > > > >> > > > for
> > > > > >> > > > > > > > adding
> > > > > >> > > > > > > > > > > > > recursive
> > > > > >> > > > > > > > > > > > > > > > > support
> > > > > >> > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > the future by
> adding a
> > > > > >> permitted
> > > > > >> > > > value
> > > > > >> > > > > of
> > > > > >> > > > > > > > > > > > "recursive".
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of the linked
> > > > tickets
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > > > > "Motivation"
> > > > > >> > > > > > > > > > > > > section,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > https://issues.apache.org/jira/browse/KAFKA-10640
> > > ,
> > > > > >> > > > > > > > > > > > has
> > > > > >> > > > > > > > > > > > > > an
> > > > > >> > > > > > > > > > > > > > > > open
> > > > > >> > > > > > > > > > > > > > > > > > PR
> > > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > that propose adding
> > > > > recursive
> > > > > >> > > support
> > > > > >> > > > > to
> > > > > >> > > > > > > some
> > > > > >> > > > > > > > > > SMTs.
> > > > > >> > > > > > > > > > > > > Have
> > > > > >> > > > > > > > > > > > > > > you
> > > > > >> > > > > > > > > > > > > > > > > > > > considered
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> > > functionality
> > > > > for
> > > > > >> > your
> > > > > >> > > > > KIP?
> > > > > >> > > > > > Or
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > your
> > > > > >> > > > > > > > > > > > aim
> > > > > >> > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > stick
> > > > > >> > > > > > > > > > > > > > > > > > > > solely
> > > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > nested field
> support?
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > I like the
> `field.style`
> > > > > >> > > configuration
> > > > > >> > > > > flag
> > > > > >> > > > > > > > > > approach.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for pointing
> out
> > > the
> > > > > >> > > `recursive`
> > > > > >> > > > > > > > approach.
> > > > > >> > > > > > > > > > > Will
> > > > > >> > > > > > > > > > > > > add
> > > > > >> > > > > > > > > > > > > > > > > `nested`
> > > > > >> > > > > > > > > > > > > > > > > > > at
> > > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's check
> the
> > > > demand
> > > > > >> for
> > > > > >> > > > > > > `recursive`
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > > > consider
> > > > > >> > > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > > > > > > > part
> > > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > > > > > > > this or another KIP.
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > I have added the
> > following
> > > > on
> > > > > >> the
> > > > > >> > > KIP:
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs could
> extend
> > > > this
> > > > > >> > support
> > > > > >> > > > > for:
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive notation:
> > > name a
> > > > > >> field
> > > > > >> > > and
> > > > > >> > > > > > apply
> > > > > >> > > > > > > it
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > all
> > > > > >> > > > > > > > > > > > > > fields
> > > > > >> > > > > > > > > > > > > > > > > > across
> > > > > >> > > > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > schema matching that
> > name.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > - Access to arrays:
> > Adding
> > > > > `[]`
> > > > > >> > > > notation
> > > > > >> > > > > to
> > > > > >> > > > > > > > > > represent
> > > > > >> > > > > > > > > > > > > > access
> > > > > >> > > > > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > > > > > > arrays
> > > > > >> > > > > > > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs to
> fields
> > > > within
> > > > > >> an
> > > > > >> > > > array.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 8, 2022
> at
> > > > 1:23
> > > > > PM
> > > > > >> > > Jorge
> > > > > >> > > > > > > Esteban
> > > > > >> > > > > > > > > > > Quilcate
> > > > > >> > > > > > > > > > > > > > > Otoya <
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > quilcate.jorge@gmail.com>
> > > > > >> wrote:
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev team,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like to start
> a
> > > new
> > > > > >> > > discussion
> > > > > >> > > > > > thread
> > > > > >> > > > > > > > on
> > > > > >> > > > > > > > > > > Kafka
> > > > > >> > > > > > > > > > > > > > > Connect
> > > > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP is aimed
> to
> > > > > include
> > > > > >> > > > support
> > > > > >> > > > > > for
> > > > > >> > > > > > > > > nested
> > > > > >> > > > > > > > > > > > > > > structures
> > > > > >> > > > > > > > > > > > > > > > on
> > > > > >> > > > > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs — where this
> > make
> > > > > >> sense.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to
> > > your
> > > > > >> > feedback.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Jorge Esteban Quilcate Otoya <qu...@gmail.com>]

Thanks Robert! And sorry for the late reply.

That's a great catch and will require the current proposal to be updated.

Using back-ticks is a good proposal. Though, the challenge we got is that
JSON allows really any character into attribute names, including back-ticks.
I wonder what's the best approach to escape back-ticks then. Given that
those use-cases should be minimal we could fallback to backslash to escape
back-ticks?

What do you think?

Many thanks,
Jorge.


On Wed, 20 Jul 2022 at 23:03, Robert Yokota <ra...@gmail.com> wrote:

> Hi,
>
> I'm late to this thread, but would like to comment on the double dot/double
> asterisk syntax.
>
> Unfortunately double dot is often used in JSON Path as a descendant
> selector, see https://www.ietf.org/id/draft-ietf-jsonpath-base-05.html
>
> I think a better notation would be to use backticks to quote any names with
> a dot, such as  root.`child.with.dot`.  See Jsonata syntax for example:
> http://docs.jsonata.org/simple
>
> Robert
>
> On Wed, Jun 29, 2022 at 3:34 AM Jorge Esteban Quilcate Otoya <
> quilcate.jorge@gmail.com> wrote:
>
> > Thanks Chris! I have updated the KIP to include this fix.
> >
> > I will keep the array as a potential improvement at the moment, and out
> of
> > the scope of this KIP.
> >
> > Thanks,
> > Jorge.
> >
> > On Tue, 28 Jun 2022 at 23:19, Chris Egerton <fe...@gmail.com>
> > wrote:
> >
> > > Hi Jorge,
> > >
> > > Apologies for the long delay, had my own KIP-related work to focus on.
> > >
> > > I think it's fine to include array accesses but it's not a blocker. I'm
> > +1
> > > either way. On that front though, I think the MaskField section might
> > need
> > > to be updated as it still mentions arrays and deep-scan?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> > > quilcate.jorge@gmail.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > I have update the KIP to the previous state voted, including the
> > > > configuration change from `field.style` to `field.syntax.version`.
> > > >
> > > > I'll bump the vote thread as well to see if there's agreement on
> adding
> > > > this feature to Connect.
> > > >
> > > > Cheers,
> > > > Jorge.
> > > >
> > > > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > > > quilcate.jorge@gmail.com> wrote:
> > > >
> > > > > Thanks, Chris. Your feedback is much appreciated!
> > > > >
> > > > > I see how the current proposal might be underestimating some edge
> > > cases.
> > > > > I'm happy to move the design for deep-scan and multi-values to
> future
> > > > > developments related with this KIP and reduce its scope, though
> open
> > > for
> > > > > more feedback.
> > > > >
> > > > > Also, just to be sure, are you proposing also to not include array
> > > access
> > > > > at this stage?
> > > > >
> > > > > Thanks,
> > > > > Jorge.
> > > > >
> > > > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton <
> fearthecellos@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Hi Jorge,
> > > > >>
> > > > >> I've done some more thinking and I hate to say it, but I think the
> > > > syntax
> > > > >> does need to be expanded. Right now it's clear what "a.b" refers
> to
> > > and
> > > > >> what "a..b" refers to, but what about "a...b"? Is that referring
> to
> > > > >> subfield ".b" of field "a", or subfield "b" of field "a."? This
> gets
> > > > even
> > > > >> more complicated when thinking about fields whose names are
> > > exclusively
> > > > >> made up of dots.
> > > > >>
> > > > >> I'm also a little hesitant to mix the cases of multi-value paths
> and
> > > > deep
> > > > >> scans. What if you only want to access one subfield deep for an
> SMT,
> > > > >> instead of recursing through all the children of a given field?
> It's
> > > > akin
> > > > >> to the distinction between * and ** with file globbing patterns,
> and
> > > > there
> > > > >> could be a substantial performance difference if you have
> > > heavily-nested
> > > > >> fields.
> > > > >>
> > > > >> Ultimately, I think that if the proposed "field.syntax.version"
> > > property
> > > > >> sits well with people, it might be better to reduce the scope of
> the
> > > KIP
> > > > >> back to the original proposal and just focus on adding support for
> > > > >> explicitly-specified nested values, with no multi-value paths
> > > > whatsoever,
> > > > >> knowing that we have an easy way to introduce new syntax and
> > features
> > > in
> > > > >> the future. (We could probably leave the "a...b" case for that
> next
> > > > >> version
> > > > >> too.)
> > > > >>
> > > > >> I was a huge fan of this KIP before we started trying to address
> > more
> > > > >> complex use cases, and although I don't want to write those off, I
> > > think
> > > > >> we
> > > > >> may have bitten off more than we can chew in time for the 3.3.0
> > > release
> > > > >> and
> > > > >> would hate to see this KIP get delayed as a result.
> > > > >>
> > > > >> I'd be really curious to hear from Joshua and Tom on this front,
> > > though.
> > > > >> Is
> > > > >> it acceptable to move more incrementally here and settle on the
> > syntax
> > > > >> version property as our means of introducing new features, or is
> it
> > > > >> preferable to implement things monolithically and try to get
> > > everything
> > > > >> (or
> > > > >> at least, as much as possible) right the first time?
> > > > >>
> > > > >> Thanks again for your continued effort on this KIP!
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Chris
> > > > >>
> > > > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate Otoya <
> > > > >> quilcate.jorge@gmail.com> wrote:
> > > > >>
> > > > >> > Thanks, Chris!
> > > > >> >
> > > > >> > Please, find my comments below:
> > > > >> >
> > > > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton <
> > fearthecellos@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jorge,
> > > > >> > >
> > > > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > > > >> > >
> > > > >> > > 1. Under the "Accessing multiple values by deep-scan" header
> > it's
> > > > >> stated
> > > > >> > > that "If deep-scan is used, it must have only one field after
> > the
> > > > >> > asterisk
> > > > >> > > level.". However, in example 3 for the Cast SMT and other
> > examples
> > > > for
> > > > >> > > other SMTs, the spec contains a field of "*.child.k2", which
> > > appears
> > > > >> to
> > > > >> > > have two fields after the asterisk level. I may be
> > > misunderstanding
> > > > >> the
> > > > >> > > proposal, but it seems like the two contradict each other.
> > > > >> > >
> > > > >> >
> > > > >> > Thanks for catching this. I have clarified it by removing this
> > > > >> restriction.
> > > > >> > Also, have extended the deep-scan scenarios.
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > 2. I'm a little unclear on why we need the special handling
> for
> > > > arrays
> > > > >> > > where, for an array field "a", the field name "a" can be
> treated
> > > as
> > > > >> > either
> > > > >> > > the array itself, or every element in the array. Is there a
> > reason
> > > > we
> > > > >> > can't
> > > > >> > > use the field name "a.*" to handle the latter case, and "a" to
> > > > handle
> > > > >> the
> > > > >> > > former?
> > > > >> > >
> > > > >> >
> > > > >> > Agree, this is confusing. I like the `a.*` approach to access
> > array
> > > > >> items.
> > > > >> > I have added this to the proposal.
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > 3. How would a user specify that they'd like to access a field
> > > with
> > > > >> the
> > > > >> > > literal name "*"?
> > > > >> > >
> > > > >> >
> > > > >> > Good one. I'm proposing an approach similar to how it's proposed
> > to
> > > > >> escape
> > > > >> > dots, with a double-asterisk. Curious on your thoughts around
> > this.
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > 4. For the Cast SMT, do you think it might bite some people if
> > > > fields
> > > > >> > that
> > > > >> > > can't be cast correctly are silently ignored? I'm imagining
> the
> > > case
> > > > >> > where
> > > > >> > > none of the fields in a multi-path expression can be cast
> > > correctly
> > > > >> and
> > > > >> > it
> > > > >> > > ends up eating half of someone's day to track down why their
> SMT
> > > > isn't
> > > > >> > > doing anything.
> > > > >> > >
> > > > >> >
> > > > >> > If I understand correctly, this challenge could be relevant
> across
> > > > SMTs.
> > > > >> > At the moment, most/all? SMTs just silently ignore.
> > > > >> > Was thinking about adding a flag `field.on.path.not.found` to
> > either
> > > > >> ignore
> > > > >> > or fail when no paths are found. What do your think?
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > 5. For the ExtractField and ValueToKey SMTs, what happens if a
> > > > >> deep-scan
> > > > >> > > field name is used, but only one field is found? Is the
> > resulting
> > > > >> field
> > > > >> > > still an array, or is it just the single field that was found?
> > > (FWIW
> > > > >> I'm
> > > > >> > > leaning towards keeping it an array just to keep schemas
> > > consistent
> > > > >> in a
> > > > >> > > pipeline in case the number of fields found fluctuates across
> > > > >> records.)
> > > > >> > >
> > > > >> > > Agree. Will clarify that an array is always produced even for
> 1
> > > or 0
> > > > >> > fields are found.
> > > > >> >
> > > > >> >
> > > > >> > > 6. (Nit) For the HeaderFrom SMT, it's stated that "As this SMT
> > > > affects
> > > > >> > only
> > > > >> > > existing fields, additional configurations will not be
> > required.".
> > > > >> Given
> > > > >> > > the new "field.syntax.version" property, this part should
> > probably
> > > > be
> > > > >> > > removed.
> > > > >> > >
> > > > >> > Agree.
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > 7. Is recursive descent intentionally excluded? That was an
> > > > important
> > > > >> > part
> > > > >> > > of Joshua's KIP and his feedback on this KIP; I think it's
> worth
> > > > >> pursuing
> > > > >> > > if we can.
> > > > >> > >
> > > > >> >
> > > > >> > My understanding from Joshua's feedback is that by including
> > support
> > > > for
> > > > >> > deep-scan, we are already covering the recursive functionality.
> > > > Though,
> > > > >> I
> > > > >> > may be missing something.
> > > > >> >
> > > > >> > >
> > > > >> > >
> > > > >> > > Cheers,
> > > > >> > >
> > > > >> > > Chris
> > > > >> > >
> > > > >> > > On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate Otoya <
> > > > >> > > quilcate.jorge@gmail.com> wrote:
> > > > >> > >
> > > > >> > > > Thanks, Chris!
> > > > >> > > >
> > > > >> > > > I have updated the KIP with the rejected alternatives
> updated.
> > > > >> Also, I
> > > > >> > > have
> > > > >> > > > drafted the support for arrays and deep scans as part of the
> > > > >> proposed
> > > > >> > > > notation to make it more complete, even though these can be
> > > > >> implemented
> > > > >> > > in
> > > > >> > > > multiple PRs.
> > > > >> > > >
> > > > >> > > > Looking forward to your feedback.
> > > > >> > > >
> > > > >> > > > Cheers,
> > > > >> > > > Jorge.
> > > > >> > > >
> > > > >> > > > On Sat, 21 May 2022 at 17:39, Chris Egerton <
> > > > >> fearthecellos@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Jorge,
> > > > >> > > > >
> > > > >> > > > > I really appreciate the effort you've made to simplify the
> > > > syntax
> > > > >> and
> > > > >> > > > > feature set of a JSONPath-based approach as much as
> > possible.
> > > > I'm
> > > > >> > still
> > > > >> > > > > hesitant to continue with it, though.
> > > > >> > > > >
> > > > >> > > > > 1. The syntax is much less friendly. Just compare
> > > > >> "top.mid.bottom" to
> > > > >> > > > > "$['top']['mid']['bottom']"... not everyone uses JSONPath
> or
> > > > even
> > > > >> > JSON,
> > > > >> > > > and
> > > > >> > > > > the learning curve for the former is going to be steeper.
> > The
> > > > >> > examples
> > > > >> > > in
> > > > >> > > > > the new KIP draft you published demonstrate this pretty
> > well,
> > > > and
> > > > >> > this
> > > > >> > > is
> > > > >> > > > > without diving into the details of what escape syntax
> would
> > > look
> > > > >> > like.
> > > > >> > > > >
> > > > >> > > > > 2. The three new major features that this syntax adds
> (array
> > > > >> > accesses,
> > > > >> > > > deep
> > > > >> > > > > scans, and multi-value paths) could all be added pretty
> > easily
> > > > to
> > > > >> the
> > > > >> > > dot
> > > > >> > > > > notation syntax without introducing brackets and dollar
> > signs.
> > > > >> Array
> > > > >> > > > > accesses can be described using the same syntax as
> > struct/map
> > > > >> field
> > > > >> > > > access,
> > > > >> > > > > deep scans can be described using '*', and multi-value
> paths
> > > can
> > > > >> be
> > > > >> > > > > described by referencing the name of a field that's
> expected
> > > to
> > > > >> have
> > > > >> > > > > children. These are all top-of-the-head ideas and can
> > probably
> > > > be
> > > > >> > > > refined,
> > > > >> > > > > but hopefully they demonstrate that we can keep the syntax
> > > > simple
> > > > >> > > without
> > > > >> > > > > sacrificing features. Of course, the question of leaving
> > room
> > > > for
> > > > >> > > future
> > > > >> > > > > features might arise... given that these are the
> > > out-of-the-box
> > > > >> SMTs
> > > > >> > > that
> > > > >> > > > > are likely to be the first that many people encounter, I'd
> > err
> > > > on
> > > > >> the
> > > > >> > > > side
> > > > >> > > > > of simplicity and a gentle learning curve; if people need
> to
> > > get
> > > > >> more
> > > > >> > > out
> > > > >> > > > > of transforms, the option to implement their own is still
> > > there.
> > > > >> If
> > > > >> > we
> > > > >> > > > can
> > > > >> > > > > address 95% of use cases with something easy to use, it's
> > not
> > > > >> worth
> > > > >> > > > making
> > > > >> > > > > the feature harder for everyone to use just to accommodate
> > the
> > > > >> > > remaining
> > > > >> > > > > 5%.
> > > > >> > > > >
> > > > >> > > > > 3. The advantage of leveraging an existing syntax is
> > twofold:
> > > > >> users
> > > > >> > who
> > > > >> > > > are
> > > > >> > > > > already familiar with that syntax don't need to learn a
> new
> > > > >> syntax,
> > > > >> > and
> > > > >> > > > > maintainers of the syntax get to leverage existing
> libraries
> > > and
> > > > >> > > > > documentation for the syntax. With the current proposal,
> > > reusing
> > > > >> > > > libraries
> > > > >> > > > > is off the table, which means that we're going to have to
> > > parse
> > > > >> this
> > > > >> > > > > ourselves (including all the escape syntax edge cases),
> and
> > we
> > > > >> won't
> > > > >> > be
> > > > >> > > > > able to automatically leverage new features added to that
> > > > syntax.
> > > > >> And
> > > > >> > > > given
> > > > >> > > > > how stripped-down the syntax is in comparison to full
> > > JSONPath,
> > > > >> > there's
> > > > >> > > > > still going to be a learning curve for users who are
> already
> > > > >> familiar
> > > > >> > > > with
> > > > >> > > > > it, and we'll still have to document how Connect's variant
> > of
> > > > >> > JSONPath
> > > > >> > > > > works either instead of or in addition to just linking to
> a
> > > > >> > > > well-maintained
> > > > >> > > > > third-party docs site.
> > > > >> > > > >
> > > > >> > > > > On the topic of "field.path" and "include.path" vs.
> > > > >> "field.style", I
> > > > >> > > > > actually think that a single property per SMT is cleaner
> and
> > > > >> simpler.
> > > > >> > > > > Allowing users to mix and match styles within the same SMT
> > > > config
> > > > >> > seems
> > > > >> > > > > like a recipe for confusion, and with a single property
> that
> > > > >> dictates
> > > > >> > > > field
> > > > >> > > > > syntax behavior, we leave the door open to change the
> > default
> > > > at a
> > > > >> > > later
> > > > >> > > > > date. We could even fully remove support for plain field
> > > > notation
> > > > >> at
> > > > >> > > some
> > > > >> > > > > point and still be able to retain the simple property
> names
> > of
> > > > >> > "field",
> > > > >> > > > > "include", etc. instead of forcing people to use
> > "field.path"
> > > > and
> > > > >> the
> > > > >> > > > like.
> > > > >> > > > > That said, the term "field.style" and the permitted values
> > > might
> > > > >> be a
> > > > >> > > > > little ambiguous. One alternative, though a little
> > > heavy-handed,
> > > > >> is
> > > > >> > to
> > > > >> > > > > change it to "field.syntax.version" with permitted values
> of
> > > > "V1"
> > > > >> > > > (default,
> > > > >> > > > > equivalent to "field.style = plain") and "V2" (equivalent
> to
> > > > >> > > > "field.style =
> > > > >> > > > > nested"). This would leave us room in the future to make
> > > further
> > > > >> > > changes
> > > > >> > > > to
> > > > >> > > > > the syntax without having to come up with new names,
> > although
> > > it
> > > > >> does
> > > > >> > > > > sacrifice a little bit in that the permitted values are no
> > > > longer
> > > > >> > > > > self-describing.
> > > > >> > > > >
> > > > >> > > > > Cheers,
> > > > >> > > > >
> > > > >> > > > > Chris
> > > > >> > > > >
> > > > >> > > > > On Sun, May 15, 2022 at 4:51 PM Jorge Esteban Quilcate
> > Otoya <
> > > > >> > > > > quilcate.jorge@gmail.com> wrote:
> > > > >> > > > >
> > > > >> > > > > > Thank you all for your feedback, and sorry for the long
> > wait
> > > > >> for a
> > > > >> > > > reply.
> > > > >> > > > > >
> > > > >> > > > > > I would like to explore the idea of
> > JSONPath-inspired/subset
> > > > >> > > notation a
> > > > >> > > > > bit
> > > > >> > > > > > further:
> > > > >> > > > > >
> > > > >> > > > > > It will need to be a much-reduced version of JSONPath:
> > > > >> > > > > > - No full support for JsonPath therefore an additional
> > > > >> dependency.
> > > > >> > > > > > - All paths must start with '$'
> > > > >> > > > > > - No functions support or other operators allowed.
> > > > >> > > > > > - JsonPath dotted and square-bracket notations can be
> > > > supported
> > > > >> to
> > > > >> > > > avoid
> > > > >> > > > > > escaping dots or other characters: `$a.b.c` and
> > > > `$['a.b']['c']`
> > > > >> -
> > > > >> > or
> > > > >> > > we
> > > > >> > > > > > could only support the second one as it's more complete.
> > > > >> > > > > > - Add support for arrays with `[<integer>]`, e.g.
> > `$a.[1].b`
> > > > >> > > > > > - Add support for multiple-value paths using array
> access
> > > > >> `$a.*.b`
> > > > >> > or
> > > > >> > > > > deep
> > > > >> > > > > > scan `$a..b`.
> > > > >> > > > > >     - Some SMTs that will benefit from this:
> `MaskField`,
> > > > >> `Cast`,
> > > > >> > > > > > `ReplaceField`.
> > > > >> > > > > > - We could introduce a `path[s]` config under the
> current
> > > > >> > > > configurations
> > > > >> > > > > to
> > > > >> > > > > > apply this feature so no compatibility issues are
> > > introduced:
> > > > >> e.g.
> > > > >> > > > > > `field.path`, `fields.paths`, `exclude.path`.
> > > > >> > > > > >
> > > > >> > > > > > With these,  100, 101, 102, and 103 will be effectively
> > > > solved.
> > > > >> > > > > >
> > > > >> > > > > > The main challenge that I see at the moment is that by
> > being
> > > > >> JSON
> > > > >> > > > > > path-like, there may be some edge cases that I can't
> > foresee
> > > > at
> > > > >> the
> > > > >> > > > > moment,
> > > > >> > > > > > that could make this hard to implement, test, and
> maintain
> > > in
> > > > >> the
> > > > >> > > long
> > > > >> > > > > run.
> > > > >> > > > > >
> > > > >> > > > > > I'll appreciate your feedback on how this JSONPath-based
> > > > >> > alternative
> > > > >> > > > > > compares to the dotted notation initially proposed, and
> if
> > > it
> > > > >> > matches
> > > > >> > > > > your
> > > > >> > > > > > feedback.
> > > > >> > > > > > I have drafted a copy of the KIP to change the examples
> to
> > > > >> JSONPath
> > > > >> > > > > > approach and validate some differences:
> > > > >> > > > > > https://cwiki.apache.org/confluence/x/9BihD
> > > > >> > > > > >
> > > > >> > > > > > About 104. I agreed with Chris. We can handle this as
> part
> > > of
> > > > a
> > > > >> new
> > > > >> > > > KIP.
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Jorge.
> > > > >> > > > > >
> > > > >> > > > > > On Sun, 24 Apr 2022 at 17:27, Chris Egerton <
> > > > >> > fearthecellos@gmail.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Joshua,
> > > > >> > > > > > >
> > > > >> > > > > > > I have a few reservations about using JsonPath
> notation
> > > > here.
> > > > >> > > > > > >
> > > > >> > > > > > > 1. There's likely to be a substantial performance
> > penalty
> > > > for
> > > > >> > > > > converting
> > > > >> > > > > > > between the Kafka Connect format and something that a
> > > > JsonPath
> > > > >> > > > library
> > > > >> > > > > > > would understand.
> > > > >> > > > > > >
> > > > >> > > > > > > 2. The complexity of the feature will be significantly
> > > > >> higher. It
> > > > >> > > > will
> > > > >> > > > > be
> > > > >> > > > > > > harder to test, document, and implement. There will be
> > > many
> > > > >> more
> > > > >> > > edge
> > > > >> > > > > > cases
> > > > >> > > > > > > to consider and support, and we'll be on the hook to
> > > handle
> > > > >> any
> > > > >> > > bugs
> > > > >> > > > or
> > > > >> > > > > > > inconsistencies that arise, either as a result of our
> > use
> > > of
> > > > >> the
> > > > >> > > > > JsonPath
> > > > >> > > > > > > library we choose, or as a result of bugs in that
> > library.
> > > > >> > > > > > >
> > > > >> > > > > > > 3. It's not clear that JsonPath is superior or even
> > > suitable
> > > > >> for
> > > > >> > > some
> > > > >> > > > > of
> > > > >> > > > > > > the SMTs proposed here. What would be the advantages
> of
> > > > >> JsonPath
> > > > >> > > with
> > > > >> > > > > the
> > > > >> > > > > > > InsertField or HoistField SMTs?
> > > > >> > > > > > >
> > > > >> > > > > > > I also don't think that adding dot notation is
> > unfriendly
> > > to
> > > > >> > users;
> > > > >> > > > > many
> > > > >> > > > > > > have proposed this type of syntax in the past, and
> it's
> > > > >> > frequently
> > > > >> > > > used
> > > > >> > > > > > in
> > > > >> > > > > > > informal discussions to refer to nested fields. If the
> > > > >> proposed
> > > > >> > > > syntax
> > > > >> > > > > > was
> > > > >> > > > > > > not already intuitive then a case against deciding on
> > our
> > > > own
> > > > >> > might
> > > > >> > > > be
> > > > >> > > > > > more
> > > > >> > > > > > > convincing, but as things stand, simple dot notation
> is
> > > > likely
> > > > >> > > going
> > > > >> > > > to
> > > > >> > > > > > be
> > > > >> > > > > > > easier for users to understand than JsonPath syntax.
> > > > >> > > > > > >
> > > > >> > > > > > > Cheers,
> > > > >> > > > > > >
> > > > >> > > > > > > Chris
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Apr 22, 2022 at 9:31 AM Joshua Grisham <
> > > > >> > > > grishamjosh@gmail.com>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hello all!
> > > > >> > > > > > > >
> > > > >> > > > > > > > Sorry that I come a bit later to the party here,
> but I
> > > am
> > > > >> the
> > > > >> > one
> > > > >> > > > who
> > > > >> > > > > > > wrote
> > > > >> > > > > > > > KIP-683 [1] for recursive support (just simply
> looping
> > > > >> through
> > > > >> > > all
> > > > >> > > > > > child
> > > > >> > > > > > > > non-primitive structures for the same matching
> > name(s))
> > > > >> which
> > > > >> > is
> > > > >> > > a
> > > > >> > > > > > > slightly
> > > > >> > > > > > > > different way to try and solve a similar requirement
> > --
> > > > >> > > > unfortunately
> > > > >> > > > > > at
> > > > >> > > > > > > > the time the dev community was not quite as active
> and
> > > > then
> > > > >> I
> > > > >> > > also
> > > > >> > > > > got
> > > > >> > > > > > > busy
> > > > >> > > > > > > > with work and just life in general so wasn't able to
> > > > follow
> > > > >> up
> > > > >> > or
> > > > >> > > > > push
> > > > >> > > > > > > it.
> > > > >> > > > > > > >
> > > > >> > > > > > > > I do think it is a very good idea to have some kind
> of
> > > > >> > path-like
> > > > >> > > > > > > expression
> > > > >> > > > > > > > to be able to specifically address a nested field,
> as
> > I
> > > > can
> > > > >> see
> > > > >> > > > that
> > > > >> > > > > > the
> > > > >> > > > > > > > simple "recursive" case could potentially result in
> > > > >> unwanted or
> > > > >> > > > > > > unexpected
> > > > >> > > > > > > > behavior, plus there is the potential to introduce a
> > bit
> > > > of
> > > > >> a
> > > > >> > > > > > performance
> > > > >> > > > > > > > hit to always loop through everything in cases where
> > the
> > > > >> > > > > schemas/values
> > > > >> > > > > > > > might be quite large.
> > > > >> > > > > > > >
> > > > >> > > > > > > > One thing I wanted to ask: instead of creating a new
> > > "path
> > > > >> > > parser"
> > > > >> > > > > > > > including some bespoke or "borrowed" syntax, why not
> > > just
> > > > >> use
> > > > >> > > > > something
> > > > >> > > > > > > > that already exists? Specifically here I am thinking
> > > about
> > > > >> > > > JsonPath (
> > > > >> > > > > > > > https://github.com/json-path/JsonPath)
> > > > >> > > > > > > >
> > > > >> > > > > > > > There is already quite nice support in JsonPath for
> > > > handling
> > > > >> > > > special
> > > > >> > > > > > > > characters in field names, for handling different
> > > > >> non-primitive
> > > > >> > > > types
> > > > >> > > > > > > > (arrays etc), for handling multiple levels of
> nesting,
> > > etc
> > > > >> etc.
> > > > >> > > > > Would
> > > > >> > > > > > it
> > > > >> > > > > > > > be possible to instead to re-think this and maybe
> have
> > > > some
> > > > >> > kind
> > > > >> > > of
> > > > >> > > > > > > > JsonPath-based Schema selector / updater and/or
> > > > >> JsonPath-based
> > > > >> > > > Value
> > > > >> > > > > > > > selector / updater? Conceptually this feels like it
> > > makes
> > > > >> sense
> > > > >> > > to
> > > > >> > > > > me,
> > > > >> > > > > > as
> > > > >> > > > > > > > from the top of my head it would be quite a natural
> > fit
> > > to
> > > > >> map
> > > > >> > a
> > > > >> > > > Json
> > > > >> > > > > > > data
> > > > >> > > > > > > > structure to the Connect API data structure (and you
> > > could
> > > > >> > > > > potentially
> > > > >> > > > > > > even
> > > > >> > > > > > > > try to leverage the existing Json-to-Connect
> > > > >> > > > serializer/deserializer
> > > > >> > > > > to
> > > > >> > > > > > > > help out with this even in a more "out of the
> > > box"-feeling
> > > > >> kind
> > > > >> > > of
> > > > >> > > > > > way).
> > > > >> > > > > > > >
> > > > >> > > > > > > > Maybe also as Tom mentioned, this part (in my
> example,
> > > > this
> > > > >> > > > > > > JsonPath-based
> > > > >> > > > > > > > "thing") could even be a generic API that could be
> > used
> > > by
> > > > >> any
> > > > >> > > SMT,
> > > > >> > > > > > > > including used in custom ones built by the
> community.
> > > > Then
> > > > >> I
> > > > >> > > think
> > > > >> > > > > to
> > > > >> > > > > > > use
> > > > >> > > > > > > > a completely separate config property somehow
> related
> > to
> > > > >> "path"
> > > > >> > > (as
> > > > >> > > > > Tom
> > > > >> > > > > > > > also mentioned) would also make a lot of sense here
> as
> > > > well.
> > > > >> > This
> > > > >> > > > > way,
> > > > >> > > > > > if
> > > > >> > > > > > > > you select based on "path" then this JsonPath-based
> > API
> > > > >> would
> > > > >> > be
> > > > >> > > > > used,
> > > > >> > > > > > > > otherwise it could use something similar to the
> > existing
> > > > >> > > get-field
> > > > >> > > > > > based
> > > > >> > > > > > > > approach (which I guess could also be refactored
> into
> > > some
> > > > >> kind
> > > > >> > > of
> > > > >> > > > > > > utility
> > > > >> > > > > > > > / API as well if it made sense?)
> > > > >> > > > > > > >
> > > > >> > > > > > > > And with that in mind, if this was the kind of
> > direction
> > > > to
> > > > >> go,
> > > > >> > > > then
> > > > >> > > > > a
> > > > >> > > > > > > > "recursive" capability like I pitched in KIP-683
> would
> > > > also
> > > > >> > > become
> > > > >> > > > > > > > unnecessary because you could easily write a
> JsonPath
> > > > >> > expression
> > > > >> > > > like
> > > > >> > > > > > > > "$..someRecuriveField" and it would do the same
> thing
> > > (on
> > > > >> top
> > > > >> > of
> > > > >> > > > > > anything
> > > > >> > > > > > > > else you would want to do that is already supported
> by
> > > > >> > JsonPath).
> > > > >> > > > > Then
> > > > >> > > > > > we
> > > > >> > > > > > > > could also kill that older KIP and do a bit of
> > clean-up
> > > :)
> > > > >> > > > > > > >
> > > > >> > > > > > > > [1] -
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> > > > >> > > > > > > >
> > > > >> > > > > > > > Just some extra food for thought. All-in-all I think
> > > this
> > > > >> is a
> > > > >> > > > super
> > > > >> > > > > > > great
> > > > >> > > > > > > > initiative!
> > > > >> > > > > > > >
> > > > >> > > > > > > > Best,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Joshua
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > Den fre 22 apr. 2022 kl 14:50 skrev Chris Egerton <
> > > > >> > > > > > > fearthecellos@gmail.com
> > > > >> > > > > > > > >:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Tom,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks for taking a look at this, and for your
> > > > thoughtful
> > > > >> > > > comments.
> > > > >> > > > > > > I'll
> > > > >> > > > > > > > > leave it up to Jorge to address most of your
> > comments
> > > > but
> > > > >> I
> > > > >> > > > wanted
> > > > >> > > > > to
> > > > >> > > > > > > > share
> > > > >> > > > > > > > > a couple quick thoughts I had regarding 103 and
> 104.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 103. Like you, I was envisioning a possible syntax
> > for
> > > > >> array
> > > > >> > > > access
> > > > >> > > > > > > that
> > > > >> > > > > > > > > used classic C-style brackets; e.g., `arr[index]`.
> > > > >> However, I
> > > > >> > > > > wonder
> > > > >> > > > > > if
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > could keep things simple and use the same syntax
> > that
> > > > >> we're
> > > > >> > > > > proposing
> > > > >> > > > > > > for
> > > > >> > > > > > > > > nested field access? In other words, instead of
> > > > >> `arr[index]`,
> > > > >> > > > you'd
> > > > >> > > > > > > write
> > > > >> > > > > > > > > `arr.index`. It'd save us and users the headache
> of
> > > > >> reserving
> > > > >> > > > > > > characters
> > > > >> > > > > > > > > now that would need to be escaped even if their
> > > > unescaped
> > > > >> > > > brethren
> > > > >> > > > > > > aren't
> > > > >> > > > > > > > > used for anything, and also avoid the question of
> > what
> > > > >> > exactly
> > > > >> > > we
> > > > >> > > > > > > should
> > > > >> > > > > > > > do
> > > > >> > > > > > > > > when we see a config that uses reserved characters
> > > that
> > > > >> > aren't
> > > > >> > > > yet
> > > > >> > > > > > > > > supported (throwing an exception seems pretty
> > > unfriendly
> > > > >> for
> > > > >> > > new
> > > > >> > > > > > > users).
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 104. This would probably be useful, but it would
> > come
> > > > with
> > > > >> > some
> > > > >> > > > > nasty
> > > > >> > > > > > > > > compatibility questions that would need to be
> > > addressed
> > > > if
> > > > >> > we'd
> > > > >> > > > > want
> > > > >> > > > > > > SMTs
> > > > >> > > > > > > > > that leverage this new API to be viable for older
> > > > >> versions of
> > > > >> > > > > > Connect.
> > > > >> > > > > > > If
> > > > >> > > > > > > > > we package and distribute this feature as a
> library
> > > > >> (either
> > > > >> > via
> > > > >> > > > an
> > > > >> > > > > > > > entirely
> > > > >> > > > > > > > > new artifact, or as part of the existing
> > > > >> connect-transforms
> > > > >> > or
> > > > >> > > > > > > > connect-api
> > > > >> > > > > > > > > artifacts), then we'd have to either sidestep the
> > > > existing
> > > > >> > > plugin
> > > > >> > > > > > > > isolation
> > > > >> > > > > > > > > logic [1] that basically makes it impossible for
> > > Connect
> > > > >> > > plugins
> > > > >> > > > to
> > > > >> > > > > > > ship
> > > > >> > > > > > > > > their own versions of Connect artifacts, or issue
> a
> > > big
> > > > >> > warning
> > > > >> > > > to
> > > > >> > > > > > > people
> > > > >> > > > > > > > > that any SMT that uses this API won't work with
> any
> > > > older
> > > > >> > > > versions
> > > > >> > > > > of
> > > > >> > > > > > > > > Connect. There's also some other features we might
> > > want
> > > > to
> > > > >> > add
> > > > >> > > in
> > > > >> > > > > an
> > > > >> > > > > > > > > SMT-utils library such as the existing, internal,
> > > utils
> > > > >> that
> > > > >> > > > > Connect
> > > > >> > > > > > > uses
> > > > >> > > > > > > > > right now [2]. It may be worth exploring this in a
> > > > >> separate
> > > > >> > KIP
> > > > >> > > > of
> > > > >> > > > > > its
> > > > >> > > > > > > > own.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > [1] -
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/d480c4aa6e513e36050d8e067931de2270525d18/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L46-L143
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > [2] -
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/kafka/tree/d480c4aa6e513e36050d8e067931de2270525d18/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Cheers,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Chris
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Fri, Apr 22, 2022 at 6:55 AM Tom Bentley <
> > > > >> > > tbentley@redhat.com
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks for the KIP, especially for the examples
> > > which
> > > > >> are
> > > > >> > > > > > > super-clear.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 100. The name `field.style` isn't so clear for
> > > > something
> > > > >> > like
> > > > >> > > > > > > > > ReplaceField:
> > > > >> > > > > > > > > > it's not so obvious that field.style applies to
> > > > >> `include`
> > > > >> > and
> > > > >> > > > > > > > `exclude`.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 101. The permitted values for `field.style`
> don't
> > > seem
> > > > >> > > terribly
> > > > >> > > > > > > > intuitive
> > > > >> > > > > > > > > > (to me anyway): the meaning of `plain` isn't
> very
> > > > >> > guessable.
> > > > >> > > > Why
> > > > >> > > > > > not
> > > > >> > > > > > > > > > `top-level` or `root` instead? Also `nested`
> could
> > > be
> > > > >> > > > > misconstrued
> > > > >> > > > > > to
> > > > >> > > > > > > > > mean
> > > > >> > > > > > > > > > nested-but-not-top-level, so perhaps `recursive`
> > or
> > > > >> > > `cascading`
> > > > >> > > > > > might
> > > > >> > > > > > > > be
> > > > >> > > > > > > > > > better?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 102. I'm torn on whether making the
> interpretation
> > > of
> > > > >> > > existing
> > > > >> > > > > > > configs
> > > > >> > > > > > > > > like
> > > > >> > > > > > > > > > `field` be dependent on `field.style` is a good
> > > idea.
> > > > I
> > > > >> can
> > > > >> > > see
> > > > >> > > > > > that
> > > > >> > > > > > > > it's
> > > > >> > > > > > > > > > the simplest thing to do, but it just feels a
> bit
> > > odd
> > > > >> that
> > > > >> > > > > > sometimes
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > `field` would actually be a path and have
> > different
> > > > >> > escaping
> > > > >> > > > > rules.
> > > > >> > > > > > > An
> > > > >> > > > > > > > > > alternative would be to come up with a parallel
> > set
> > > of
> > > > >> > config
> > > > >> > > > > names
> > > > >> > > > > > > > (e.g.
> > > > >> > > > > > > > > > as well as "field" an SMT might support "path")
> > > which
> > > > >> were
> > > > >> > > > > defined
> > > > >> > > > > > to
> > > > >> > > > > > > > > > always take paths, thus avoiding the changeable
> > > > >> > > interpretation
> > > > >> > > > of
> > > > >> > > > > > the
> > > > >> > > > > > > > > > existing configs. The SMT's #configure() would
> > need
> > > to
> > > > >> > throw
> > > > >> > > in
> > > > >> > > > > the
> > > > >> > > > > > > > case
> > > > >> > > > > > > > > > that both configs were given. I can see that
> that
> > > > would
> > > > >> be
> > > > >> > > more
> > > > >> > > > > > work
> > > > >> > > > > > > in
> > > > >> > > > > > > > > > implementation, but it feels cleaner.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 103. I think in order to allow for supporting
> > arrays
> > > > in
> > > > >> a
> > > > >> > > later
> > > > >> > > > > KIP
> > > > >> > > > > > > > > (which
> > > > >> > > > > > > > > > certainly seems like it could be useful), we'd
> > want
> > > to
> > > > >> > > specify
> > > > >> > > > > the
> > > > >> > > > > > > > syntax
> > > > >> > > > > > > > > > now, even if it wasn't implemented under this
> KIP.
> > > > >> That's
> > > > >> > > > > because I
> > > > >> > > > > > > > don't
> > > > >> > > > > > > > > > think you can't exclude the possibility that
> > > > characters
> > > > >> > such
> > > > >> > > as
> > > > >> > > > > `[`
> > > > >> > > > > > > and
> > > > >> > > > > > > > > `]`
> > > > >> > > > > > > > > > appear in field names. So you'd have a
> > compatibility
> > > > >> > problem
> > > > >> > > if
> > > > >> > > > > > > people
> > > > >> > > > > > > > > > started using the features of this KIP to access
> > > such
> > > > >> > fields,
> > > > >> > > > > only
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > those characters to change their meaning under a
> > > later
> > > > >> KIP.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 104. I also wonder whether making paths into a
> > > public
> > > > >> Java
> > > > >> > > API,
> > > > >> > > > > for
> > > > >> > > > > > > use
> > > > >> > > > > > > > > by
> > > > >> > > > > > > > > > 3rd party SMTs, would be valuable.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks again,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Tom
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Wed, 20 Apr 2022 at 17:53, Chris Egerton <
> > > > >> > > > > > fearthecellos@gmail.com
> > > > >> > > > > > > >
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > 💯 Thanks Jorge, LGTM!
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Wed, Apr 20, 2022, 12:40 Jorge Esteban
> > Quilcate
> > > > >> Otoya
> > > > >> > <
> > > > >> > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thank you, Chris! Not possible without your
> > > > >> feedback.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Tue, 19 Apr 2022 at 23:04, Chris Egerton
> <
> > > > >> > > > > > > > fearthecellos@gmail.com
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thank you for sticking through this. I
> have
> > > one
> > > > >> small
> > > > >> > > > > remark
> > > > >> > > > > > > and
> > > > >> > > > > > > > > one
> > > > >> > > > > > > > > > > > small
> > > > >> > > > > > > > > > > > > clarification; assuming you agree with me
> on
> > > > them
> > > > >> > then
> > > > >> > > > I'm
> > > > >> > > > > > > ready
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > vote
> > > > >> > > > > > > > > > > > on
> > > > >> > > > > > > > > > > > > the KIP.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > 1. InsertField: The
> > "field.on.missing.parent"
> > > > and
> > > > >> > > > > > > > > > > > "field.on.existing.field"
> > > > >> > > > > > > > > > > > > docs both mention a permitted value of
> > > "ingore";
> > > > >> this
> > > > >> > > > > should
> > > > >> > > > > > be
> > > > >> > > > > > > > > > > "ignore",
> > > > >> > > > > > > > > > > > > right?
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Of course, one more typo :)
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > 2. InsertField: The examples are still
> > missing
> > > > the
> > > > >> > > > > > > "field.style"
> > > > >> > > > > > > > > > > property
> > > > >> > > > > > > > > > > > > from the configurations. They should all
> > > include
> > > > >> the
> > > > >> > > > > property
> > > > >> > > > > > > > > > > > > "transforms.smt1.field.style": "nested",
> > > > correct?
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Yes, it is there. I think I know what you
> mean
> > > > now,
> > > > >> > seems
> > > > >> > > > > that
> > > > >> > > > > > > > > > Confluence
> > > > >> > > > > > > > > > > > is putting everything in one line when it's
> in
> > > > >> separate
> > > > >> > > > lines
> > > > >> > > > > > in
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > editor.
> > > > >> > > > > > > > > > > > Hopefully, it's fixed now.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks again for working through this, and
> > > > >> > > > congratulations
> > > > >> > > > > > on a
> > > > >> > > > > > > > > > > > > well-written KIP!
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Chris
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Tue, Apr 19, 2022 at 2:06 PM Jorge
> > Esteban
> > > > >> > Quilcate
> > > > >> > > > > Otoya
> > > > >> > > > > > <
> > > > >> > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Thank you, Chris! I apply these
> > improvements
> > > > to
> > > > >> the
> > > > >> > > > KIP,
> > > > >> > > > > > let
> > > > >> > > > > > > me
> > > > >> > > > > > > > > > know
> > > > >> > > > > > > > > > > > how
> > > > >> > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > looks now.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On Mon, 11 Apr 2022 at 23:43, Chris
> > Egerton
> > > <
> > > > >> > > > > > > > > > fearthecellos@gmail.com
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Wow, those examples are great! A few
> > more
> > > > >> > remarks,
> > > > >> > > > but
> > > > >> > > > > I
> > > > >> > > > > > > > think
> > > > >> > > > > > > > > > > we're
> > > > >> > > > > > > > > > > > > > > getting close:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > 1. The examples differ across SMTs
> with
> > > the
> > > > >> name
> > > > >> > of
> > > > >> > > > the
> > > > >> > > > > > > > > > > > > newly-introduced
> > > > >> > > > > > > > > > > > > > > style property; some of them use
> > > > >> "field.style",
> > > > >> > and
> > > > >> > > > > some
> > > > >> > > > > > > use
> > > > >> > > > > > > > > > > > > > > "fields.style". I think for
> > consistency's
> > > > >> sake we
> > > > >> > > > > should
> > > > >> > > > > > > > stick
> > > > >> > > > > > > > > > with
> > > > >> > > > > > > > > > > > > just
> > > > >> > > > > > > > > > > > > > > "field.style"; otherwise it could be
> > > painful
> > > > >> for
> > > > >> > > > users
> > > > >> > > > > to
> > > > >> > > > > > > > have
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > > > remember
> > > > >> > > > > > > > > > > > > > > which to use.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Great catch. Agree, I fixed the config
> > names
> > > > to
> > > > >> > > > > > > `field.style`.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > 2. Some of the examples are off:
> > > > >> > > > > > > > > > > > > > > - TimestampConverter: the input in the
> > > > second
> > > > >> > > example
> > > > >> > > > > > > ("when
> > > > >> > > > > > > > > > field
> > > > >> > > > > > > > > > > > > names
> > > > >> > > > > > > > > > > > > > > include dots") doesn't contain a field
> > > with
> > > > a
> > > > >> > > dotted
> > > > >> > > > > name
> > > > >> > > > > > > > > > > > > > > - ValueToKey: the config in the third
> > > > example
> > > > >> > > ("when
> > > > >> > > > > > field
> > > > >> > > > > > > > > names
> > > > >> > > > > > > > > > > > > include
> > > > >> > > > > > > > > > > > > > > dots") should probably use
> > > > "parent..child.k2"
> > > > >> as
> > > > >> > > the
> > > > >> > > > > > > > > > > > > > > "transforms.smt1.fields" property
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Fixed. Thanks!
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > 3. RE changes to InsertField:
> > > > >> > > > > > > > > > > > > > > - The InsertField SMT should also come
> > > with
> > > > >> the
> > > > >> > new
> > > > >> > > > > > > > > "field.style"
> > > > >> > > > > > > > > > > > > > property
> > > > >> > > > > > > > > > > > > > > in order to preserve backwards
> > > > compatibility,
> > > > >> > > right?
> > > > >> > > > I
> > > > >> > > > > > > don't
> > > > >> > > > > > > > > see
> > > > >> > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > included in the example configs for
> that
> > > > one,
> > > > >> > just
> > > > >> > > > want
> > > > >> > > > > > to
> > > > >> > > > > > > > make
> > > > >> > > > > > > > > > > sure
> > > > >> > > > > > > > > > > > > > > - I don't know of any cases where we
> use
> > > > >> > snake_case
> > > > >> > > > for
> > > > >> > > > > > > > > property
> > > > >> > > > > > > > > > > > names
> > > > >> > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > Kafka; we should probably use
> > > > >> "on.missing.parent"
> > > > >> > > and
> > > > >> > > > > > > > > > > > > "on.existing.field"
> > > > >> > > > > > > > > > > > > > > as the new property names for
> > InsertField.
> > > > >> > > > > > > > > > > > > > > - Why is the "on_existing_field" (or
> > > > >> > > > > "on.existing.field")
> > > > >> > > > > > > > > > property
> > > > >> > > > > > > > > > > > only
> > > > >> > > > > > > > > > > > > > > applied when the field style is
> nested?
> > > > >> Couldn't
> > > > >> > it
> > > > >> > > > be
> > > > >> > > > > > > useful
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > non-nested fields as well?
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Great points! I have applied these
> > > suggestions
> > > > >> to
> > > > >> > the
> > > > >> > > > > KIP.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Chris
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > On Sat, Apr 9, 2022 at 12:40 PM Jorge
> > > > Esteban
> > > > >> > > > Quilcate
> > > > >> > > > > > > Otoya
> > > > >> > > > > > > > <
> > > > >> > > > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Again, great feedback Chris. Much
> > > > >> appreciated.
> > > > >> > > > > > > > > > > > > > > > Added my comments below:
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > On Tue, 5 Apr 2022 at 20:22, Chris
> > > > Egerton <
> > > > >> > > > > > > > > > > > fearthecellos@gmail.com>
> > > > >> > > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Looking good! I have a few
> comments
> > > left
> > > > >> but
> > > > >> > > all
> > > > >> > > > > but
> > > > >> > > > > > > one
> > > > >> > > > > > > > or
> > > > >> > > > > > > > > > two
> > > > >> > > > > > > > > > > > are
> > > > >> > > > > > > > > > > > > > > > minor.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 1. The motivation section states
> > "This
> > > > >> KIP is
> > > > >> > > > aimed
> > > > >> > > > > > to
> > > > >> > > > > > > > > > include
> > > > >> > > > > > > > > > > > > > support
> > > > >> > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > nested structures on the existing
> > > > SMTs...
> > > > >> and
> > > > >> > > to
> > > > >> > > > > > > include
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > abstractions
> > > > >> > > > > > > > > > > > > > > > > to reuse this in future SMTs". A
> > good
> > > > >> > > > > implementation
> > > > >> > > > > > of
> > > > >> > > > > > > > > this
> > > > >> > > > > > > > > > > KIP
> > > > >> > > > > > > > > > > > > will
> > > > >> > > > > > > > > > > > > > > > > definitely isolate reusable logic
> > > into a
> > > > >> > > separate
> > > > >> > > > > > > > > abstraction
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > can
> > > > >> > > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > > easily pulled in to the SMTs we
> want
> > > to
> > > > >> add
> > > > >> > > > nested
> > > > >> > > > > > > field
> > > > >> > > > > > > > > > > support
> > > > >> > > > > > > > > > > > > to,
> > > > >> > > > > > > > > > > > > > > but
> > > > >> > > > > > > > > > > > > > > > > unless we plan on making this kind
> > of
> > > > >> > > abstraction
> > > > >> > > > > > > > publicly
> > > > >> > > > > > > > > > > > > available
> > > > >> > > > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > > > > > > some kind of utility method or
> class
> > > > that
> > > > >> > > > external
> > > > >> > > > > > SMT
> > > > >> > > > > > > > > > > developers
> > > > >> > > > > > > > > > > > > can
> > > > >> > > > > > > > > > > > > > > > > leverage, we can probably leave
> this
> > > > part
> > > > >> out
> > > > >> > > as
> > > > >> > > > > it's
> > > > >> > > > > > > > more
> > > > >> > > > > > > > > of
> > > > >> > > > > > > > > > > an
> > > > >> > > > > > > > > > > > > > > > > implementation detail.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Make sense, will leave this out of
> the
> > > > KIP.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 2. The Cast example is a little
> > > > >> misleading,
> > > > >> > > isn't
> > > > >> > > > > it?
> > > > >> > > > > > > It
> > > > >> > > > > > > > > > > > > demonstrates
> > > > >> > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > escape syntax for fields with dot
> > > > >> literals in
> > > > >> > > > their
> > > > >> > > > > > > > names,
> > > > >> > > > > > > > > > but
> > > > >> > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > doesn't
> > > > >> > > > > > > > > > > > > > > > > demonstrate a way to actually use
> > the
> > > > Cast
> > > > >> > (or
> > > > >> > > > any
> > > > >> > > > > > > other)
> > > > >> > > > > > > > > SMT
> > > > >> > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > access a
> > > > >> > > > > > > > > > > > > > > > > nested field in a record, which is
> > the
> > > > >> whole
> > > > >> > > > point
> > > > >> > > > > of
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > KIP.
> > > > >> > > > > > > > > > > I
> > > > >> > > > > > > > > > > > > like
> > > > >> > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > example of escape syntax but we
> > should
> > > > >> > probably
> > > > >> > > > > also
> > > > >> > > > > > > add
> > > > >> > > > > > > > > one
> > > > >> > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > > > field access.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Agree. I have added examples to each
> > SMT
> > > > to
> > > > >> be
> > > > >> > > more
> > > > >> > > > > > clear
> > > > >> > > > > > > > > about
> > > > >> > > > > > > > > > > how
> > > > >> > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > > affects each function
> > > > >> > > > > > > > > > > > > > > > .
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 3. With the InsertField SMT, I'm
> > > > wondering
> > > > >> > what
> > > > >> > > > the
> > > > >> > > > > > > > > specific
> > > > >> > > > > > > > > > > > > behavior
> > > > >> > > > > > > > > > > > > > > > will
> > > > >> > > > > > > > > > > > > > > > > be when some or all of the "middle
> > > > layer"
> > > > >> > > nested
> > > > >> > > > > > fields
> > > > >> > > > > > > > are
> > > > >> > > > > > > > > > > > > missing.
> > > > >> > > > > > > > > > > > > > > For
> > > > >> > > > > > > > > > > > > > > > > example, if I have a record with a
> > > value
> > > > >> of {
> > > > >> > > > "k1":
> > > > >> > > > > > > "v1 }
> > > > >> > > > > > > > > > and I
> > > > >> > > > > > > > > > > > > apply
> > > > >> > > > > > > > > > > > > > > > > InsertField with topic.field =
> > > > >> > n1.n2.n3.topic,
> > > > >> > > > what
> > > > >> > > > > > > will
> > > > >> > > > > > > > > > > happen?
> > > > >> > > > > > > > > > > > > Will
> > > > >> > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > updated value become { "k1": "v1",
> > > > "n1": {
> > > > >> > > "n2":
> > > > >> > > > {
> > > > >> > > > > > > "n3":
> > > > >> > > > > > > > > > > "topic"
> > > > >> > > > > > > > > > > > > }}},
> > > > >> > > > > > > > > > > > > > > > will
> > > > >> > > > > > > > > > > > > > > > > an exception be thrown, or
> something
> > > > else?
> > > > >> > This
> > > > >> > > > > seems
> > > > >> > > > > > > > > > analogous
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > command line mkdir command, which
> > (at
> > > > >> least
> > > > >> > on
> > > > >> > > > some
> > > > >> > > > > > > > > operating
> > > > >> > > > > > > > > > > > > > systems)
> > > > >> > > > > > > > > > > > > > > > > fails by default if you try to
> > create
> > > a
> > > > >> new
> > > > >> > > > nested
> > > > >> > > > > > > > > directory
> > > > >> > > > > > > > > > > > where
> > > > >> > > > > > > > > > > > > > > > anything
> > > > >> > > > > > > > > > > > > > > > > but the last element in the path
> > > doesn't
> > > > >> > exist,
> > > > >> > > > but
> > > > >> > > > > > can
> > > > >> > > > > > > > be
> > > > >> > > > > > > > > > > > invoked
> > > > >> > > > > > > > > > > > > > > with a
> > > > >> > > > > > > > > > > > > > > > > flag that instructs it to go ahead
> > and
> > > > >> create
> > > > >> > > all
> > > > >> > > > > > > levels
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > > > directory instead. I'm leaning on
> > the
> > > > >> side of
> > > > >> > > > "just
> > > > >> > > > > > > > create
> > > > >> > > > > > > > > > > > > > everything"
> > > > >> > > > > > > > > > > > > > > > but
> > > > >> > > > > > > > > > > > > > > > > would be interested in your
> > thoughts,
> > > > and
> > > > >> > > either
> > > > >> > > > > way,
> > > > >> > > > > > > we
> > > > >> > > > > > > > > > should
> > > > >> > > > > > > > > > > > > > > probably
> > > > >> > > > > > > > > > > > > > > > > make sure the intended behavior is
> > > > >> > well-defined
> > > > >> > > > > > before
> > > > >> > > > > > > > > > voting.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > This is an interesting case, thanks
> > for
> > > > >> > catching
> > > > >> > > > > this!
> > > > >> > > > > > > > > > > > > > > > The default behavior I see is to
> > create
> > > > >> parents
> > > > >> > > if
> > > > >> > > > > they
> > > > >> > > > > > > are
> > > > >> > > > > > > > > > > missing
> > > > >> > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > overwrite fields
> > > > >> > > > > > > > > > > > > > > > if they already exist.
> > > > >> > > > > > > > > > > > > > > > I'm planning to include the
> following
> > > two
> > > > >> flags
> > > > >> > > if
> > > > >> > > > > > there
> > > > >> > > > > > > > is a
> > > > >> > > > > > > > > > > need
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > overwrite this behavior:
> > > > >> > > > > > > > > > > > > > > > - `on_missing_parent` =
> > (CREATE|IGNORE),
> > > > >> > > > > default=CREATE
> > > > >> > > > > > > > > > > > > > > > - `on_existing_field` =
> > > > (OVERWRITE|IGNORE),
> > > > >> > > > > > > > default=OVERWRITE
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 4. Similarly, what will the
> behavior
> > > be
> > > > if
> > > > >> > any
> > > > >> > > of
> > > > >> > > > > the
> > > > >> > > > > > > > field
> > > > >> > > > > > > > > > > > > elements
> > > > >> > > > > > > > > > > > > > > > > specified with InsertField already
> > > exist
> > > > >> in
> > > > >> > the
> > > > >> > > > > > record
> > > > >> > > > > > > > > value?
> > > > >> > > > > > > > > > > > Will
> > > > >> > > > > > > > > > > > > we
> > > > >> > > > > > > > > > > > > > > > just
> > > > >> > > > > > > > > > > > > > > > > overwrite them? What's the
> behavior
> > of
> > > > >> > > > InsertField
> > > > >> > > > > > > today
> > > > >> > > > > > > > > > under
> > > > >> > > > > > > > > > > > > > similar
> > > > >> > > > > > > > > > > > > > > > > circumstances?
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > The current behavior is to overwrite
> > the
> > > > >> value.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Chris
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at 4:15 PM
> > Jorge
> > > > >> Esteban
> > > > >> > > > > > Quilcate
> > > > >> > > > > > > > > Otoya
> > > > >> > > > > > > > > > <
> > > > >> > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com> wrote:
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Thanks, Chris! Much appreciated
> > all
> > > > the
> > > > >> > > > feedback
> > > > >> > > > > > > here.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > 1. You nailed it setting the
> > design
> > > > goal
> > > > >> > > here:
> > > > >> > > > > "it
> > > > >> > > > > > > > > > shouldn't
> > > > >> > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > > impossible
> > > > >> > > > > > > > > > > > > > > > > > to use this new feature for any
> > > field
> > > > >> name,
> > > > >> > > no
> > > > >> > > > > > matter
> > > > >> > > > > > > > how
> > > > >> > > > > > > > > > > > > > convoluted.
> > > > >> > > > > > > > > > > > > > > > > It's
> > > > >> > > > > > > > > > > > > > > > > > fine if edge cases introduce
> > > > difficulty
> > > > >> > (such
> > > > >> > > > as
> > > > >> > > > > > > > > > > less-readable
> > > > >> > > > > > > > > > > > > > > > > > configurations), but it's not
> fine
> > > if
> > > > >> they
> > > > >> > > > can't
> > > > >> > > > > be
> > > > >> > > > > > > > > > addressed
> > > > >> > > > > > > > > > > > at
> > > > >> > > > > > > > > > > > > > > all."
> > > > >> > > > > > > > > > > > > > > > > > Back to the previous proposals
> > > (using
> > > > >> only
> > > > >> > > dots
> > > > >> > > > > as
> > > > >> > > > > > > > > > > separators)
> > > > >> > > > > > > > > > > > we
> > > > >> > > > > > > > > > > > > > > have
> > > > >> > > > > > > > > > > > > > > > 2
> > > > >> > > > > > > > > > > > > > > > > > alternatives:
> > > > >> > > > > > > > > > > > > > > > > > 1. escaping with backslashes
> > > > >> > > > > > > > > > > > > > > > > > 2. escaping with dots itself
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > I'll lean for alternative 2, as
> > you
> > > > >> > proposed
> > > > >> > > > > > before.
> > > > >> > > > > > > > > Feels
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > me
> > > > >> > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > backslashes have more potential
> to
> > > > lead
> > > > >> to
> > > > >> > > > > > confusion
> > > > >> > > > > > > in
> > > > >> > > > > > > > > > JSON
> > > > >> > > > > > > > > > > > > > configs,
> > > > >> > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > CSV seems like a good precedent
> to
> > > use
> > > > >> the
> > > > >> > > same
> > > > >> > > > > > > > character
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > > escape
> > > > >> > > > > > > > > > > > > > > > > itself.
> > > > >> > > > > > > > > > > > > > > > > > KIP is updated to reflect this.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > 2. Thanks! I'll add an example,
> > and
> > > > >> stick
> > > > >> > > with
> > > > >> > > > > the
> > > > >> > > > > > > > > current
> > > > >> > > > > > > > > > > > > approach
> > > > >> > > > > > > > > > > > > > > > > > defining the style per
> individual
> > > > >> transform
> > > > >> > > > > > > > > configuration.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > 3. Yes, thanks! KIP updated.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > 4. Of course. KIP updated.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > On Mon, 28 Mar 2022 at 21:59,
> > Chris
> > > > >> > Egerton <
> > > > >> > > > > > > > > > > > > > fearthecellos@gmail.com
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > Thanks for addressing my
> > comments;
> > > > the
> > > > >> > KIP
> > > > >> > > > > looks
> > > > >> > > > > > > > > > up-to-date
> > > > >> > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > pretty
> > > > >> > > > > > > > > > > > > > > > > > > readable now, and the rejected
> > > > >> > alternatives
> > > > >> > > > > > section
> > > > >> > > > > > > > > does
> > > > >> > > > > > > > > > a
> > > > >> > > > > > > > > > > > > great
> > > > >> > > > > > > > > > > > > > > job
> > > > >> > > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > outlining the discussion so
> far
> > > and
> > > > >> > > providing
> > > > >> > > > > > > context
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > > > > anyone
> > > > >> > > > > > > > > > > > > > > else
> > > > >> > > > > > > > > > > > > > > > > who
> > > > >> > > > > > > > > > > > > > > > > > > might want to join in.
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > 1. Thoughts on choice of
> > > delimiter:
> > > > >> > > > > > > > > > > > > > > > > > > - I like the optimization for
> > > simple
> > > > >> > cases,
> > > > >> > > > > but I
> > > > >> > > > > > > > think
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > new
> > > > >> > > > > > > > > > > > > > > > > proposal
> > > > >> > > > > > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > > > > a little too restrictive. What
> > if
> > > > >> > there's a
> > > > >> > > > > field
> > > > >> > > > > > > > whose
> > > > >> > > > > > > > > > > name
> > > > >> > > > > > > > > > > > > > > contains
> > > > >> > > > > > > > > > > > > > > > > all
> > > > >> > > > > > > > > > > > > > > > > > > of the permitted options
> > > (currently
> > > > >> just
> > > > >> > > ".",
> > > > >> > > > > > ",",
> > > > >> > > > > > > > and
> > > > >> > > > > > > > > > > "/")?
> > > > >> > > > > > > > > > > > > > > > > > > - If we expand the set of
> > > permitted
> > > > >> > > > delimiters
> > > > >> > > > > to
> > > > >> > > > > > > > allow
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > > > any
> > > > >> > > > > > > > > > > > > > > > > > > single-character string,
> > > > configuration
> > > > >> > > > > complexity
> > > > >> > > > > > > > will
> > > > >> > > > > > > > > > > > increase
> > > > >> > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > readability may decrease
> > > > >> > > > > > > > > > > > > > > > > > > - Also worth pointing out that
> > > there
> > > > >> is
> > > > >> > > some
> > > > >> > > > > > > > convention
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > doubling
> > > > >> > > > > > > > > > > > > > > > a
> > > > >> > > > > > > > > > > > > > > > > > > delimiter character as an
> escape
> > > > >> > mechanism
> > > > >> > > > with
> > > > >> > > > > > > > formats
> > > > >> > > > > > > > > > > like
> > > > >> > > > > > > > > > > > > CSV
> > > > >> > > > > > > > > > > > > > > [1]
> > > > >> > > > > > > > > > > > > > > > > > > - Overall I think we may be
> > > > >> approaching
> > > > >> > the
> > > > >> > > > > > > > saturation
> > > > >> > > > > > > > > > > point
> > > > >> > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > productive
> > > > >> > > > > > > > > > > > > > > > > > > discussion on delimiter syntax
> > so
> > > I
> > > > >> don't
> > > > >> > > > want
> > > > >> > > > > to
> > > > >> > > > > > > > spend
> > > > >> > > > > > > > > > too
> > > > >> > > > > > > > > > > > > much
> > > > >> > > > > > > > > > > > > > > more
> > > > >> > > > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > your time on it. I think the
> one
> > > > point
> > > > >> > I'd
> > > > >> > > > like
> > > > >> > > > > > to
> > > > >> > > > > > > > > leave
> > > > >> > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > now
> > > > >> > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > > > > > shouldn't be impossible to use
> > > this
> > > > >> new
> > > > >> > > > feature
> > > > >> > > > > > for
> > > > >> > > > > > > > any
> > > > >> > > > > > > > > > > field
> > > > >> > > > > > > > > > > > > > name,
> > > > >> > > > > > > > > > > > > > > > no
> > > > >> > > > > > > > > > > > > > > > > > > matter how convoluted. It's
> fine
> > > if
> > > > >> edge
> > > > >> > > > cases
> > > > >> > > > > > > > > introduce
> > > > >> > > > > > > > > > > > > > difficulty
> > > > >> > > > > > > > > > > > > > > > > (such
> > > > >> > > > > > > > > > > > > > > > > > > as less-readable
> > configurations),
> > > > but
> > > > >> > it's
> > > > >> > > > not
> > > > >> > > > > > fine
> > > > >> > > > > > > > if
> > > > >> > > > > > > > > > they
> > > > >> > > > > > > > > > > > > can't
> > > > >> > > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > > > > addressed at all.
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > 2.
> > > > >> > > > > > > > > > > > > > > > > > > The configuration style where
> > you
> > > > >> define
> > > > >> > > > > > > > > > > > > "transforms.field.style"
> > > > >> > > > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > connector config, and then
> this
> > > > >> applies
> > > > >> > to
> > > > >> > > > all
> > > > >> > > > > > SMTs
> > > > >> > > > > > > > for
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > connector,
> > > > >> > > > > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > > > > very interesting. However, it
> > > > doesn't
> > > > >> > > follow
> > > > >> > > > > > > > convention
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > existing
> > > > >> > > > > > > > > > > > > > > > > > SMTs.
> > > > >> > > > > > > > > > > > > > > > > > > Right now, if you want to
> > > configure
> > > > an
> > > > >> > SMT,
> > > > >> > > > you
> > > > >> > > > > > > > define
> > > > >> > > > > > > > > > its
> > > > >> > > > > > > > > > > > name
> > > > >> > > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > connector config (for example,
> > > > >> > > "transforms":
> > > > >> > > > > > > "smt1"),
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > > then
> > > > >> > > > > > > > > > > > > > > define
> > > > >> > > > > > > > > > > > > > > > > all
> > > > >> > > > > > > > > > > > > > > > > > > of the properties for that SMT
> > in
> > > > the
> > > > >> > > > connector
> > > > >> > > > > > > > config
> > > > >> > > > > > > > > > > using
> > > > >> > > > > > > > > > > > a
> > > > >> > > > > > > > > > > > > > > > > > namespacing
> > > > >> > > > > > > > > > > > > > > > > > > mechanism specific to that SMT
> > > (for
> > > > >> > > example,
> > > > >> > > > > > > > > > > > > > > "transforms.smt1.prop1":
> > > > >> > > > > > > > > > > > > > > > > > > "val1"). That SMT then sees
> only
> > > the
> > > > >> > > > properties
> > > > >> > > > > > > > defined
> > > > >> > > > > > > > > > in
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > namespace,
> > > > >> > > > > > > > > > > > > > > > > > > with the prefix stripped (for
> > > > example,
> > > > >> > > > "prop1":
> > > > >> > > > > > > > "val1")
> > > > >> > > > > > > > > > in
> > > > >> > > > > > > > > > > > its
> > > > >> > > > > > > > > > > > > > > > > configure
> > > > >> > > > > > > > > > > > > > > > > > > [2] [3] method.
> > > > >> > > > > > > > > > > > > > > > > > > If we want to continue to
> follow
> > > > this
> > > > >> > > > > convention,
> > > > >> > > > > > > > then
> > > > >> > > > > > > > > > > > instead
> > > > >> > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > specifying
> > > "transforms.field.style"
> > > > >> in a
> > > > >> > > > > > connector
> > > > >> > > > > > > > > > config,
> > > > >> > > > > > > > > > > we
> > > > >> > > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > > > > > expect
> > > > >> > > > > > > > > > > > > > > > > > > users to configure
> > > > >> > > > > > "transforms.<name>.field.style",
> > > > >> > > > > > > > for
> > > > >> > > > > > > > > > > each
> > > > >> > > > > > > > > > > > > SMT
> > > > >> > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > they
> > > > >> > > > > > > > > > > > > > > > > > > want to configure a field
> style
> > > for.
> > > > >> This
> > > > >> > > > would
> > > > >> > > > > > > > require
> > > > >> > > > > > > > > > > more
> > > > >> > > > > > > > > > > > > work
> > > > >> > > > > > > > > > > > > > > on
> > > > >> > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > part of the user, but would be
> > > > >> simpler to
> > > > >> > > > > reason
> > > > >> > > > > > > > about
> > > > >> > > > > > > > > > and
> > > > >> > > > > > > > > > > > > easier
> > > > >> > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > implement.
> > > > >> > > > > > > > > > > > > > > > > > > If we want to explore an
> > > alternative
> > > > >> > where
> > > > >> > > > > users
> > > > >> > > > > > > can
> > > > >> > > > > > > > > > > specify
> > > > >> > > > > > > > > > > > > > global
> > > > >> > > > > > > > > > > > > > > > > > > properties that apply to all
> > > > >> transforms
> > > > >> > in
> > > > >> > > a
> > > > >> > > > > > > > connector
> > > > >> > > > > > > > > > > > config,
> > > > >> > > > > > > > > > > > > > then
> > > > >> > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > semantics for this need to be
> > > > defined
> > > > >> in
> > > > >> > > the
> > > > >> > > > > KIP.
> > > > >> > > > > > > > This
> > > > >> > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > have
> > > > >> > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > include whether this will
> apply
> > > only
> > > > >> for
> > > > >> > > the
> > > > >> > > > > > > special
> > > > >> > > > > > > > > case
> > > > >> > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > "field.style" and possibly
> > > > >> > > "field.separator"
> > > > >> > > > > > > > properties
> > > > >> > > > > > > > > > or
> > > > >> > > > > > > > > > > if
> > > > >> > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > > > > available more generally for
> > other
> > > > >> > > > properties,
> > > > >> > > > > > > > whether
> > > > >> > > > > > > > > it
> > > > >> > > > > > > > > > > > will
> > > > >> > > > > > > > > > > > > > > apply
> > > > >> > > > > > > > > > > > > > > > > only
> > > > >> > > > > > > > > > > > > > > > > > > for the SMTs outlined in the
> KIP
> > > or
> > > > if
> > > > >> > the
> > > > >> > > > > > > > > "field.style"
> > > > >> > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > possibly
> > > > >> > > > > > > > > > > > > > > > > > > "field.separator" properties
> > would
> > > > >> also
> > > > >> > be
> > > > >> > > > > passed
> > > > >> > > > > > > > into
> > > > >> > > > > > > > > > > custom
> > > > >> > > > > > > > > > > > > > SMTs
> > > > >> > > > > > > > > > > > > > > so
> > > > >> > > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > they could choose to act on
> them
> > > if
> > > > >> > > > applicable,
> > > > >> > > > > > how
> > > > >> > > > > > > > > edge
> > > > >> > > > > > > > > > > > cases
> > > > >> > > > > > > > > > > > > > like
> > > > >> > > > > > > > > > > > > > > > > > having
> > > > >> > > > > > > > > > > > > > > > > > > an SMT named "field" in your
> > > > connector
> > > > >> > > config
> > > > >> > > > > > would
> > > > >> > > > > > > > be
> > > > >> > > > > > > > > > > > handled,
> > > > >> > > > > > > > > > > > > > > etc.
> > > > >> > > > > > > > > > > > > > > > > > > Either way, it might help to
> > have
> > > an
> > > > >> > > example
> > > > >> > > > in
> > > > >> > > > > > the
> > > > >> > > > > > > > KIP
> > > > >> > > > > > > > > > > > > outlining
> > > > >> > > > > > > > > > > > > > > how
> > > > >> > > > > > > > > > > > > > > > > one
> > > > >> > > > > > > > > > > > > > > > > > > of the to-be-augmented SMTs
> can
> > be
> > > > >> > > configured
> > > > >> > > > > > with
> > > > >> > > > > > > > this
> > > > >> > > > > > > > > > new
> > > > >> > > > > > > > > > > > > > feature
> > > > >> > > > > > > > > > > > > > > > > and a
> > > > >> > > > > > > > > > > > > > > > > > > before/after of how a record
> > value
> > > > >> would
> > > > >> > be
> > > > >> > > > > > > > transformed
> > > > >> > > > > > > > > > > with
> > > > >> > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > configuration.
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > 3. The docstring for the
> > > > >> > > > > "transforms.field.style"
> > > > >> > > > > > > > > > property
> > > > >> > > > > > > > > > > > > > mentions
> > > > >> > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > the permitted values are
> "plain"
> > > and
> > > > >> > > > "nested",
> > > > >> > > > > > but
> > > > >> > > > > > > > then
> > > > >> > > > > > > > > > > > > describes
> > > > >> > > > > > > > > > > > > > > > > > behavior
> > > > >> > > > > > > > > > > > > > > > > > > with a value of "root". Should
> > > that
> > > > be
> > > > >> > > > "plain"
> > > > >> > > > > > > > instead?
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > 4. The docstring for the
> > > > >> > > > > > > "transforms.field.separator"
> > > > >> > > > > > > > > > > > property
> > > > >> > > > > > > > > > > > > > > > > > exclusively
> > > > >> > > > > > > > > > > > > > > > > > > mentions structs, but the
> > feature
> > > is
> > > > >> > > intended
> > > > >> > > > > to
> > > > >> > > > > > > work
> > > > >> > > > > > > > > > with
> > > > >> > > > > > > > > > > > maps
> > > > >> > > > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > > > > > > well.
> > > > >> > > > > > > > > > > > > > > > > > > Can we update it to reflect
> > this?
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > References:
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > [1] -
> > > > >> > https://stackoverflow.com/a/17808731
> > > > >> > > > > > > > > > > > > > > > > > > [2] -
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/connect/api/src/main/java/org/apache/kafka/connect/transforms/Transformation.java#L30
> > > > >> > > > > > > > > > > > > > > > > > > [3] -
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/7243facb8d69a7252e6b9556b5eaee13e41bab7f/clients/src/main/java/org/apache/kafka/common/Configurable.java#L26-L29
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > Chris
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 28, 2022 at 1:32
> PM
> > > > Jorge
> > > > >> > > Esteban
> > > > >> > > > > > > > Quilcate
> > > > >> > > > > > > > > > > Otoya
> > > > >> > > > > > > > > > > > <
> > > > >> > > > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > wrote:
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > Thanks, Chris!
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > 1. I'd argue
> > "this..field.child"
> > > > >> could
> > > > >> > be
> > > > >> > > > > > harder
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > grasp
> > > > >> > > > > > > > > > > > > than
> > > > >> > > > > > > > > > > > > > > > > > > > "this.field/child" +
> > separator:
> > > > "/".
> > > > >> > > > > > > > > > > > > > > > > > > > Even though this represents
> > > > >> additional
> > > > >> > > > > > > information,
> > > > >> > > > > > > > > it
> > > > >> > > > > > > > > > > > > follows
> > > > >> > > > > > > > > > > > > > a
> > > > >> > > > > > > > > > > > > > > > > > similar
> > > > >> > > > > > > > > > > > > > > > > > > > approach as the
> > > > "Flatten#delimeter"
> > > > >> > > > > > > configuration.
> > > > >> > > > > > > > > > > > > > > > > > > > I want to give the separator
> > > > >> approach
> > > > >> > > > another
> > > > >> > > > > > > try,
> > > > >> > > > > > > > > so I
> > > > >> > > > > > > > > > > > have
> > > > >> > > > > > > > > > > > > > > > updated
> > > > >> > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > KIP with the separator
> > proposal,
> > > > >> > sticking
> > > > >> > > > to
> > > > >> > > > > > > only 2
> > > > >> > > > > > > > > > > > > > alternatives
> > > > >> > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > > should hopefully cover most
> > > > >> scenarios.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > 2. Agree. KIP has been
> updated
> > > > with
> > > > >> > this
> > > > >> > > > > > > > improvement.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > 3. You're right. I have
> > updated
> > > > this
> > > > >> > > > section
> > > > >> > > > > > > > > > accordingly.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > 4. Good catch! I've replaced
> > it
> > > > with
> > > > >> > > > > > > `DropHeaders`.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > Looking forward to your
> > > feedback.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > > > > > > > > > > Jorge.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > On Wed, 9 Mar 2022 at 21:33,
> > > Chris
> > > > >> > > Egerton
> > > > >> > > > <
> > > > >> > > > > > > > > > > > > > > > fearthecellos@gmail.com>
> > > > >> > > > > > > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > Looking good! Got a few
> more
> > > > >> > thoughts.
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > 1. Sorry to revisit this,
> > but
> > > I
> > > > >> think
> > > > >> > > we
> > > > >> > > > > may
> > > > >> > > > > > > want
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > adopt
> > > > >> > > > > > > > > > > > > a
> > > > >> > > > > > > > > > > > > > > > > slightly
> > > > >> > > > > > > > > > > > > > > > > > > > > different escape syntax
> > style.
> > > > >> > > > Backslashes
> > > > >> > > > > > are
> > > > >> > > > > > > > > great,
> > > > >> > > > > > > > > > > but
> > > > >> > > > > > > > > > > > > > since
> > > > >> > > > > > > > > > > > > > > > > > they're
> > > > >> > > > > > > > > > > > > > > > > > > > > already used by JSON,
> using
> > > them
> > > > >> as
> > > > >> > an
> > > > >> > > > > escape
> > > > >> > > > > > > > > > sequence
> > > > >> > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > field
> > > > >> > > > > > > > > > > > > > > > > > > notation
> > > > >> > > > > > > > > > > > > > > > > > > > > would also lead to some
> > pretty
> > > > >> ugly
> > > > >> > > > > connector
> > > > >> > > > > > > > > > configs.
> > > > >> > > > > > > > > > > > > Anyone
> > > > >> > > > > > > > > > > > > > > > who's
> > > > >> > > > > > > > > > > > > > > > > > had
> > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > write regular expressions
> > with
> > > > >> > > > backslashes
> > > > >> > > > > in
> > > > >> > > > > > > > Java
> > > > >> > > > > > > > > is
> > > > >> > > > > > > > > > > > > > probably
> > > > >> > > > > > > > > > > > > > > > > > already
> > > > >> > > > > > > > > > > > > > > > > > > > > familiar with this:
> > > > >> > > > > > > > > > > > > > >
> > > "this\\\\.is\\\\.not\\\\.very\\\\.readable".
> > > > >> > > > > > > > > > > > > > > > > What
> > > > >> > > > > > > > > > > > > > > > > > > do
> > > > >> > > > > > > > > > > > > > > > > > > > > you think about using the
> > dot
> > > > >> > character
> > > > >> > > > to
> > > > >> > > > > > > escape
> > > > >> > > > > > > > > > > itself?
> > > > >> > > > > > > > > > > > > In
> > > > >> > > > > > > > > > > > > > > > other
> > > > >> > > > > > > > > > > > > > > > > > > words,
> > > > >> > > > > > > > > > > > > > > > > > > > > to access a single field
> > named
> > > > >> > > > > "this.field",
> > > > >> > > > > > > > > instead
> > > > >> > > > > > > > > > of
> > > > >> > > > > > > > > > > > > using
> > > > >> > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > syntax
> > > > >> > > > > > > > > > > > > > > > > > > > > "this\.field" (which in
> JSON
> > > > would
> > > > >> > have
> > > > >> > > > to
> > > > >> > > > > be
> > > > >> > > > > > > > > > expressed
> > > > >> > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > > > > > > > > > "this\\.field"),
> > > > >> > > > > > > > > > > > > > > > > > > > > we could use
> "this..field",
> > > and
> > > > >> for a
> > > > >> > > > > single
> > > > >> > > > > > > > field
> > > > >> > > > > > > > > > > named
> > > > >> > > > > > > > > > > > > > > > > > "this\field",
> > > > >> > > > > > > > > > > > > > > > > > > > > instead of using the
> syntax
> > > > >> > > "this\\field"
> > > > >> > > > > > (or,
> > > > >> > > > > > > in
> > > > >> > > > > > > > > > JSON,
> > > > >> > > > > > > > > > > > > > > > > > > "this\\\\field"),
> > > > >> > > > > > > > > > > > > > > > > > > > > we could use "this\field"
> > (or,
> > > > in
> > > > >> > JSON,
> > > > >> > > > > > > > > > "this\\field").
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > 2. Could you flesh out the
> > > > >> details on
> > > > >> > > the
> > > > >> > > > > new
> > > > >> > > > > > > > > > > > "field.style"
> > > > >> > > > > > > > > > > > > > > > > property,
> > > > >> > > > > > > > > > > > > > > > > > > > > including the type,
> default
> > > > value,
> > > > >> > > > > > importance,
> > > > >> > > > > > > > and
> > > > >> > > > > > > > > a
> > > > >> > > > > > > > > > > > > > > preliminary
> > > > >> > > > > > > > > > > > > > > > > > > > docstring?
> > > > >> > > > > > > > > > > > > > > > > > > > > See
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
> > > > >> > > > > > > > > > > > > > > > > > > > > for an example.
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > 3. Is the "Compatibility,
> > > > >> > Deprecation,
> > > > >> > > > and
> > > > >> > > > > > > > > Migration
> > > > >> > > > > > > > > > > > Plan"
> > > > >> > > > > > > > > > > > > > > > section
> > > > >> > > > > > > > > > > > > > > > > > > still
> > > > >> > > > > > > > > > > > > > > > > > > > > accurate after the latest
> > > > update?
> > > > >> > Seems
> > > > >> > > > > like
> > > > >> > > > > > > it's
> > > > >> > > > > > > > > > still
> > > > >> > > > > > > > > > > > > > written
> > > > >> > > > > > > > > > > > > > > > > with
> > > > >> > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > assumption that nested
> field
> > > > >> syntax
> > > > >> > > will
> > > > >> > > > be
> > > > >> > > > > > > > > hardcoded
> > > > >> > > > > > > > > > > or
> > > > >> > > > > > > > > > > > > > > opt-in,
> > > > >> > > > > > > > > > > > > > > > > > which
> > > > >> > > > > > > > > > > > > > > > > > > > IIUC
> > > > >> > > > > > > > > > > > > > > > > > > > > isn't the case anymore.
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > 4. Nit: The "These SMTs do
> > not
> > > > >> > require
> > > > >> > > > > nested
> > > > >> > > > > > > > > > structure
> > > > >> > > > > > > > > > > > > > > support"
> > > > >> > > > > > > > > > > > > > > > > > > section
> > > > >> > > > > > > > > > > > > > > > > > > > > mentions a "Drop" SMT. I
> > think
> > > > >> this
> > > > >> > may
> > > > >> > > > be
> > > > >> > > > > > > > > referring
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > Confluent
> > > > >> > > > > > > > > > > > > > > > > > > > Drop
> > > > >> > > > > > > > > > > > > > > > > > > > > SMT, which isn't a part of
> > > > Apache
> > > > >> > > Kafka.
> > > > >> > > > > > Should
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > > drop
> > > > >> > > > > > > > > > > > > (heh)
> > > > >> > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > SMT
> > > > >> > > > > > > > > > > > > > > > > > > > from
> > > > >> > > > > > > > > > > > > > > > > > > > > the list? Or perhaps just
> > > > replace
> > > > >> it
> > > > >> > > with
> > > > >> > > > > > > > > > > "DropHeaders",
> > > > >> > > > > > > > > > > > > > which
> > > > >> > > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > > > > > currently
> > > > >> > > > > > > > > > > > > > > > > > > > > missing from the list and
> > > > >> shouldn't
> > > > >> > > > require
> > > > >> > > > > > any
> > > > >> > > > > > > > > > > > > nested-field
> > > > >> > > > > > > > > > > > > > > > > related
> > > > >> > > > > > > > > > > > > > > > > > > > > updates?
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > Chris
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > On Mon, Feb 28, 2022 at
> 2:12
> > > PM
> > > > >> Jorge
> > > > >> > > > > Esteban
> > > > >> > > > > > > > > > Quilcate
> > > > >> > > > > > > > > > > > > Otoya
> > > > >> > > > > > > > > > > > > > <
> > > > >> > > > > > > > > > > > > > > > > > > > > quilcate.jorge@gmail.com>
> > > > wrote:
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > Thank you, Chris! and
> > sorry
> > > > for
> > > > >> the
> > > > >> > > > > delayed
> > > > >> > > > > > > > > > response.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > Please, find my comments
> > > > below:
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > On Mon, 14 Feb 2022 at
> > > 17:34,
> > > > >> Chris
> > > > >> > > > > Egerton
> > > > >> > > > > > > > > > > > > > > > > > > > <chrise@confluent.io.invalid
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Jorge,
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP!
> I'd
> > > love
> > > > >> to
> > > > >> > see
> > > > >> > > > > > support
> > > > >> > > > > > > > for
> > > > >> > > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > fields
> > > > >> > > > > > > > > > > > > > > > > > added
> > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > > out-of-the-box SMTs
> > > provided
> > > > >> with
> > > > >> > > > > > Connect.
> > > > >> > > > > > > > Here
> > > > >> > > > > > > > > > are
> > > > >> > > > > > > > > > > > my
> > > > >> > > > > > > > > > > > > > > > initial
> > > > >> > > > > > > > > > > > > > > > > > > > > thoughts:
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 1. I agree that
> there's
> > a
> > > > >> case to
> > > > >> > > be
> > > > >> > > > > made
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > > > expanding
> > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > >> > > > > > > > > > > > > > > > > > > > > with a
> > > > >> > > > > > > > > > > > > > > > > > > > > > > new config property
> for
> > > > >> > > identifying a
> > > > >> > > > > > > nested,
> > > > >> > > > > > > > > > > > > > to-be-hoisted
> > > > >> > > > > > > > > > > > > > > > > > field,
> > > > >> > > > > > > > > > > > > > > > > > > > but
> > > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > > example in the KIP
> > doesn't
> > > > >> really
> > > > >> > > > > > > demonstrate
> > > > >> > > > > > > > > why
> > > > >> > > > > > > > > > > > this
> > > > >> > > > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > > > > > > > valuable. I
> > > > >> > > > > > > > > > > > > > > > > > > > > > > think it'd be helpful
> to
> > > > >> expand
> > > > >> > the
> > > > >> > > > > > example
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > add
> > > > >> > > > > > > > > > > > > other
> > > > >> > > > > > > > > > > > > > > > fields
> > > > >> > > > > > > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > > > > > > > order
> > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > > show how adding nested
> > > field
> > > > >> > > support
> > > > >> > > > > > > enables
> > > > >> > > > > > > > > > users
> > > > >> > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > hoist a
> > > > >> > > > > > > > > > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > > > > > > > > field
> > > > >> > > > > > > > > > > > > > > > > > > > > > > without dropping other
> > > > fields
> > > > >> > from
> > > > >> > > > the
> > > > >> > > > > > > value.
> > > > >> > > > > > > > > > Maybe
> > > > >> > > > > > > > > > > > > > > something
> > > > >> > > > > > > > > > > > > > > > > > like
> > > > >> > > > > > > > > > > > > > > > > > > > > this:
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >     source =
> nested.val
> > > > >> > > > > > > > > > > > > > > > > > > > > > >     field = line
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >     value (before):
> > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > >> > > > > > > > > > > > > > > > > > > > > > >             "nested":
> {
> > > > >> > > > > > > > > > > > > > > > > > > > > > >                 "val":
> > 42,
> > > > >> > > > > > > > > > > > > > > > > > > > > > >                 "other
> > > val":
> > > > >> 96
> > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >     value (after):
> > > > >> > > > > > > > > > > > > > > > > > > > > > >         {
> > > > >> > > > > > > > > > > > > > > > > > > > > > >             "nested":
> {
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
>  "line":
> > {
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> >  "val":
> > > > 42,
> > > > >> > > > > > > > > > > > > > > > > > > > > > >                 }
> > > > >> > > > > > > > > > > > > > > > > > > > > > >                 "other
> > > val":
> > > > >> 96
> > > > >> > > > > > > > > > > > > > > > > > > > > > >             }
> > > > >> > > > > > > > > > > > > > > > > > > > > > >         }
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 2. Nit: I think
> "source"
> > > is
> > > > a
> > > > >> > > little
> > > > >> > > > > > > strange
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > new
> > > > >> > > > > > > > > > > > > > > > > > HoistField
> > > > >> > > > > > > > > > > > > > > > > > > > > > > property name. Maybe
> > > > >> "hoisted" or
> > > > >> > > > > > > > > "hoisted.field"
> > > > >> > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > more
> > > > >> > > > > > > > > > > > > > > > > > > > > > > descriptive?
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > About 1. and 2.:
> > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. The example for
> > this
> > > > SMT
> > > > >> is
> > > > >> > > > > updated
> > > > >> > > > > > > and
> > > > >> > > > > > > > > have
> > > > >> > > > > > > > > > > > added
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > `hoisted`
> > > > >> > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 3. Is there a
> reasonable
> > > use
> > > > >> case
> > > > >> > > for
> > > > >> > > > > > > > expanding
> > > > >> > > > > > > > > > > > Flatten
> > > > >> > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > > > able
> > > > >> > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten specific
> fields?
> > > My
> > > > >> > > > > understanding
> > > > >> > > > > > > is
> > > > >> > > > > > > > > that
> > > > >> > > > > > > > > > > > it's
> > > > >> > > > > > > > > > > > > > > mostly
> > > > >> > > > > > > > > > > > > > > > > > > useful
> > > > >> > > > > > > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > > > > > > writing to systems
> like
> > > > >> databases
> > > > >> > > > that
> > > > >> > > > > > > don't
> > > > >> > > > > > > > > > > support
> > > > >> > > > > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > > > > values
> > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > > > > > require everything to
> > be a
> > > > >> flat
> > > > >> > > list
> > > > >> > > > of
> > > > >> > > > > > > > > key-value
> > > > >> > > > > > > > > > > > > pairs.
> > > > >> > > > > > > > > > > > > > > > Being
> > > > >> > > > > > > > > > > > > > > > > > able
> > > > >> > > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > > flatten a nested field
> > > > >> wouldn't
> > > > >> > > > provide
> > > > >> > > > > > any
> > > > >> > > > > > > > > > > advantage
> > > > >> > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > use
> > > > >> > > > > > > > > > > > > > > > > > > > > case.
> > > > >> > > > > > > > > > > > > > > > > > > > > > > Are there other cases
> > > where
> > > > it
> > > > >> > > would?
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 4. I don't think we
> > should
> > > > >> > > > > > unconditionally
> > > > >> > > > > > > > > change
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > default
> > > > >> > > > > > > > > > > > > > > > > > > > delimiter
> > > > >> > > > > > > > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > > > > > > Flatten. It's a
> > > > >> > > > backwards-incompatible,
> > > > >> > > > > > > > > breaking
> > > > >> > > > > > > > > > > > change
> > > > >> > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > could
> > > > >> > > > > > > > > > > > > > > > > > > > > cause
> > > > >> > > > > > > > > > > > > > > > > > > > > > > headaches for users.
> It
> > > > might
> > > > >> be
> > > > >> > > > > > reasonable
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > change
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > default
> > > > >> > > > > > > > > > > > > > > > > > > > value
> > > > >> > > > > > > > > > > > > > > > > > > > > > > dynamically based on
> > > whether
> > > > >> the
> > > > >> > > user
> > > > >> > > > > has
> > > > >> > > > > > > > > > > specified a
> > > > >> > > > > > > > > > > > > > value
> > > > >> > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > "field"
> > > > >> > > > > > > > > > > > > > > > > > > > > > > property, but
> > considering
> > > > the
> > > > >> > > > > motivation
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > > changing
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > default
> > > > >> > > > > > > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > > > > > it creates conflicts
> > with
> > > > the
> > > > >> > > > > > > > to-be-introduced
> > > > >> > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > field
> > > > >> > > > > > > > > > > > > > > > > > syntax
> > > > >> > > > > > > > > > > > > > > > > > > > > (which
> > > > >> > > > > > > > > > > > > > > > > > > > > > > could arise with
> > > downstream
> > > > >> SMTs
> > > > >> > > > > > regardless
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > > > whether
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > user
> > > > >> > > > > > > > > > > > > > > > > > has
> > > > >> > > > > > > > > > > > > > > > > > > > > > > explicitly configured
> > > > Flatten
> > > > >> > with
> > > > >> > > > the
> > > > >> > > > > > > > "field"
> > > > >> > > > > > > > > > > > > > property), I
> > > > >> > > > > > > > > > > > > > > > > don't
> > > > >> > > > > > > > > > > > > > > > > > > > know
> > > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > > > > > this would be too
> useful
> > > > >> either.
> > > > >> > I
> > > > >> > > > have
> > > > >> > > > > > > some
> > > > >> > > > > > > > > > > thoughts
> > > > >> > > > > > > > > > > > > > below
> > > > >> > > > > > > > > > > > > > > > on
> > > > >> > > > > > > > > > > > > > > > > > how
> > > > >> > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > > handle possible
> > conflicts
> > > > >> between
> > > > >> > > > names
> > > > >> > > > > > > with
> > > > >> > > > > > > > > dots
> > > > >> > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > their
> > > > >> > > > > > > > > > > > > > > > > names
> > > > >> > > > > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > >> > > > > > > > > > > > > > > > > > > > > > > syntax for nested
> field
> > > > >> > references
> > > > >> > > > that
> > > > >> > > > > > > > should
> > > > >> > > > > > > > > > > > > hopefully
> > > > >> > > > > > > > > > > > > > > make
> > > > >> > > > > > > > > > > > > > > > > > > either
> > > > >> > > > > > > > > > > > > > > > > > > > > > change
> > > > >> > > > > > > > > > > > > > > > > > > > > > > unnecessary.
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > Fair enough. With the
> > > support
> > > > >> for
> > > > >> > > > nested
> > > > >> > > > > > > fields
> > > > >> > > > > > > > > in
> > > > >> > > > > > > > > > > > other
> > > > >> > > > > > > > > > > > > > > SMTs,
> > > > >> > > > > > > > > > > > > > > > > > > Flatten
> > > > >> > > > > > > > > > > > > > > > > > > > > > could stay as it is.
> > > > >> > > > > > > > > > > > > > > > > > > > > > This removes the need
> for
> > > (4)
> > > > >> > > changing
> > > > >> > > > > > > Flatten
> > > > >> > > > > > > > > > config
> > > > >> > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > > > > well.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 5. I think it's fine
> to
> > > > expand
> > > > >> > > > > > ExtractField
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > > support
> > > > >> > > > > > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > > > > > > notation,
> > > > >> > > > > > > > > > > > > > > > > > > > > > but
> > > > >> > > > > > > > > > > > > > > > > > > > > > > it might be worth
> noting
> > > in
> > > > >> the
> > > > >> > > > > rejected
> > > > >> > > > > > > > > > > alternatives
> > > > >> > > > > > > > > > > > > > > section
> > > > >> > > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > >> > > > > > > > > > > > > > > > > > > > > > > isn't strictly
> necessary
> > > > since
> > > > >> > you
> > > > >> > > > can
> > > > >> > > > > > > > replace
> > > > >> > > > > > > > > > any
> > > > >> > > > > > > > > > > > > single
> > > > >> > > > > > > > > > > > > > > > > > > invocation
> > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > > > > > that SMT that uses
> > nested
> > > > >> field
> > > > >> > > > > notation
> > > > >> > > > > > > with
> > > > >> > > > > > > > > > > > multiple
> > > > >> > > > > > > > > > > > > > > > > > invocations
> > > > >> > > > > > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > > > > > > > > > that use non-nested
> > > > notation.
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > Agree. Adding it.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 6. Nit: "RegerRouter"
> > > should
> > > > >> be
> > > > >> > > > > > > "RegexRouter"
> > > > >> > > > > > > > > in
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > list
> > > > >> > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > SMTs
> > > > >> > > > > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > > > do
> > > > >> > > > > > > > > > > > > > > > > > > > > > > not require nested
> > > structure
> > > > >> > > support.
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > Ack. Fixing it.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 7. It may be rare for
> > dots
> > > > in
> > > > >> > field
> > > > >> > > > > names
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > occur
> > > > >> > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > wild
> > > > >> > > > > > > > > > > > > > > > > > > > > (although
> > > > >> > > > > > > > > > > > > > > > > > > > > > I
> > > > >> > > > > > > > > > > > > > > > > > > > > > > wouldn't be so certain
> > of
> > > > >> this),
> > > > >> > > but
> > > > >> > > > > > unless
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > > want
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > inflict
> > > > >> > > > > > > > > > > > > > > > > > > > > headaches
> > > > >> > > > > > > > > > > > > > > > > > > > > > on
> > > > >> > > > > > > > > > > > > > > > > > > > > > > users of Flatten, I
> > think
> > > > >> we're
> > > > >> > > going
> > > > >> > > > > to
> > > > >> > > > > > > have
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > think
> > > > >> > > > > > > > > > > > > > > about
> > > > >> > > > > > > > > > > > > > > > > > > > conflicts
> > > > >> > > > > > > > > > > > > > > > > > > > > > > between dotted
> notation
> > > and
> > > > >> > > > non-nested
> > > > >> > > > > > > fields
> > > > >> > > > > > > > > > whose
> > > > >> > > > > > > > > > > > > names
> > > > >> > > > > > > > > > > > > > > > > contain
> > > > >> > > > > > > > > > > > > > > > > > > > > dots. I
> > > > >> > > > > > > > > > > > > > > > > > > > > > > don't think this is
> > > actually
> > > > >> > such a
> > > > >> > > > bad
> > > > >> > > > > > > > thing,
> > > > >> > > > > > > > > > > > though.
> > > > >> > > > > > > > > > > > > I
> > > > >> > > > > > > > > > > > > > > > agree
> > > > >> > > > > > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > > > > > > > > dotted
> > > > >> > > > > > > > > > > > > > > > > > > > > > > notation is intuitive
> > and
> > > > >> pretty
> > > > >> > > > > > > commonplace
> > > > >> > > > > > > > > (in
> > > > >> > > > > > > > > > > > tools
> > > > >> > > > > > > > > > > > > > like
> > > > >> > > > > > > > > > > > > > > > jq,
> > > > >> > > > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > > > > > > example), so I'd like
> it
> > > if
> > > > we
> > > > >> > > could
> > > > >> > > > > > stick
> > > > >> > > > > > > to
> > > > >> > > > > > > > > it.
> > > > >> > > > > > > > > > > > What
> > > > >> > > > > > > > > > > > > > > about
> > > > >> > > > > > > > > > > > > > > > > > > > > introducing
> > > > >> > > > > > > > > > > > > > > > > > > > > > > escape syntax, using a
> > > > >> backslash?
> > > > >> > > > That
> > > > >> > > > > > way,
> > > > >> > > > > > > > > users
> > > > >> > > > > > > > > > > > could
> > > > >> > > > > > > > > > > > > > > > > > > disambiguate
> > > > >> > > > > > > > > > > > > > > > > > > > > > > between "this.field"
> > > (which
> > > > >> would
> > > > >> > > > refer
> > > > >> > > > > > to
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > field
> > > > >> > > > > > > > > > > > > > > > > > "field"
> > > > >> > > > > > > > > > > > > > > > > > > > > under
> > > > >> > > > > > > > > > > > > > > > > > > > > > > the top-level "this"
> > > field),
> > > > >> and
> > > > >> > > > > > > > "this\.field"
> > > > >> > > > > > > > > > > (which
> > > > >> > > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > > > > refer
> > > > >> > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > > field named
> > "this.field").
> > > > >> Like
> > > > >> > > with
> > > > >> > > > > most
> > > > >> > > > > > > > > > languages
> > > > >> > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > > use
> > > > >> > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > backslash
> > > > >> > > > > > > > > > > > > > > > > > > > > > > for escape sequences,
> it
> > > > could
> > > > >> > also
> > > > >> > > > be
> > > > >> > > > > > used
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > escape
> > > > >> > > > > > > > > > > > > > > itself,
> > > > >> > > > > > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > event
> > > > >> > > > > > > > > > > > > > > > > > > > > > > that a field name
> > > contains a
> > > > >> > > > > backslash. I
> > > > >> > > > > > > > think
> > > > >> > > > > > > > > > > this
> > > > >> > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > more
> > > > >> > > > > > > > > > > > > > > > > > > > intuitive
> > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > > > > > simpler than, e.g.,
> > > adding a
> > > > >> new
> > > > >> > > > config
> > > > >> > > > > > > > > property
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > toggle
> > > > >> > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > delimiter
> > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > > be used when parsing
> > > nested
> > > > >> field
> > > > >> > > > > > > references.
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > I like this approach.
> > Adding
> > > > to
> > > > >> the
> > > > >> > > > KIP.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 8. I don't think we
> can
> > > > >> > > > unconditionally
> > > > >> > > > > > > turn
> > > > >> > > > > > > > > this
> > > > >> > > > > > > > > > > > > feature
> > > > >> > > > > > > > > > > > > > > on.
> > > > >> > > > > > > > > > > > > > > > > The
> > > > >> > > > > > > > > > > > > > > > > > > > risk
> > > > >> > > > > > > > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > > > > > breaking existing
> > > pipelines
> > > > >> > > > (especially
> > > > >> > > > > > > ones
> > > > >> > > > > > > > > that
> > > > >> > > > > > > > > > > > > > involve,
> > > > >> > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > > > > example, a
> > > > >> > > > > > > > > > > > > > > > > > > > > > > combination of the
> > Flatten
> > > > and
> > > > >> > Cast
> > > > >> > > > > SMTs)
> > > > >> > > > > > > is
> > > > >> > > > > > > > > > pretty
> > > > >> > > > > > > > > > > > > > high. I
> > > > >> > > > > > > > > > > > > > > > > think
> > > > >> > > > > > > > > > > > > > > > > > > > this
> > > > >> > > > > > > > > > > > > > > > > > > > > > > should be an opt-in
> > > feature,
> > > > >> at
> > > > >> > > least
> > > > >> > > > > > until
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > next
> > > > >> > > > > > > > > > > > > > major
> > > > >> > > > > > > > > > > > > > > > > > release.
> > > > >> > > > > > > > > > > > > > > > > > > > One
> > > > >> > > > > > > > > > > > > > > > > > > > > > way
> > > > >> > > > > > > > > > > > > > > > > > > > > > > we could accomplish
> this
> > > is
> > > > by
> > > > >> > > > > > introducing
> > > > >> > > > > > > a
> > > > >> > > > > > > > > new
> > > > >> > > > > > > > > > > > > > > > "field.style"
> > > > >> > > > > > > > > > > > > > > > > > > (name
> > > > >> > > > > > > > > > > > > > > > > > > > > > > obviously subject to
> > > change)
> > > > >> > > property
> > > > >> > > > > > with
> > > > >> > > > > > > > > values
> > > > >> > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > "plain"
> > > > >> > > > > > > > > > > > > > > > > > > > (default)
> > > > >> > > > > > > > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > > > > > "nested". If set to
> > > "plain"
> > > > >> then
> > > > >> > > the
> > > > >> > > > > > > current
> > > > >> > > > > > > > > > > > non-nested
> > > > >> > > > > > > > > > > > > > > > > behavior
> > > > >> > > > > > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > > > > > > used,
> > > > >> > > > > > > > > > > > > > > > > > > > > > > and if set to
> "nested",
> > > then
> > > > >> the
> > > > >> > > > > proposed
> > > > >> > > > > > > > > nested
> > > > >> > > > > > > > > > > > > behavior
> > > > >> > > > > > > > > > > > > > > is.
> > > > >> > > > > > > > > > > > > > > > > We
> > > > >> > > > > > > > > > > > > > > > > > > can
> > > > >> > > > > > > > > > > > > > > > > > > > > > > consider updating the
> > > > default
> > > > >> > value
> > > > >> > > > to
> > > > >> > > > > > > > "nested"
> > > > >> > > > > > > > > > in
> > > > >> > > > > > > > > > > a
> > > > >> > > > > > > > > > > > > > future
> > > > >> > > > > > > > > > > > > > > > > major
> > > > >> > > > > > > > > > > > > > > > > > > > > release
> > > > >> > > > > > > > > > > > > > > > > > > > > > > (or even codify that
> > plan
> > > in
> > > > >> this
> > > > >> > > > KIP,
> > > > >> > > > > if
> > > > >> > > > > > > > > there's
> > > > >> > > > > > > > > > > > > enough
> > > > >> > > > > > > > > > > > > > > > > support
> > > > >> > > > > > > > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > > > > > > > > it).
> > > > >> > > > > > > > > > > > > > > > > > > > > > > This would also leave
> > the
> > > > door
> > > > >> > open
> > > > >> > > > for
> > > > >> > > > > > > > adding
> > > > >> > > > > > > > > > > > > recursive
> > > > >> > > > > > > > > > > > > > > > > support
> > > > >> > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > SMTs
> > > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > > > > > > > > > the future by adding a
> > > > >> permitted
> > > > >> > > > value
> > > > >> > > > > of
> > > > >> > > > > > > > > > > > "recursive".
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > 9. One of the linked
> > > tickets
> > > > >> in
> > > > >> > the
> > > > >> > > > > > > > > "Motivation"
> > > > >> > > > > > > > > > > > > section,
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > >
> https://issues.apache.org/jira/browse/KAFKA-10640
> > ,
> > > > >> > > > > > > > > > > > has
> > > > >> > > > > > > > > > > > > > an
> > > > >> > > > > > > > > > > > > > > > open
> > > > >> > > > > > > > > > > > > > > > > > PR
> > > > >> > > > > > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > > > > KIP
> > > > >> > > > > > > > > > > > > > > > > > > > > > > that propose adding
> > > > recursive
> > > > >> > > support
> > > > >> > > > > to
> > > > >> > > > > > > some
> > > > >> > > > > > > > > > SMTs.
> > > > >> > > > > > > > > > > > > Have
> > > > >> > > > > > > > > > > > > > > you
> > > > >> > > > > > > > > > > > > > > > > > > > considered
> > > > >> > > > > > > > > > > > > > > > > > > > > > > this type of
> > functionality
> > > > for
> > > > >> > your
> > > > >> > > > > KIP?
> > > > >> > > > > > Or
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > your
> > > > >> > > > > > > > > > > > aim
> > > > >> > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > stick
> > > > >> > > > > > > > > > > > > > > > > > > > solely
> > > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > > > > > nested field support?
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > I like the `field.style`
> > > > >> > > configuration
> > > > >> > > > > flag
> > > > >> > > > > > > > > > approach.
> > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks for pointing out
> > the
> > > > >> > > `recursive`
> > > > >> > > > > > > > approach.
> > > > >> > > > > > > > > > > Will
> > > > >> > > > > > > > > > > > > add
> > > > >> > > > > > > > > > > > > > > > > `nested`
> > > > >> > > > > > > > > > > > > > > > > > > at
> > > > >> > > > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > moment, let's check the
> > > demand
> > > > >> for
> > > > >> > > > > > > `recursive`
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > > > consider
> > > > >> > > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > > > > > > > part
> > > > >> > > > > > > > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > > > > > > > this or another KIP.
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > I have added the
> following
> > > on
> > > > >> the
> > > > >> > > KIP:
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > >> > > > > > > > > > > > > > > > > > > > > > Future KIPs could extend
> > > this
> > > > >> > support
> > > > >> > > > > for:
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > - Recursive notation:
> > name a
> > > > >> field
> > > > >> > > and
> > > > >> > > > > > apply
> > > > >> > > > > > > it
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > all
> > > > >> > > > > > > > > > > > > > fields
> > > > >> > > > > > > > > > > > > > > > > > across
> > > > >> > > > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > schema matching that
> name.
> > > > >> > > > > > > > > > > > > > > > > > > > > > - Access to arrays:
> Adding
> > > > `[]`
> > > > >> > > > notation
> > > > >> > > > > to
> > > > >> > > > > > > > > > represent
> > > > >> > > > > > > > > > > > > > access
> > > > >> > > > > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > > > > > > arrays
> > > > >> > > > > > > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > > > > > > applying SMTs to fields
> > > within
> > > > >> an
> > > > >> > > > array.
> > > > >> > > > > > > > > > > > > > > > > > > > > > ```
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 8, 2022 at
> > > 1:23
> > > > PM
> > > > >> > > Jorge
> > > > >> > > > > > > Esteban
> > > > >> > > > > > > > > > > Quilcate
> > > > >> > > > > > > > > > > > > > > Otoya <
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > quilcate.jorge@gmail.com>
> > > > >> wrote:
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi Dev team,
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> > new
> > > > >> > > discussion
> > > > >> > > > > > thread
> > > > >> > > > > > > > on
> > > > >> > > > > > > > > > > Kafka
> > > > >> > > > > > > > > > > > > > > Connect
> > > > >> > > > > > > > > > > > > > > > > > > KIP-821:
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > This KIP is aimed to
> > > > include
> > > > >> > > > support
> > > > >> > > > > > for
> > > > >> > > > > > > > > nested
> > > > >> > > > > > > > > > > > > > > structures
> > > > >> > > > > > > > > > > > > > > > on
> > > > >> > > > > > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > > > > > > > > existing
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > SMTs — where this
> make
> > > > >> sense.
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to
> > your
> > > > >> > feedback.
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > Regards,
> > > > >> > > > > > > > > > > > > > > > > > > > > > > > Jorge.
> > > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>