You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Liam Clarke-Hutchinson <li...@adscale.co.nz> on 2020/05/08 00:14:14 UTC
Kafka Connect SMT to insert key into message
Hi all,
I've been double checking the docs, and before I write a custom transform,
am I correct that there's no supported way in the InsertField SMT to insert
the key as a field of the value?
Cheers,
Liam Clarke-Hutchinson
Re: Kafka Connect SMT to insert key into message
Posted by Andrew Schofield <an...@live.com>.
Hi,
I think you're right. There's a ValueToKey transformation, but not a KeyToValue.
I think that would be a better fit than adding to InsertField because you always
use the concrete transformations InsertField$Key and InsertField$Value, and they
have the same configuration.
Probably a fairly simple SMT to write yourself too.
Cheers,
Andrew Schofield
On 08/05/2020, 03:04, "Liam Clarke-Hutchinson" <li...@adscale.co.nz> wrote:
> So looking at the code of InsertField, it looks like there isn't an obvious
> way, unless there's some way to chaining SMTs to achieve it.
>
> Question then is, is it worth adding it to the InsertField SMT? The change
> looks reasonably straightforward, and I'm happy to do a PR if it fits with
> the aims of the project.
>
> My use case were records keyed on the topic by user id, and I wanted to
> persist them into S3 with the user id as part of the one JSON object per
> line data.
>
> Thanks,
>
> Liam Clarke-Hutchinson
>
> On Fri, May 8, 2020 at 12:14 PM Liam Clarke-Hutchinson <
> liam.clarke@adscale.co.nz> wrote:
>
> > Hi all,
> >
> > I've been double checking the docs, and before I write a custom transform,
> > am I correct that there's no supported way in the InsertField SMT to insert
> > the key as a field of the value?
> >
> > Cheers,
> >
> > Liam Clarke-Hutchinson
> >
Re: Kafka Connect SMT to insert key into message
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
So looking at the code of InsertField, it looks like there isn't an obvious
way, unless there's some way to chaining SMTs to achieve it.
Question then is, is it worth adding it to the InsertField SMT? The change
looks reasonably straightforward, and I'm happy to do a PR if it fits with
the aims of the project.
My use case were records keyed on the topic by user id, and I wanted to
persist them into S3 with the user id as part of the one JSON object per
line data.
Thanks,
Liam Clarke-Hutchinson
On Fri, May 8, 2020 at 12:14 PM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> Hi all,
>
> I've been double checking the docs, and before I write a custom transform,
> am I correct that there's no supported way in the InsertField SMT to insert
> the key as a field of the value?
>
> Cheers,
>
> Liam Clarke-Hutchinson
>