You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Anup Tiwari <an...@gmail.com> on 2021/05/07 07:39:02 UTC
Mirror Maker2 Event filter capability from topic before replication
Hi Team,
I have a topic which contains JSON data and multiple user click events. I
just wanted to know if we can filter out events based on keys of JSON
before replicating it to some other kafka.
Regards,
Anup Tiwari
Re: Mirror Maker2 Event filter capability from topic before replication
Posted by Robin Moffatt <ro...@confluent.io.INVALID>.
There's also this
https://forum.confluent.io/t/kafka-connect-jmespath-expressive-content-based-record-filtering/1104
which is worth checking out - I came across it after recording that video
unfortunately but it looks like a useful predicate implementation.
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Fri, 7 May 2021 at 11:30, Tom Bentley <tb...@redhat.com> wrote:
> Just to be clear: It's only the Predicate implementation for filtering on a
> JSON key that you'd need to write. MM2 already supports SMTs and
> predicates.
>
> Alternatively
>
> https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/#_filtering_based_on_the_contents_of_a_message
> might work for you, but it's not an Apache Kafka feature currently. I'm not
> aware that anyone is currently planning on contributing a JSONPath-based
> Predicate to Apache Kafka.
>
> On Fri, May 7, 2021 at 10:06 AM Anup Tiwari <an...@gmail.com>
> wrote:
>
> > Hi Tom,
> >
> > Thanks for quick reply. As per your comments, it seems we will have to
> > build something of our own together to work with MM2.
> > So just wanted to confirm that it is not an inbuilt feature.. right? Like
> > confluent replicator provide this.
> > Also are we planning to add this in near future?
> >
> >
> > On Fri, 7 May 2021 13:33 Tom Bentley, <tb...@redhat.com> wrote:
> >
> > > Hi Anup,
> > >
> > > This should be possible using the Filter SMT in Kafka Connect together
> > with
> > > a custom Predicate which you'd have to write yourself and provide on
> the
> > > plugin path.
> > >
> > > See http://kafka.apache.org/documentation.html#connect_predicates and
> > >
> > >
> >
> https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/
> > > for info about Filter and predicates, and
> > >
> > >
> >
> https://stackoverflow.com/questions/61742209/smt-does-not-work-when-added-to-connect-mirror-maker-properties-mirrormaker-2
> > > for a MM2-specific example of SMTs
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > > On Fri, May 7, 2021 at 8:49 AM Anup Tiwari <an...@gmail.com>
> > wrote:
> > >
> > > > Hi Team,
> > > >
> > > > I have a topic which contains JSON data and multiple user click
> > events. I
> > > > just wanted to know if we can filter out events based on keys of JSON
> > > > before replicating it to some other kafka.
> > > >
> > > > Regards,
> > > > Anup Tiwari
> > > >
> > >
> >
>
Re: Mirror Maker2 Event filter capability from topic before replication
Posted by Tom Bentley <tb...@redhat.com>.
Just to be clear: It's only the Predicate implementation for filtering on a
JSON key that you'd need to write. MM2 already supports SMTs and
predicates.
Alternatively
https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/#_filtering_based_on_the_contents_of_a_message
might work for you, but it's not an Apache Kafka feature currently. I'm not
aware that anyone is currently planning on contributing a JSONPath-based
Predicate to Apache Kafka.
On Fri, May 7, 2021 at 10:06 AM Anup Tiwari <an...@gmail.com> wrote:
> Hi Tom,
>
> Thanks for quick reply. As per your comments, it seems we will have to
> build something of our own together to work with MM2.
> So just wanted to confirm that it is not an inbuilt feature.. right? Like
> confluent replicator provide this.
> Also are we planning to add this in near future?
>
>
> On Fri, 7 May 2021 13:33 Tom Bentley, <tb...@redhat.com> wrote:
>
> > Hi Anup,
> >
> > This should be possible using the Filter SMT in Kafka Connect together
> with
> > a custom Predicate which you'd have to write yourself and provide on the
> > plugin path.
> >
> > See http://kafka.apache.org/documentation.html#connect_predicates and
> >
> >
> https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/
> > for info about Filter and predicates, and
> >
> >
> https://stackoverflow.com/questions/61742209/smt-does-not-work-when-added-to-connect-mirror-maker-properties-mirrormaker-2
> > for a MM2-specific example of SMTs
> >
> > Kind regards,
> >
> > Tom
> >
> > On Fri, May 7, 2021 at 8:49 AM Anup Tiwari <an...@gmail.com>
> wrote:
> >
> > > Hi Team,
> > >
> > > I have a topic which contains JSON data and multiple user click
> events. I
> > > just wanted to know if we can filter out events based on keys of JSON
> > > before replicating it to some other kafka.
> > >
> > > Regards,
> > > Anup Tiwari
> > >
> >
>
Re: Mirror Maker2 Event filter capability from topic before replication
Posted by Anup Tiwari <an...@gmail.com>.
Hi Tom,
Thanks for quick reply. As per your comments, it seems we will have to
build something of our own together to work with MM2.
So just wanted to confirm that it is not an inbuilt feature.. right? Like
confluent replicator provide this.
Also are we planning to add this in near future?
On Fri, 7 May 2021 13:33 Tom Bentley, <tb...@redhat.com> wrote:
> Hi Anup,
>
> This should be possible using the Filter SMT in Kafka Connect together with
> a custom Predicate which you'd have to write yourself and provide on the
> plugin path.
>
> See http://kafka.apache.org/documentation.html#connect_predicates and
>
> https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/
> for info about Filter and predicates, and
>
> https://stackoverflow.com/questions/61742209/smt-does-not-work-when-added-to-connect-mirror-maker-properties-mirrormaker-2
> for a MM2-specific example of SMTs
>
> Kind regards,
>
> Tom
>
> On Fri, May 7, 2021 at 8:49 AM Anup Tiwari <an...@gmail.com> wrote:
>
> > Hi Team,
> >
> > I have a topic which contains JSON data and multiple user click events. I
> > just wanted to know if we can filter out events based on keys of JSON
> > before replicating it to some other kafka.
> >
> > Regards,
> > Anup Tiwari
> >
>
Re: Mirror Maker2 Event filter capability from topic before replication
Posted by Tom Bentley <tb...@redhat.com>.
Hi Anup,
This should be possible using the Filter SMT in Kafka Connect together with
a custom Predicate which you'd have to write yourself and provide on the
plugin path.
See http://kafka.apache.org/documentation.html#connect_predicates and
https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/
for info about Filter and predicates, and
https://stackoverflow.com/questions/61742209/smt-does-not-work-when-added-to-connect-mirror-maker-properties-mirrormaker-2
for a MM2-specific example of SMTs
Kind regards,
Tom
On Fri, May 7, 2021 at 8:49 AM Anup Tiwari <an...@gmail.com> wrote:
> Hi Team,
>
> I have a topic which contains JSON data and multiple user click events. I
> just wanted to know if we can filter out events based on keys of JSON
> before replicating it to some other kafka.
>
> Regards,
> Anup Tiwari
>