You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by raman gugnani <ra...@gmail.com> on 2021/03/10 08:38:13 UTC
Filter Kafka Messages
HI Team,
I want to filter out some of the kafka messages from source kafka cluster
to destination kafka cluster via some logic.
Logics to filter
1. If possible to filter data on the basis of static data.
2. If possible, filter data dynamically based on a custom provider (some
jar or some other way around).
Looking to do the same via any of the below features
1. Kafka mirror
2. Kafka Connect
3. Kafka Streams.
--
Raman Gugnani
Re: Filter Kafka Messages
Posted by Tom Bentley <tb...@redhat.com>.
Hi Raman,
I believe for MM2 you just specify the transforms and predicates on each
mirror source connector, like this:
us-west->us-east.transforms=myFilter
us-west->us-east.transforms.myFilter.type=Filter
And similarly with the predicates, as described in Robin's blog or the
documentation.
Kind regards,
Tom
On Wed, Mar 10, 2021 at 10:13 AM Robin Moffatt <ro...@confluent.io.invalid>
wrote:
> You can find details of the Filter function and its use here:
>
> https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/
>
> Something I need to add to this page is a new JMESPath-based Predicate that
> I found recently which looks really useful. The author wrote details and
> example of it here:
>
> https://forum.confluent.io/t/kafka-connect-jmespath-expressive-content-based-record-filtering/1104
>
>
> --
>
> Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
>
>
> On Wed, 10 Mar 2021 at 09:55, raman gugnani <ra...@gmail.com>
> wrote:
>
> > HI Tom,
> >
> > For which connector class connector do I need to submit ?
> > Can you share some samples for the same?
> >
> > On Wed, 10 Mar 2021 at 14:32, Tom Bentley <tb...@redhat.com> wrote:
> >
> > > Kafka Connect has the Filter SMT which can be used with a predicate to
> > > filter out certain messages, see
> > > https://kafka.apache.org/documentation/#connect_predicates
> > >
> > > Kinds regards,
> > >
> > > Tom
> > >
> > > On Wed, Mar 10, 2021 at 8:46 AM raman gugnani <
> > ramangugnani.007@gmail.com>
> > > wrote:
> > >
> > > > HI Team,
> > > >
> > > > I want to filter out some of the kafka messages from source kafka
> > cluster
> > > > to destination kafka cluster via some logic.
> > > >
> > > > Logics to filter
> > > >
> > > > 1. If possible to filter data on the basis of static data.
> > > > 2. If possible, filter data dynamically based on a custom provider
> > > (some
> > > > jar or some other way around).
> > > >
> > > >
> > > > Looking to do the same via any of the below features
> > > >
> > > > 1. Kafka mirror
> > > > 2. Kafka Connect
> > > > 3. Kafka Streams.
> > > >
> > > > --
> > > > Raman Gugnani
> > > >
> > >
> >
> >
> > --
> > Raman Gugnani
> >
>
Re: Filter Kafka Messages
Posted by Robin Moffatt <ro...@confluent.io.INVALID>.
You can find details of the Filter function and its use here:
https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/
Something I need to add to this page is a new JMESPath-based Predicate that
I found recently which looks really useful. The author wrote details and
example of it here:
https://forum.confluent.io/t/kafka-connect-jmespath-expressive-content-based-record-filtering/1104
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Wed, 10 Mar 2021 at 09:55, raman gugnani <ra...@gmail.com>
wrote:
> HI Tom,
>
> For which connector class connector do I need to submit ?
> Can you share some samples for the same?
>
> On Wed, 10 Mar 2021 at 14:32, Tom Bentley <tb...@redhat.com> wrote:
>
> > Kafka Connect has the Filter SMT which can be used with a predicate to
> > filter out certain messages, see
> > https://kafka.apache.org/documentation/#connect_predicates
> >
> > Kinds regards,
> >
> > Tom
> >
> > On Wed, Mar 10, 2021 at 8:46 AM raman gugnani <
> ramangugnani.007@gmail.com>
> > wrote:
> >
> > > HI Team,
> > >
> > > I want to filter out some of the kafka messages from source kafka
> cluster
> > > to destination kafka cluster via some logic.
> > >
> > > Logics to filter
> > >
> > > 1. If possible to filter data on the basis of static data.
> > > 2. If possible, filter data dynamically based on a custom provider
> > (some
> > > jar or some other way around).
> > >
> > >
> > > Looking to do the same via any of the below features
> > >
> > > 1. Kafka mirror
> > > 2. Kafka Connect
> > > 3. Kafka Streams.
> > >
> > > --
> > > Raman Gugnani
> > >
> >
>
>
> --
> Raman Gugnani
>
Re: Filter Kafka Messages
Posted by raman gugnani <ra...@gmail.com>.
HI Tom,
For which connector class connector do I need to submit ?
Can you share some samples for the same?
On Wed, 10 Mar 2021 at 14:32, Tom Bentley <tb...@redhat.com> wrote:
> Kafka Connect has the Filter SMT which can be used with a predicate to
> filter out certain messages, see
> https://kafka.apache.org/documentation/#connect_predicates
>
> Kinds regards,
>
> Tom
>
> On Wed, Mar 10, 2021 at 8:46 AM raman gugnani <ra...@gmail.com>
> wrote:
>
> > HI Team,
> >
> > I want to filter out some of the kafka messages from source kafka cluster
> > to destination kafka cluster via some logic.
> >
> > Logics to filter
> >
> > 1. If possible to filter data on the basis of static data.
> > 2. If possible, filter data dynamically based on a custom provider
> (some
> > jar or some other way around).
> >
> >
> > Looking to do the same via any of the below features
> >
> > 1. Kafka mirror
> > 2. Kafka Connect
> > 3. Kafka Streams.
> >
> > --
> > Raman Gugnani
> >
>
--
Raman Gugnani
Re: Filter Kafka Messages
Posted by Tom Bentley <tb...@redhat.com>.
Kafka Connect has the Filter SMT which can be used with a predicate to
filter out certain messages, see
https://kafka.apache.org/documentation/#connect_predicates
Kinds regards,
Tom
On Wed, Mar 10, 2021 at 8:46 AM raman gugnani <ra...@gmail.com>
wrote:
> HI Team,
>
> I want to filter out some of the kafka messages from source kafka cluster
> to destination kafka cluster via some logic.
>
> Logics to filter
>
> 1. If possible to filter data on the basis of static data.
> 2. If possible, filter data dynamically based on a custom provider (some
> jar or some other way around).
>
>
> Looking to do the same via any of the below features
>
> 1. Kafka mirror
> 2. Kafka Connect
> 3. Kafka Streams.
>
> --
> Raman Gugnani
>