Kafka Connect Skip Messages

[mbschorer <bs...@tyntec.com>]

Hi!

I am evaluating the suitability of Kafka Connect to leverage our internal event sourcing system.
Since custom deserializers are supported, it looks quite promising so far.

One open question is, whether Kafka Connect is capable of filtering out, i.e. skipping certain messages based on conditions on the value.
Or does it pursue a take all or nothing approach?

All I could find so far is single message transform, which didn't reveal this feature.
Does anybody in this mailing list have more knowledge regarding this?

Thanks in advance!
Martin

Re: Kafka Connect Skip Messages

[Robin Moffatt <ro...@confluent.io>]

It should be possible using a custom Single Message Transform (
https://docs.confluent.io/current/connect/javadocs/org/apache/kafka/connect/transforms/Transformation.html).
There is a good talk here about when SMTs are appropriate (and when not):
https://kafka-summit.org/sessions/single-message-transformations-not-transformations-youre-looking/

You might also want to look at Kafka Streams for applying logic to a stream
before you ingest it with Kafka Connect.


-- 

Robin Moffatt | Developer Advocate | robin@confluent.io | @rmoff


On Fri, 25 Jan 2019 at 13:29, Sönke Liebau
<so...@opencore.com.invalid> wrote:

> Hi Martin,
>
> I don't think that Connect offers this functionality as is. However
> you can implement custom SMTs, which should allow you to do what need.
> Alternatively you could write a ConsumerInterceptor that filters out
> messages based on your criteria - but: this would affect all running
> Connectors as this setting can only be set for the entire Connect
> worker.
> Depending on your exact needs one of the two approaches might be
> preferable. If this filter has to be applied to everything that
> Connect reads than the Interceptor might be preferrable, as you'll
> only have to define it once and then not specify it again for every
> connector.
>
> Hope that helps.
>
> Best regards,
> Sönke
>
> On Fri, Jan 25, 2019 at 1:58 PM mbschorer <bs...@tyntec.com> wrote:
> >
> > Hi!
> >
> > I am evaluating the suitability of Kafka Connect to leverage our
> internal event sourcing system.
> > Since custom deserializers are supported, it looks quite promising so
> far.
> >
> > One open question is, whether Kafka Connect is capable of filtering out,
> i.e. skipping certain messages based on conditions on the value.
> > Or does it pursue a take all or nothing approach?
> >
> > All I could find so far is single message transform, which didn't reveal
> this feature.
> > Does anybody in this mailing list have more knowledge regarding this?
> >
> > Thanks in advance!
> > Martin
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>

Re: Kafka Connect Skip Messages

[Sönke Liebau <so...@opencore.com.INVALID>]

Hi Martin,

I don't think that Connect offers this functionality as is. However
you can implement custom SMTs, which should allow you to do what need.
Alternatively you could write a ConsumerInterceptor that filters out
messages based on your criteria - but: this would affect all running
Connectors as this setting can only be set for the entire Connect
worker.
Depending on your exact needs one of the two approaches might be
preferable. If this filter has to be applied to everything that
Connect reads than the Interceptor might be preferrable, as you'll
only have to define it once and then not specify it again for every
connector.

Hope that helps.

Best regards,
Sönke

On Fri, Jan 25, 2019 at 1:58 PM mbschorer <bs...@tyntec.com> wrote:
>
> Hi!
>
> I am evaluating the suitability of Kafka Connect to leverage our internal event sourcing system.
> Since custom deserializers are supported, it looks quite promising so far.
>
> One open question is, whether Kafka Connect is capable of filtering out, i.e. skipping certain messages based on conditions on the value.
> Or does it pursue a take all or nothing approach?
>
> All I could find so far is single message transform, which didn't reveal this feature.
> Does anybody in this mailing list have more knowledge regarding this?
>
> Thanks in advance!
> Martin
>


-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany