Re: Filter plugins in Kafka

[Andrew Otto <ot...@wikimedia.org>]

If you want something really simple and hacky, you could use kafkatee[1]
and kafkacat[2] together:

kafkatee.conf:

input [encoding=string] pipe tail -f a.log
output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic

[1] https://github.com/wikimedia/analytics-kafkatee
[2] https://github.com/edenhill/kafkacat

On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <ra...@gruchalski.com>
wrote:

> Apache Samza is the way to go. Never used Kafka Streams so no opinion on
> that one.
>
>
>
>
>
>
>
>
>
>
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> radek@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/)
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
>
> On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote:
>
> > Hi,
> >
> > When a stream of data passes through Kafka, wanted to apply the filter
> and
> > then let that message pass through to partitions.
> >
> > Regards,
> > Subramanian. K
> > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bonaci@sematext.com
> (mailto:marko.bonaci@sematext.com)> wrote:
> >
> > > Instantly reminded me of Streams API, where you can use Java8 streams
> > > semantics (filter being one of them) to do the first thing in
> Gouzhang's
> > > response (filter messages from one topic into another - I assume that's
> > > what you were looking for).
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangguoz@gmail.com
> (mailto:wangguoz@gmail.com)> wrote:
> > >
> > > > Hi Subramanian,
> > > >
> > > > Could you elaborate a bit more on "filtering"? Do you want to read
> raw
> > > data
> > > > from Kafka, and send the filtered data back to Kafka as a separate
> topic,
> > > > or do you want to read raw data from an external service and send the
> > > > filtered data into Kafka?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > > sububack@gmail.com (mailto:sububack@gmail.com)
> > > > > wrote:
> > > >
> > > >
> > > > > Hi,
> > > > >
> > > > > Do we have any plugin which is available, which can be used as a
> > > > filtering
> > > > > mechanism on the data it's working on?
> > > > >
> > > > > Regards,
> > > > > Subramanian. K
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> >
> >
> >
>
>
>

Re: Filter plugins in Kafka

[Subramanian Karunanithi <su...@gmail.com>]

Thanks everyone, shall try these options.

Regards,
Subramanian. K

On Mon, May 2, 2016 at 9:43 AM, Andrew Otto <ot...@wikimedia.org> wrote:

> If you want something really simple and hacky, you could use kafkatee[1]
> and kafkacat[2] together:
>
> kafkatee.conf:
>
> input [encoding=string] pipe tail -f a.log
> output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic
>
> [1] https://github.com/wikimedia/analytics-kafkatee
> [2] https://github.com/edenhill/kafkacat
>
> On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <ra...@gruchalski.com>
> wrote:
>
> > Apache Samza is the way to go. Never used Kafka Streams so no opinion on
> > that one.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Best regards,
> > Radek Gruchalski
> > radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> > radek@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (
> > http://de.linkedin.com/in/radgruchalski/)
> >
> > Confidentiality:
> > This communication is intended for the above-named person and may be
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor
> > must you copy or show it to anyone; please delete/destroy and inform the
> > sender immediately.
> >
> >
> >
> > On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote:
> >
> > > Hi,
> > >
> > > When a stream of data passes through Kafka, wanted to apply the filter
> > and
> > > then let that message pass through to partitions.
> > >
> > > Regards,
> > > Subramanian. K
> > > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bonaci@sematext.com
> > (mailto:marko.bonaci@sematext.com)> wrote:
> > >
> > > > Instantly reminded me of Streams API, where you can use Java8 streams
> > > > semantics (filter being one of them) to do the first thing in
> > Gouzhang's
> > > > response (filter messages from one topic into another - I assume
> that's
> > > > what you were looking for).
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangguoz@gmail.com
> > (mailto:wangguoz@gmail.com)> wrote:
> > > >
> > > > > Hi Subramanian,
> > > > >
> > > > > Could you elaborate a bit more on "filtering"? Do you want to read
> > raw
> > > > data
> > > > > from Kafka, and send the filtered data back to Kafka as a separate
> > topic,
> > > > > or do you want to read raw data from an external service and send
> the
> > > > > filtered data into Kafka?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > > > sububack@gmail.com (mailto:sububack@gmail.com)
> > > > > > wrote:
> > > > >
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Do we have any plugin which is available, which can be used as a
> > > > > filtering
> > > > > > mechanism on the data it's working on?
> > > > > >
> > > > > > Regards,
> > > > > > Subramanian. K
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>