You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Subramanian Karunanithi <su...@gmail.com> on 2016/04/26 17:07:23 UTC
Filter plugins in Kafka
Hi,
Do we have any plugin which is available, which can be used as a filtering
mechanism on the data it's working on?
Regards,
Subramanian. K
Re: Filter plugins in Kafka
Posted by Lohith Samaga M <Lo...@mphasis.com>.
Hi
Using Storm would be another way. This will scale as well.
Spark streaming would fit as well
It all depends on the complexity of the filter and any additional processing required.
HTH
Lohith
Sent from my Sony Xperia™ smartphone
---- Gerard Klijs wrote ----
Using kafka streams is one way, I used camel before with kafka, which also
has a nice way of using filters.
On Fri, Apr 29, 2016 at 1:51 PM Subramanian Karunanithi <su...@gmail.com>
wrote:
> Hi,
>
> When a stream of data passes through Kafka, wanted to apply the filter and
> then let that message pass through to partitions.
>
> Regards,
> Subramanian. K
> On Apr 26, 2016 12:33, "Marko Bonaći" <ma...@sematext.com> wrote:
>
> > Instantly reminded me of Streams API, where you can use Java8 streams
> > semantics (filter being one of them) to do the first thing in Gouzhang's
> > response (filter messages from one topic into another - I assume that's
> > what you were looking for).
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hi Subramanian,
> > >
> > > Could you elaborate a bit more on "filtering"? Do you want to read raw
> > data
> > > from Kafka, and send the filtered data back to Kafka as a separate
> topic,
> > > or do you want to read raw data from an external service and send the
> > > filtered data into Kafka?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > sububack@gmail.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Do we have any plugin which is available, which can be used as a
> > > filtering
> > > > mechanism on the data it's working on?
> > > >
> > > > Regards,
> > > > Subramanian. K
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.
Re: Filter plugins in Kafka
Posted by Gerard Klijs <ge...@dizzit.com>.
Using kafka streams is one way, I used camel before with kafka, which also
has a nice way of using filters.
On Fri, Apr 29, 2016 at 1:51 PM Subramanian Karunanithi <su...@gmail.com>
wrote:
> Hi,
>
> When a stream of data passes through Kafka, wanted to apply the filter and
> then let that message pass through to partitions.
>
> Regards,
> Subramanian. K
> On Apr 26, 2016 12:33, "Marko Bonaći" <ma...@sematext.com> wrote:
>
> > Instantly reminded me of Streams API, where you can use Java8 streams
> > semantics (filter being one of them) to do the first thing in Gouzhang's
> > response (filter messages from one topic into another - I assume that's
> > what you were looking for).
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hi Subramanian,
> > >
> > > Could you elaborate a bit more on "filtering"? Do you want to read raw
> > data
> > > from Kafka, and send the filtered data back to Kafka as a separate
> topic,
> > > or do you want to read raw data from an external service and send the
> > > filtered data into Kafka?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > sububack@gmail.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Do we have any plugin which is available, which can be used as a
> > > filtering
> > > > mechanism on the data it's working on?
> > > >
> > > > Regards,
> > > > Subramanian. K
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Re: Filter plugins in Kafka
Posted by Subramanian Karunanithi <su...@gmail.com>.
Thanks everyone, shall try these options.
Regards,
Subramanian. K
On Mon, May 2, 2016 at 9:43 AM, Andrew Otto <ot...@wikimedia.org> wrote:
> If you want something really simple and hacky, you could use kafkatee[1]
> and kafkacat[2] together:
>
> kafkatee.conf:
>
> input [encoding=string] pipe tail -f a.log
> output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic
>
> [1] https://github.com/wikimedia/analytics-kafkatee
> [2] https://github.com/edenhill/kafkacat
>
> On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <ra...@gruchalski.com>
> wrote:
>
> > Apache Samza is the way to go. Never used Kafka Streams so no opinion on
> > that one.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Best regards,
> > Radek Gruchalski
> > radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> > radek@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (
> > http://de.linkedin.com/in/radgruchalski/)
> >
> > Confidentiality:
> > This communication is intended for the above-named person and may be
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor
> > must you copy or show it to anyone; please delete/destroy and inform the
> > sender immediately.
> >
> >
> >
> > On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote:
> >
> > > Hi,
> > >
> > > When a stream of data passes through Kafka, wanted to apply the filter
> > and
> > > then let that message pass through to partitions.
> > >
> > > Regards,
> > > Subramanian. K
> > > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bonaci@sematext.com
> > (mailto:marko.bonaci@sematext.com)> wrote:
> > >
> > > > Instantly reminded me of Streams API, where you can use Java8 streams
> > > > semantics (filter being one of them) to do the first thing in
> > Gouzhang's
> > > > response (filter messages from one topic into another - I assume
> that's
> > > > what you were looking for).
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangguoz@gmail.com
> > (mailto:wangguoz@gmail.com)> wrote:
> > > >
> > > > > Hi Subramanian,
> > > > >
> > > > > Could you elaborate a bit more on "filtering"? Do you want to read
> > raw
> > > > data
> > > > > from Kafka, and send the filtered data back to Kafka as a separate
> > topic,
> > > > > or do you want to read raw data from an external service and send
> the
> > > > > filtered data into Kafka?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > > > sububack@gmail.com (mailto:sububack@gmail.com)
> > > > > > wrote:
> > > > >
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Do we have any plugin which is available, which can be used as a
> > > > > filtering
> > > > > > mechanism on the data it's working on?
> > > > > >
> > > > > > Regards,
> > > > > > Subramanian. K
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
Re: Filter plugins in Kafka
Posted by Andrew Otto <ot...@wikimedia.org>.
If you want something really simple and hacky, you could use kafkatee[1]
and kafkacat[2] together:
kafkatee.conf:
input [encoding=string] pipe tail -f a.log
output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic
[1] https://github.com/wikimedia/analytics-kafkatee
[2] https://github.com/edenhill/kafkacat
On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <ra...@gruchalski.com>
wrote:
> Apache Samza is the way to go. Never used Kafka Streams so no opinion on
> that one.
>
>
>
>
>
>
>
>
>
>
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> radek@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/)
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
>
> On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote:
>
> > Hi,
> >
> > When a stream of data passes through Kafka, wanted to apply the filter
> and
> > then let that message pass through to partitions.
> >
> > Regards,
> > Subramanian. K
> > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bonaci@sematext.com
> (mailto:marko.bonaci@sematext.com)> wrote:
> >
> > > Instantly reminded me of Streams API, where you can use Java8 streams
> > > semantics (filter being one of them) to do the first thing in
> Gouzhang's
> > > response (filter messages from one topic into another - I assume that's
> > > what you were looking for).
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangguoz@gmail.com
> (mailto:wangguoz@gmail.com)> wrote:
> > >
> > > > Hi Subramanian,
> > > >
> > > > Could you elaborate a bit more on "filtering"? Do you want to read
> raw
> > > data
> > > > from Kafka, and send the filtered data back to Kafka as a separate
> topic,
> > > > or do you want to read raw data from an external service and send the
> > > > filtered data into Kafka?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > > sububack@gmail.com (mailto:sububack@gmail.com)
> > > > > wrote:
> > > >
> > > >
> > > > > Hi,
> > > > >
> > > > > Do we have any plugin which is available, which can be used as a
> > > > filtering
> > > > > mechanism on the data it's working on?
> > > > >
> > > > > Regards,
> > > > > Subramanian. K
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> >
> >
> >
>
>
>
Re: Filter plugins in Kafka
Posted by Rad Gruchalski <ra...@gruchalski.com>.
Apache Samza is the way to go. Never used Kafka Streams so no opinion on that one.
Best regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote:
> Hi,
>
> When a stream of data passes through Kafka, wanted to apply the filter and
> then let that message pass through to partitions.
>
> Regards,
> Subramanian. K
> On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bonaci@sematext.com (mailto:marko.bonaci@sematext.com)> wrote:
>
> > Instantly reminded me of Streams API, where you can use Java8 streams
> > semantics (filter being one of them) to do the first thing in Gouzhang's
> > response (filter messages from one topic into another - I assume that's
> > what you were looking for).
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangguoz@gmail.com (mailto:wangguoz@gmail.com)> wrote:
> >
> > > Hi Subramanian,
> > >
> > > Could you elaborate a bit more on "filtering"? Do you want to read raw
> > data
> > > from Kafka, and send the filtered data back to Kafka as a separate topic,
> > > or do you want to read raw data from an external service and send the
> > > filtered data into Kafka?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > sububack@gmail.com (mailto:sububack@gmail.com)
> > > > wrote:
> > >
> > >
> > > > Hi,
> > > >
> > > > Do we have any plugin which is available, which can be used as a
> > > filtering
> > > > mechanism on the data it's working on?
> > > >
> > > > Regards,
> > > > Subramanian. K
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
>
>
>
Re: Filter plugins in Kafka
Posted by Subramanian Karunanithi <su...@gmail.com>.
Hi,
When a stream of data passes through Kafka, wanted to apply the filter and
then let that message pass through to partitions.
Regards,
Subramanian. K
On Apr 26, 2016 12:33, "Marko Bonaći" <ma...@sematext.com> wrote:
> Instantly reminded me of Streams API, where you can use Java8 streams
> semantics (filter being one of them) to do the first thing in Gouzhang's
> response (filter messages from one topic into another - I assume that's
> what you were looking for).
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Subramanian,
> >
> > Could you elaborate a bit more on "filtering"? Do you want to read raw
> data
> > from Kafka, and send the filtered data back to Kafka as a separate topic,
> > or do you want to read raw data from an external service and send the
> > filtered data into Kafka?
> >
> >
> > Guozhang
> >
> >
> > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > sububack@gmail.com
> > > wrote:
> >
> > > Hi,
> > >
> > > Do we have any plugin which is available, which can be used as a
> > filtering
> > > mechanism on the data it's working on?
> > >
> > > Regards,
> > > Subramanian. K
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: Filter plugins in Kafka
Posted by Marko Bonaći <ma...@sematext.com>.
Instantly reminded me of Streams API, where you can use Java8 streams
semantics (filter being one of them) to do the first thing in Gouzhang's
response (filter messages from one topic into another - I assume that's
what you were looking for).
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Subramanian,
>
> Could you elaborate a bit more on "filtering"? Do you want to read raw data
> from Kafka, and send the filtered data back to Kafka as a separate topic,
> or do you want to read raw data from an external service and send the
> filtered data into Kafka?
>
>
> Guozhang
>
>
> On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> sububack@gmail.com
> > wrote:
>
> > Hi,
> >
> > Do we have any plugin which is available, which can be used as a
> filtering
> > mechanism on the data it's working on?
> >
> > Regards,
> > Subramanian. K
> >
>
>
>
> --
> -- Guozhang
>
Re: Filter plugins in Kafka
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Subramanian,
Could you elaborate a bit more on "filtering"? Do you want to read raw data
from Kafka, and send the filtered data back to Kafka as a separate topic,
or do you want to read raw data from an external service and send the
filtered data into Kafka?
Guozhang
On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <sububack@gmail.com
> wrote:
> Hi,
>
> Do we have any plugin which is available, which can be used as a filtering
> mechanism on the data it's working on?
>
> Regards,
> Subramanian. K
>
--
-- Guozhang