sFlow/NetFlow/Pcap Plugin for Kafka Producer

[Aris Risdianto <ar...@smartx.kr>]

​Hello,


​Is there any plan or implementation to use Kafka for delivering
sFlow/NetFlow/Pcap messages?


Best Regards,
Aris.

Re: sFlow/NetFlow/Pcap Plugin for Kafka Producer

[Michael Noll <mi...@confluent.io>]

Aris,

I am not aware of an out of the box tool for Pcap->Kafka ingestion (in my
case back then we wrote our own).  Maybe others know.



On Monday, October 10, 2016, Aris Risdianto <ar...@smartx.kr> wrote:

> Thank you for answer Michael.
>
> Actually, I have made a simple producer from Pcap to Kafka. Since it is not
> structured, so it is difficult for further processing by a consumer. But, I
> will take a look at Avro as you mentioned.
>
> I just wondering, if there are any proper implementation for this
> requirement, because I couldn't find any tool in the kafka ecosystem page.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
>
> Best Regards,
> Aris.
>
>
> On Mon, Oct 10, 2016 at 6:55 PM, Michael Noll <michael@confluent.io
> <javascript:;>> wrote:
>
> > Aris,
> >
> > even today you can already use Kafka to deliver Netflow/Pcap/etc.
> messages,
> > and people are already using it for that (I did that in previous projects
> > of mine, too).
> >
> > Simply encode your Pcap/... messages appropriately (I'd recommend to
> take a
> > look at Avro, which allows you to structure your data similar to e.g.
> > Pcap's native format [1]), and then write the encoded messages to Kafka.
> > Your downstream applications can then read the encoded messages back from
> > Kafka, decode, and commence processing.
> >
> > That was a brief summary to get you started, feel free to take a look at
> > the Apache Kafka docs at kafka.apache.org and/or ask further questions
> > here.
> >
> > -Michael
> >
> >
> >
> >
> > [1] https://wiki.wireshark.org/Development/LibpcapFileFormat
> >
> > On Mon, Oct 10, 2016 at 11:19 AM, Aris Risdianto <aris@smartx.kr
> <javascript:;>> wrote:
> >
> > > ​Hello,
> > >
> > >
> > > ​Is there any plan or implementation to use Kafka for delivering
> > > sFlow/NetFlow/Pcap messages?
> > >
> > >
> > > Best Regards,
> > > Aris.
> > >
> >
>


-- 
*Michael G. Noll*
Product Manager | Confluent
+1 650 453 5860 | @miguno <https://twitter.com/miguno>
Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
<http://www.confluent.io/blog>

Re: sFlow/NetFlow/Pcap Plugin for Kafka Producer

[Aris Risdianto <ar...@smartx.kr>]

Thank you for answer Michael.

Actually, I have made a simple producer from Pcap to Kafka. Since it is not
structured, so it is difficult for further processing by a consumer. But, I
will take a look at Avro as you mentioned.

I just wondering, if there are any proper implementation for this
requirement, because I couldn't find any tool in the kafka ecosystem page.

https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem


Best Regards,
Aris.


On Mon, Oct 10, 2016 at 6:55 PM, Michael Noll <mi...@confluent.io> wrote:

> Aris,
>
> even today you can already use Kafka to deliver Netflow/Pcap/etc. messages,
> and people are already using it for that (I did that in previous projects
> of mine, too).
>
> Simply encode your Pcap/... messages appropriately (I'd recommend to take a
> look at Avro, which allows you to structure your data similar to e.g.
> Pcap's native format [1]), and then write the encoded messages to Kafka.
> Your downstream applications can then read the encoded messages back from
> Kafka, decode, and commence processing.
>
> That was a brief summary to get you started, feel free to take a look at
> the Apache Kafka docs at kafka.apache.org and/or ask further questions
> here.
>
> -Michael
>
>
>
>
> [1] https://wiki.wireshark.org/Development/LibpcapFileFormat
>
> On Mon, Oct 10, 2016 at 11:19 AM, Aris Risdianto <ar...@smartx.kr> wrote:
>
> > ​Hello,
> >
> >
> > ​Is there any plan or implementation to use Kafka for delivering
> > sFlow/NetFlow/Pcap messages?
> >
> >
> > Best Regards,
> > Aris.
> >
>

Re: sFlow/NetFlow/Pcap Plugin for Kafka Producer

[Michael Noll <mi...@confluent.io>]

Aris,

even today you can already use Kafka to deliver Netflow/Pcap/etc. messages,
and people are already using it for that (I did that in previous projects
of mine, too).

Simply encode your Pcap/... messages appropriately (I'd recommend to take a
look at Avro, which allows you to structure your data similar to e.g.
Pcap's native format [1]), and then write the encoded messages to Kafka.
Your downstream applications can then read the encoded messages back from
Kafka, decode, and commence processing.

That was a brief summary to get you started, feel free to take a look at
the Apache Kafka docs at kafka.apache.org and/or ask further questions here.

-Michael




[1] https://wiki.wireshark.org/Development/LibpcapFileFormat

On Mon, Oct 10, 2016 at 11:19 AM, Aris Risdianto <ar...@smartx.kr> wrote:

> ​Hello,
>
>
> ​Is there any plan or implementation to use Kafka for delivering
> sFlow/NetFlow/Pcap messages?
>
>
> Best Regards,
> Aris.
>