You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Ian Abreu <ia...@wayfair.com> on 2017/06/21 18:57:51 UTC
Netflow Aggregator data into metron pipeline
Hey All,
We've got an architecture which aggregates multiple tiers of netflow data f= or ingestion to a much more centralized point of ingestion. Because of this= , it'd be prohibitive to go and spin up an entirely separate architecture j= ust for getting IPFIX data to be parsed by Kafka, and into Metron.
My question: Can we/how do we use our existing netflow aggregators, and lev= erage IPFIX parsing so that our existing data + aggregation can be used and= ingested by kafka?
Thanks in advance!
Re: Netflow Aggregator data into metron pipeline
Posted by Nick Allen <ni...@nickallen.org>.
> IMO The best route, would just be the ability to parse netflow from a
listening UDP socket,
Agreed, I think. I would look for a third-party tool to capture Netflow
off-the-wire, decode it into some kind of textual representation, and then
pipe that into Kafka for Metron to consume.
On Thu, Jun 22, 2017 at 1:22 PM, Ian Abreu <ia...@wayfair.com> wrote:
> Hey Nick,
>
>
>
> We’re just using RFC compliant UDP forwarding at this point to a single
> aggregator. We’d probably spin up a UDP collector/forwarder, to control the
> flow from a multiple input, multiple output perspective as the most
> efficient means for implementation. IMO The best route, would just be the
> ability to parse netflow from a listening UDP socket, and allow the
> aggregation/forwarding to happen out of scope to metron.
>
>
>
> With regards to brand, Cisco, and palo’s primarily.
>
>
>
> Cheers!
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, June 21, 2017 4:00 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Netflow Aggregator data into metron pipeline
>
>
>
> Hi Ian -
>
>
>
> How do you get data off of your Netflow aggregators; a TAP/SPAN port?
> Care to share the brand/make?
>
>
>
> On Wed, Jun 21, 2017 at 2:57 PM, Ian Abreu <ia...@wayfair.com> wrote:
>
> Hey All,
>
>
>
> We've got an architecture which aggregates multiple tiers of netflow data
> f= or ingestion to a much more centralized point of ingestion. Because of
> this= , it'd be prohibitive to go and spin up an entirely separate
> architecture j= ust for getting IPFIX data to be parsed by Kafka, and into
> Metron.
>
>
>
> My question: Can we/how do we use our existing netflow aggregators, and
> lev= erage IPFIX parsing so that our existing data + aggregation can be
> used and= ingested by kafka?
>
>
>
> Thanks in advance!
>
>
>
RE: Netflow Aggregator data into metron pipeline
Posted by Ian Abreu <ia...@wayfair.com>.
Hey Nick,
We’re just using RFC compliant UDP forwarding at this point to a single aggregator. We’d probably spin up a UDP collector/forwarder, to control the flow from a multiple input, multiple output perspective as the most efficient means for implementation. IMO The best route, would just be the ability to parse netflow from a listening UDP socket, and allow the aggregation/forwarding to happen out of scope to metron.
With regards to brand, Cisco, and palo’s primarily.
Cheers!
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Wednesday, June 21, 2017 4:00 PM
To: user@metron.apache.org
Subject: Re: Netflow Aggregator data into metron pipeline
Hi Ian -
How do you get data off of your Netflow aggregators; a TAP/SPAN port? Care to share the brand/make?
On Wed, Jun 21, 2017 at 2:57 PM, Ian Abreu <ia...@wayfair.com>> wrote:
Hey All,
We've got an architecture which aggregates multiple tiers of netflow data f= or ingestion to a much more centralized point of ingestion. Because of this= , it'd be prohibitive to go and spin up an entirely separate architecture j= ust for getting IPFIX data to be parsed by Kafka, and into Metron.
My question: Can we/how do we use our existing netflow aggregators, and lev= erage IPFIX parsing so that our existing data + aggregation can be used and= ingested by kafka?
Thanks in advance!
Re: Netflow Aggregator data into metron pipeline
Posted by Nick Allen <ni...@nickallen.org>.
Hi Ian -
How do you get data off of your Netflow aggregators; a TAP/SPAN port? Care
to share the brand/make?
On Wed, Jun 21, 2017 at 2:57 PM, Ian Abreu <ia...@wayfair.com> wrote:
> Hey All,
>
>
>
> We've got an architecture which aggregates multiple tiers of netflow data
> f= or ingestion to a much more centralized point of ingestion. Because of
> this= , it'd be prohibitive to go and spin up an entirely separate
> architecture j= ust for getting IPFIX data to be parsed by Kafka, and into
> Metron.
>
>
>
> My question: Can we/how do we use our existing netflow aggregators, and
> lev= erage IPFIX parsing so that our existing data + aggregation can be
> used and= ingested by kafka?
>
>
>
> Thanks in advance!
>