Capture PCAP data from multiple sources

[Vladimir Shlyakhtin <Vl...@sstech.us>]

Hello,


What is good practice to capture data from multiple sources to avoid duplication of packets.
I mean if we have multiple network points (e.g. routers) and capture data from every of these points how to avoid duplication of packets.
According to Metron' PCAP docs PCAP topology just stores data from Kafka topic to HDFS.
Is it possible to detects and skip duplicates, stitch packets and do other processing?
Or the best way to do it in the beginning (send only unique packets) to Kafka or in post-processing.

Thank you

Regards,
- Vladimir

RE: Capture PCAP data from multiple sources

[Vladimir Shlyakhtin <Vl...@sstech.us>]

No, we are just researching for now and trying to understand capabilities, requirements and limitation for capturing network data.
From my understanding there is no one universal solution and this is something that needs network administrator assistance.

Thanks

Regards,
- Vladimir
________________________________
From: Zeolla@GMail.com [zeolla@gmail.com]
Sent: Wednesday, March 07, 2018 12:31 PM
To: user@metron.apache.org
Subject: Re: Capture PCAP data from multiple sources

Depending on what you are tapping and your architecture, you may be able to accomplish this upstream.  Do you have a network packet broker?

Jon

On Wed, Mar 7, 2018, 11:34 Vladimir Shlyakhtin <Vl...@sstech.us>> wrote:
Hello,


What is good practice to capture data from multiple sources to avoid duplication of packets.
I mean if we have multiple network points (e.g. routers) and capture data from every of these points how to avoid duplication of packets.
According to Metron' PCAP docs PCAP topology just stores data from Kafka topic to HDFS.
Is it possible to detects and skip duplicates, stitch packets and do other processing?
Or the best way to do it in the beginning (send only unique packets) to Kafka or in post-processing.

Thank you

Regards,
- Vladimir
--

Jon

Re: Capture PCAP data from multiple sources

["Zeolla@GMail.com" <ze...@gmail.com>]

Depending on what you are tapping and your architecture, you may be able to
accomplish this upstream.  Do you have a network packet broker?

Jon

On Wed, Mar 7, 2018, 11:34 Vladimir Shlyakhtin <
Vladimir.Shlyakhtin@sstech.us> wrote:

> Hello,
>
>
> What is good practice to capture data from multiple sources to avoid
> duplication of packets.
> I mean if we have multiple network points (e.g. routers) and capture data
> from every of these points how to avoid duplication of packets.
> According to Metron' PCAP docs PCAP topology just stores data from Kafka
> topic to HDFS.
> Is it possible to detects and skip duplicates, stitch packets and do other
> processing?
> Or the best way to do it in the beginning (send only unique packets) to
> Kafka or in post-processing.
>
> Thank you
>
> Regards,
> - Vladimir
>
-- 

Jon