How to process and store PCAP packet data

[Keiko Murakami <k....@nanaroq.com>]

Hi all,

Would you please tell me how to deploy pcap packet processing and writing into hdfs?
I've been executing metron 0.2.0BETA on single node CentOS server by using following ansible playbooks.
- ansible-playbook -i inventory/(myinventory) playbooks/ambari_install.yml
- ansible-playbook -i inventory/(myinventory) playbooks/metron_install.yml

But I could not see any pcap related data on kibana and any pcap data in HDFS.
I guess this reason is $METRON_HOME/lib/metron-writer-0.2.0BETA.jar was not deployed as topology.
Would you please tell me commands to deploy topology $METRON_HOME/lib/metron-writer-0.2.0BETA.jar into storm
and to get pcap data from hdfs?

Best Regards,
Keiko

RE: How to process and store PCAP packet data

[Keiko Murakami <k....@nanaroq.com>]

Anand, 
Thank you for your quick response.

I already know pcap_replay's behavior, this test tool showed me usefull pcap packets,
And I could see the result of enrichment in snort or yaf flow tables.
And I also know how to run pycapa.
In my environment, pycapa got realtime packets normally, 
kafka also successfully got these data at pcap topic,
storm kafaspout in pcap parser topology also successfully got these data.
I could not see any data in kibana's presentation real time packets via pycapa.
I could not know stored place of these packets. 

I believe that pcap real packets will store into hdfs database 
via pycapa->kafka pcap topic->storm kafkaspout->some writer bolt.
And I coult not deploy metron-writer-0.2.0BETA.jar by using metron's playbook.

Best Regards,
Keiko

-----Original Message-----
From: Anand Subramanian [mailto:asubramanian@hortonworks.com] 
Sent: Monday, August 8, 2016 2:27 PM
To: user@metron.incubator.apache.org
Subject: Re: How to process and store PCAP packet data

Hello Keiko,

If you’re looking to test the PCAP related data on Metron, you can try the PCAP replay service, which lets you replay data on an interface. Details are available here:
https://github.com/apache/incubator-metron/tree/master/metron-deployment/roles/pcap_replay


You can also try the Pycapa which can get data off the wire and also from kafka topics. Details are here:
https://github.com/apache/incubator-metron/blob/master/metron-sensors/pycapa/README.md


HTH,
Anand




On 8/8/16, 8:42 AM, "Keiko Murakami" <k....@nanaroq.com> wrote:

>Hi all,
>
>Would you please tell me how to deploy pcap packet processing and writing into hdfs?
>I've been executing metron 0.2.0BETA on single node CentOS server by using following ansible playbooks.
>- ansible-playbook -i inventory/(myinventory) 
>playbooks/ambari_install.yml
>- ansible-playbook -i inventory/(myinventory) 
>playbooks/metron_install.yml
>
>But I could not see any pcap related data on kibana and any pcap data in HDFS.
>I guess this reason is $METRON_HOME/lib/metron-writer-0.2.0BETA.jar was not deployed as topology.
>Would you please tell me commands to deploy topology 
>$METRON_HOME/lib/metron-writer-0.2.0BETA.jar into storm and to get pcap data from hdfs?
>
>Best Regards,
>Keiko
>
>

Re: How to process and store PCAP packet data

[Anand Subramanian <as...@hortonworks.com>]

Hello Keiko,

If you’re looking to test the PCAP related data on Metron, you can try the PCAP replay service, which lets you replay data on an interface. Details are available here:
https://github.com/apache/incubator-metron/tree/master/metron-deployment/roles/pcap_replay


You can also try the Pycapa which can get data off the wire and also from kafka topics. Details are here:
https://github.com/apache/incubator-metron/blob/master/metron-sensors/pycapa/README.md


HTH,
Anand




On 8/8/16, 8:42 AM, "Keiko Murakami" <k....@nanaroq.com> wrote:

>Hi all,
>
>Would you please tell me how to deploy pcap packet processing and writing into hdfs?
>I've been executing metron 0.2.0BETA on single node CentOS server by using following ansible playbooks.
>- ansible-playbook -i inventory/(myinventory) playbooks/ambari_install.yml
>- ansible-playbook -i inventory/(myinventory) playbooks/metron_install.yml
>
>But I could not see any pcap related data on kibana and any pcap data in HDFS.
>I guess this reason is $METRON_HOME/lib/metron-writer-0.2.0BETA.jar was not deployed as topology.
>Would you please tell me commands to deploy topology $METRON_HOME/lib/metron-writer-0.2.0BETA.jar into storm
>and to get pcap data from hdfs?
>
>Best Regards,
>Keiko
>
>