You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/04/14 21:02:56 UTC
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/76
METRON-110 Create lightweight packet capture script
The fast packet capture process is difficult to manage and deploy in test and development environments. There are difficulties in integrating it with the existing deployment environments which makes it difficult to test and demo functionality that is dependent on packet capture data. Need to create a lightweight packet capture process that replicates what the fast packet capture process does. This would effectively be a replacement for Pycapa; one that is compatible with the Apache license.
Pycapa has two primary runtime modes.
### Producer Mode
Pycapa can be configured to capture packets from a network interface and then forward those packets to a Kafka topic. The following example will capture packets from the `eth0` network interface and forward those to a Kafka topic called `pcap` running on `localhost`.
```
pycapa --producer --kafka localhost:9092 --topic pcap -i eth0
```
To output debug messages every 100 captured packets, run the following.
```
pycapa --producer --kafka localhost:9092 --topic pcap -i eth0 --debug 100
```
### Consumer Mode
Pycapa can be configured to consume packets from a Kafka topic and then write those packets to a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat). To read 100 packets from a kafka topic and then write those to a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat), run the following command. The file `out.pcap` can then be opened with a tool such as Wireshark for further validation.
```
pycapa --consumer --kafka localhost:9092 --topic pcap --n 100 > out.pcap
```
To consume packets from Kafka continuously and print debug messages every 10 packets, run the following command.
```
pycapa --consumer --kafka localhost:9092 --topic pcap --debug 10
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-110
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/76.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #76
----
commit d05bccb7bffcf49dedf0698c457f442e69fb7f75
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-04-14T18:54:42Z
METRON-110 Create lightweight packet capture script
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210534256
I don't think the license for argparse is kosher. Please investigate and
remove if inappropriate or adjust the license file if that's required.
On Fri, Apr 15, 2016 at 15:23 Nick Allen <no...@github.com> wrote:
> @cestella <https://github.com/cestella> I was able to use this tool to
> build valid libpcap-compliant files from packet data in Kafka. This was
> successful when using either pycapa or the 'fast packet capture' code as
> the source of packet data. This kind of test has not been automated, but at
> least now I have the means to automate it.
>
> Also, this should help show exactly what kind of headers we need added by
> your 'pcap extraction topology' effort.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-metron/pull/76#issuecomment-210481160>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210547443
Ahh I see I thought there was some weirdness with GPL, but as I look it's
Python licensed now. It only became part of the standard package as of 2.7
and I think ambari requires 2.6...is there any issue installing this on the
vagrant vm?
Sorry I'm on my phone since wifi on my computer is slightly busted and it's
a bit hard to investigate fully or type easily ;)
On Fri, Apr 15, 2016 at 17:48 Nick Allen <no...@github.com> wrote:
> Argparse is part of the standard library and does not need to be
> separately installed. Removed the need to install it separately via pip.
> Python standard lib is license compatible with Apache.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-metron/pull/76#issuecomment-210542984>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210481160
@cestella I was able to use this tool to build valid libpcap-compliant files from packet data in Kafka. This was successful when using either pycapa or the 'fast packet capture' code as the source of packet data. This kind of test has not been automated, but at least now I have the means to automate it.
Also, this should help show exactly what kind of headers we need added by your 'pcap extraction topology' effort.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210552058
Interesting. I manually deployed on CentOS 6 with the latest change and it worked. But it appears `argparse` was already installed by something else. This node went through a partial Metron install, but no pycapa.
```
[vagrant@node1 pycapa]$ ls -l /usr/lib/python2.6/site-packages/argparse*
-rw-r--r--. 1 root root 87791 Mar 31 2011 /usr/lib/python2.6/site-packages/argparse.py
-rw-r--r--. 1 root root 66661 Jul 23 2015 /usr/lib/python2.6/site-packages/argparse.pyc
-rw-r--r--. 1 root root 66491 Jul 23 2015 /usr/lib/python2.6/site-packages/argparse.pyo
/usr/lib/python2.6/site-packages/argparse-1.2.1-py2.6.egg-info:
total 16
-rw-r--r--. 1 root root 1 Jul 23 2015 dependency_links.txt
-rw-r--r--. 1 root root 2927 Jul 23 2015 PKG-INFO
-rw-r--r--. 1 root root 250 Jul 23 2015 SOURCES.txt
-rw-r--r--. 1 root root 9 Jul 23 2015 top_level.txt
```
Whether it is standard lib or installed via pip, the license issue is moot now right? So I could just revert the last commit since I know the original code worked on 2.6 and 2.7?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210557717
Yeah the license is fine. We are going to have it installed in the vagrant
box as part of ansible, right?
On Fri, Apr 15, 2016 at 18:16 Nick Allen <no...@github.com> wrote:
> Interesting. I manually deployed on CentOS 6 with the latest change and it
> worked. But it appears argparse was already installed by something else.
> This node went through a partial Metron install, but no pycapa.
>
> [vagrant@node1 pycapa]$ ls -l /usr/lib/python2.6/site-packages/argparse*
> -rw-r--r--. 1 root root 87791 Mar 31 2011 /usr/lib/python2.6/site-packages/argparse.py
> -rw-r--r--. 1 root root 66661 Jul 23 2015 /usr/lib/python2.6/site-packages/argparse.pyc
> -rw-r--r--. 1 root root 66491 Jul 23 2015 /usr/lib/python2.6/site-packages/argparse.pyo
>
> /usr/lib/python2.6/site-packages/argparse-1.2.1-py2.6.egg-info:
> total 16
> -rw-r--r--. 1 root root 1 Jul 23 2015 dependency_links.txt
> -rw-r--r--. 1 root root 2927 Jul 23 2015 PKG-INFO
> -rw-r--r--. 1 root root 250 Jul 23 2015 SOURCES.txt
> -rw-r--r--. 1 root root 9 Jul 23 2015 top_level.txt
>
> Whether it is standard lib or installed via pip, the license issue is moot
> now right? So I could just revert the last commit since I know the original
> code worked on 2.6 and 2.7?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-metron/pull/76#issuecomment-210552058>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/76
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210562709
Yes
On Fri, Apr 15, 2016 at 18:46 Nick Allen <no...@github.com> wrote:
> Ok good - Leaving argparse in the requirements.txt.
>
> Does "Sound good" == +1?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-metron/pull/76#issuecomment-210561883>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210560794
Sounds good!
On Fri, Apr 15, 2016 at 18:37 Nick Allen <no...@github.com> wrote:
> Right. Next step is to update deployment/roles/pycapa to use this instead
> of OpenSOC's pycapa.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-metron/pull/76#issuecomment-210558957>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210542984
Argparse is part of the standard library and does not need to be separately installed. Removed the need to install it separately via pip. Python standard lib is license compatible with Apache.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210561883
Ok good - Leaving argparse in the requirements.txt.
Does "Sound good" == +1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-110 Create lightweight packe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/76#issuecomment-210558957
Right. Next step is to update `deployment/roles/pycapa` to use this instead of OpenSOC's pycapa.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---