You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/11/15 13:03:00 UTC
[jira] [Commented] (METRON-813) Migrate metron-bro-plugin-kafka to
be a bro package
[ https://issues.apache.org/jira/browse/METRON-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253416#comment-16253416 ]
ASF GitHub Bot commented on METRON-813:
---------------------------------------
GitHub user JonZeolla opened a pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/3
METRON-813: Migrate metron-bro-plugin-kafka to be a bro package
This should turn this repo into a bro package containing a bro plugin.
# Testing
The below testing plan assumes CentOS/RHEL, but can be tweaked to work on most mainstream linux distros.
1. Install [Kafka 0.10.0.1](https://kafka.apache.org/0101/documentation.html#quickstart), [Zookeeper 3.4.6](https://zookeeper.apache.org/doc/r3.4.6/zookeeperStarted.html) (The same versions from [HDP 2.5.5](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_release-notes/content/ch01s01.html)), and any package dependancies for testing.
```
# cd
# yum -y install java screen
# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz{,.sha1}
# if [[ "$(sha1sum zookeeper-3.4.6.tar.gz)" == "$(cat zookeeper-3.4.6.tar.gz.sha1)" ]]; then tar -xvf zookeeper-3.4.6.tar.gz; else echo "sha1 sums do not match"; fi
# cd zookeeper-3.4.6
# cp conf/zoo_sample.cfg conf/zoo.cfg
# bin/zkServer.sh start
# cd
# wget https://mirrors.sonic.net/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz
# wget https://dist.apache.org/repos/dist/release/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz.md5
# # Compare MD5s using md5sum
# tar -xvf kafka_2.10-0.10.0.1.tgz
# cd kafka_2.10-0.10.0.1
# bin/kafka-server-start.sh config/server.properties &
# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bro
```
1. [Install bro 2.5.2](https://www.bro.org/sphinx/install/install.html) and [bro-pkg](http://bro-package-manager.readthedocs.io/en/stable/quickstart.html#installation). Make sure you are running at least bro 2.5 and bro-pkg 1.2.0, and configure bro-pkg properly.
```
# export PATH=$PATH:/usr/local/bro/bin
# bro --version
bro version 2.5.2
# bro-pkg --version
bro-pkg 1.2.2
# bro-pkg autoconfig
```
1. Create a working directory and pull in this PR (selfishly using my branch of `checkout-pr` from [metron-commit-stuff](https://github.com/jonzeolla/metron-commit-stuff/tree/support-bro-plugin) to test some recent updates)
```
# git clone https://github.com/jonzeolla/metron-commit-stuff ~/metron-commit-stuff
# cd ~/metron-commit-stuff
# git checkout support-bro-plugin
# cd
# ~/metron-commit-stuff/checkout-pr 3
Please select a repository:
1) metron
2) metron-bro-plugin-kafka
Selection [metron]: bro
<snip>
```
1. Install the package, and all of its dependancies, from the PR branch. Ensure it passes its unit tests.
```
# # Install librdkafka by following ONLY instruction 1 [here](https://github.com/apache/metron-bro-plugin-kafka#installation)
# cd ~/metron-bro-plugin-kafka-pr3/
# bro-pkg install .
```
1. Configure the plugin.
```
cat << EOF >> /usr/local/bro/share/bro/site/local.bro
# Activate metron-bro-plugin-kafka
@load metron-bro-plugin-kafka-pr3/Bro/Kafka
# Configure metron-bro-plugin-kafka
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG);
EOF
```
1. Run bro manually while monitoring kafka to confirm things are working.
```
# mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic
# wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap
# wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap
# export PATH=$PATH:~/kafka_2.11-0.10.1.0/bin
# screen
# kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
# # Ctrl+A c to make a new screen window
# cd ~/brotmp/example-traffic
# bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
# # Use Ctrl+A n to cycle through screen sessions for validation. To run another test, on your second window, do
# cd ~/brotmp/nitroba
# bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-813
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron-bro-plugin-kafka/pull/3.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3
----
commit f21e51f4f91452d66b644b1c041e9a3ae3b39bd7
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-07T12:12:53Z
METRON-1303: Reorganize the metron-bro-plugin-kafka
commit c2f8b2c347f647076c1d0ba17dad5b3794d7957d
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-07T12:22:39Z
Fix broken link
commit 28594e75e423166eb5ca6e6df7a3f7ed2deeecd7
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-14T12:39:04Z
METRON-813: Migrate metron-bro-plugin-kafka to be a bro package
commit 22bbb7e1da6c3b207f9b92d9e959ae31d3fe83a0
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-14T18:55:02Z
Fix load statements
commit 84993a9fb55d274b63b22474b1758903bce64f28
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-15T13:01:27Z
Merge branch 'master' of https://github.com/apache/metron-bro-plugin-kafka into METRON-813
----
> Migrate metron-bro-plugin-kafka to be a bro package
> ---------------------------------------------------
>
> Key: METRON-813
> URL: https://issues.apache.org/jira/browse/METRON-813
> Project: Metron
> Issue Type: Improvement
> Reporter: Jon Zeolla
> Assignee: Jon Zeolla
>
> Per a [discussion](https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E) on the dev mailing list, the kafka plugin should be hosted as a [bro package](https://github.com/bro/packages) and mirrored to https://github.com/apache/incubator-metron-bro-plugin-kafka. The bro kafka plugin should be installed, when necessary, using the bro package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)