You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by GitBox <gi...@apache.org> on 2019/02/24 04:34:07 UTC
[GitHub] JonZeolla opened a new pull request #26: METRON-2003: Bro plugin
topic should fall back to $path
JonZeolla opened a new pull request #26: METRON-2003: Bro plugin topic should fall back to $path
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/26
## Contributor Comments
A regression was introduced with METRON-1992 which caused kafka topic name selection to no longer fall back to the log writer's `$path`.
### Testing
Apply this patch and run `./run_end_to_end.sh`. Most of these changes are due to `docker/`'s internal hard codings of a topic name of "bro" which I changed to "thepath".
```
diff --git a/docker/in_docker_scripts/configure_bro_plugin.sh b/docker/in_docker_scripts/configure_bro_plugin.sh
index c292504..323cb35 100755
--- a/docker/in_docker_scripts/configure_bro_plugin.sh
+++ b/docker/in_docker_scripts/configure_bro_plugin.sh
@@ -28,11 +28,11 @@ shopt -s nocasematch
echo "Configuring kafka plugin"
{
echo "@load packages"
- echo "redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);"
- echo "redef Kafka::topic_name = \"bro\";"
+ echo "redef Kafka::logs_to_send = set(Conn::LOG);"
+ echo "redef Kafka::topic_name = \"\";"
echo "redef Kafka::tag_json = T;"
echo "redef Kafka::kafka_conf = table([\"metadata.broker.list\"] = \"kafka:9092\");"
- echo "redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);"
+ echo "redef Kafka::logs_to_exclude = set(DHCP::LOG);"
echo "redef Known::cert_tracking = ALL_HOSTS;"
echo "redef Software::asset_tracking = ALL_HOSTS;"
} >> /usr/local/bro/share/bro/site/local.bro
diff --git a/docker/in_docker_scripts/set_path.sh b/docker/in_docker_scripts/set_path.sh
new file mode 100755
index 0000000..53829d0
--- /dev/null
+++ b/docker/in_docker_scripts/set_path.sh
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+# Set $path to something other than the stream name to make it a real test
+sed -i '46 a $path = "thepath",' /usr/local/bro/share/bro/site/code/logs-to-kafka.bro
diff --git a/docker/run_end_to_end.sh b/docker/run_end_to_end.sh
index 1b0e241..7e987da 100755
--- a/docker/run_end_to_end.sh
+++ b/docker/run_end_to_end.sh
@@ -157,6 +157,8 @@ rc=$?; if [[ ${rc} != 0 ]]; then
exit ${rc}
fi
+# Add the test $path
+bash "${SCRIPT_DIR}"/docker_execute_set_path.sh
# for each pcap in the data directory, we want to
# run bro then read the output from kafka
@@ -175,7 +177,7 @@ do
# get the current offset in kafka
# this is where we are going to _start_
- OFFSET=$(bash "${SCRIPT_DIR}"/docker_run_get_offset_bro_kafka.sh | sed 's/^bro:0:\(.*\)$/\1/')
+ OFFSET=$(bash "${SCRIPT_DIR}"/docker_run_get_offset_bro_kafka.sh | sed 's/^thepath:0:\(.*\)$/\1/')
echo "OFFSET------------------> ${OFFSET}"
bash "${SCRIPT_DIR}"/docker_execute_process_data_file.sh --pcap-file-name="${BASE_FILE_NAME}" --output-directory-name="${DOCKER_DIRECTORY_NAME}"
diff --git a/docker/scripts/docker_execute_set_path.sh b/docker/scripts/docker_execute_set_path.sh
new file mode 100755
index 0000000..ae42e42
--- /dev/null
+++ b/docker/scripts/docker_execute_set_path.sh
@@ -0,0 +1,4 @@
+#
+# Executes the set_path.sh in the docker container
+#
+docker exec -w /root "bro" bash -c /root/built_in_scripts/set_path.sh
diff --git a/docker/scripts/docker_run_consume_bro_kafka.sh b/docker/scripts/docker_run_consume_bro_kafka.sh
index b233e2a..74277ed 100755
--- a/docker/scripts/docker_run_consume_bro_kafka.sh
+++ b/docker/scripts/docker_run_consume_bro_kafka.sh
@@ -82,5 +82,5 @@ for i in "$@"; do
done
docker run --rm --network "${NETWORK_NAME}" ches/kafka \
- kafka-console-consumer.sh --topic bro --offset "${OFFSET}" --partition 0 --bootstrap-server kafka:9092 --timeout-ms 1000
+ kafka-console-consumer.sh --topic thepath --offset "${OFFSET}" --partition 0 --bootstrap-server kafka:9092 --timeout-ms 1000
diff --git a/docker/scripts/docker_run_create_bro_topic_in_kafka.sh b/docker/scripts/docker_run_create_bro_topic_in_kafka.sh
index cb00e2c..04120a2 100755
--- a/docker/scripts/docker_run_create_bro_topic_in_kafka.sh
+++ b/docker/scripts/docker_run_create_bro_topic_in_kafka.sh
@@ -70,5 +70,5 @@ echo "NETWORK_NAME = $NETWORK_NAME"
echo "==================================================="
docker run --rm --network "${NETWORK_NAME}" ches/kafka \
- kafka-topics.sh --create --topic bro --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
+ kafka-topics.sh --create --topic thepath --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
diff --git a/docker/scripts/docker_run_get_offset_bro_kafka.sh b/docker/scripts/docker_run_get_offset_bro_kafka.sh
index d3b6621..5d103f0 100755
--- a/docker/scripts/docker_run_get_offset_bro_kafka.sh
+++ b/docker/scripts/docker_run_get_offset_bro_kafka.sh
@@ -71,5 +71,5 @@ for i in "$@"; do
done
docker run --rm --network "${NETWORK_NAME}" ches/kafka \
- kafka-run-class.sh kafka.tools.GetOffsetShell --topic bro --broker-list kafka:9092
+ kafka-run-class.sh kafka.tools.GetOffsetShell --topic thepath --broker-list kafka:9092
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been executed via:
```
bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services