You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by jo...@apache.org on 2017/11/14 12:36:02 UTC
metron-bro-plugin-kafka git commit: METRON-1303 Reorganize the
metron-bro-plugin-kafka (JonZeolla) closes apache/metron-bro-plugin-kafka#1
Repository: metron-bro-plugin-kafka
Updated Branches:
refs/heads/master 25e40be1c -> c035becc1
METRON-1303 Reorganize the metron-bro-plugin-kafka (JonZeolla) closes apache/metron-bro-plugin-kafka#1
Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/c035becc
Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/c035becc
Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/c035becc
Branch: refs/heads/master
Commit: c035becc16cfa5f0b37ecdc6c09beb360955d4c0
Parents: 25e40be
Author: JonZeolla <ze...@gmail.com>
Authored: Tue Nov 14 07:35:00 2017 -0500
Committer: jonzeolla <jo...@apache.org>
Committed: Tue Nov 14 07:35:00 2017 -0500
----------------------------------------------------------------------
CMakeLists.txt | 2 +-
README.md | 29 ++++++++++++-----------------
src/Plugin.cc | 6 +++---
src/Plugin.h | 2 +-
tests/kafka/show-plugin.bro | 2 +-
5 files changed, 18 insertions(+), 23 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 30bf3b5..2bc8d4b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -23,7 +23,7 @@ find_package(OpenSSL)
if (LIBRDKAFKA_FOUND AND OPENSSL_FOUND)
include_directories(BEFORE ${LibRDKafka_INCLUDE_DIR} ${OpenSSL_INCLUDE_DIR})
- bro_plugin_begin(BRO KAFKA)
+ bro_plugin_begin(APACHE KAFKA)
bro_plugin_cc(src/KafkaWriter.cc)
bro_plugin_cc(src/Plugin.cc)
bro_plugin_cc(src/TaggedJSON.cc)
http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 427988b..2901e18 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,8 @@
-Logging Bro Output to Kafka
-===========================
+# Logging Bro Output to Kafka
A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro.
-Installation
-------------
+## Installation
1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4.
@@ -29,12 +27,11 @@ Installation
1. Run the following command to ensure that the plugin was installed successfully.
```
- $ bro -N Bro::Kafka
- Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
+ $ bro -N Apache::Kafka
+ Apache::Kafka - Writes logs to Kafka (dynamic, version 0.1)
```
-Activation
-----------
+## Activation
The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to demonstrate the example.
@@ -46,7 +43,7 @@ The goal in this example is to send all HTTP and DNS records to a Kafka topic na
* Defining `logs_to_send` will ensure that only HTTP and DNS records are sent.
```
-@load Bro/Kafka/logs-to-kafka.bro
+@load Apache/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::kafka_conf = table(
@@ -63,7 +60,7 @@ It is also possible to send each log stream to a uniquely named topic. The goal
* Each log writer accepts a separate configuration table.
```
-@load Bro/Kafka/logs-to-kafka.bro
+@load Apache/Kafka/logs-to-kafka.bro
redef Kafka::topic_name = "";
redef Kafka::tag_json = T;
@@ -101,7 +98,7 @@ You may want to configure bro to filter log messages with certain characteristic
* If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka.
```
-@load Bro/Kafka/logs-to-kafka.bro
+@load Apache/Kafka/logs-to-kafka.bro
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
@@ -141,11 +138,10 @@ event bro_init() &priority=-5
#### Notes
* `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`.
- * You can also filter IPv6 logs from within your Metron cluster [using Stellar](../../metron-stellar/stellar-common#IS_IP). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.
+ * You can also filter IPv6 logs from within your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#is_ip). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.
* It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin.
-Settings
---------
+## Settings
### `kafka_conf`
@@ -204,8 +200,7 @@ enable. The available contexts are:
* feature
* all
-Kerberos
---------
+## Kerberos
This plugin supports producing messages from a kerberized kafka. There
are a couple of prerequisites and a couple of settings to set.
@@ -242,7 +237,7 @@ ${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.Si
The following is how the `${BRO_HOME}/share/bro/site/local.bro` looks:
```
-@load Bro/Kafka/logs-to-kafka.bro
+@load Apache/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/src/Plugin.cc
----------------------------------------------------------------------
diff --git a/src/Plugin.cc b/src/Plugin.cc
index f3dec8b..f2f63a5 100644
--- a/src/Plugin.cc
+++ b/src/Plugin.cc
@@ -17,18 +17,18 @@
#include "Plugin.h"
-namespace plugin { namespace Bro_Kafka {
+namespace plugin { namespace Apache_Kafka {
Plugin plugin;
}}
-using namespace plugin::Bro_Kafka;
+using namespace plugin::Apache_Kafka;
plugin::Configuration Plugin::Configure()
{
AddComponent(new ::logging::Component("KafkaWriter", ::logging::writer::KafkaWriter::Instantiate));
plugin::Configuration config;
- config.name = "Bro::Kafka";
+ config.name = "Apache::Kafka";
config.description = "Writes logs to Kafka";
config.version.major = 0;
config.version.minor = 1;
http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/src/Plugin.h
----------------------------------------------------------------------
diff --git a/src/Plugin.h b/src/Plugin.h
index a169076..e99e91b 100644
--- a/src/Plugin.h
+++ b/src/Plugin.h
@@ -21,7 +21,7 @@
#include "KafkaWriter.h"
#include <plugin/Plugin.h>
-namespace plugin { namespace Bro_Kafka {
+namespace plugin { namespace Apache_Kafka {
class Plugin : public ::plugin::Plugin {
protected:
http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/tests/kafka/show-plugin.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/show-plugin.bro b/tests/kafka/show-plugin.bro
index 4e8dd6a..1eb3dde 100644
--- a/tests/kafka/show-plugin.bro
+++ b/tests/kafka/show-plugin.bro
@@ -15,5 +15,5 @@
# limitations under the License.
#
-# @TEST-EXEC: bro -NN Bro::Kafka >output
+# @TEST-EXEC: bro -NN Apache::Kafka >output
# @TEST-EXEC: btest-diff output