You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by JonZeolla <gi...@git.apache.org> on 2018/11/09 21:49:52 UTC
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/17
METRON-1866: Improve metron-bro-plugin-kafka documentation
## Contributor Comments
This depends on METRON-1304. Pure documentation change
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1866
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron-bro-plugin-kafka/pull/17.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17
----
commit d0e59108238d7f36c707e7f44a0f00bbbdd73c74
Author: Jon Zeolla <ze...@...>
Date: 2018-11-09T21:49:01Z
METRON-1866
----
---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/17#discussion_r232417353
--- Diff: README.md ---
@@ -145,23 +217,35 @@ event bro_init() &priority=-5
#### Notes
* `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`.
+ * In Bro 2.5.x the bro project introduced a [logger function](https://www.bro.org/sphinx/cluster/index.html#logger) which removes the logging functions from the manager thread, and taking advantage of that is highly recommended. If you are running this plugin on Bro 2.4.x, you may encounter issues where the manager thread is taking on too much responsibility and pinning a single CPU core without the ability to spread the load across additional cores. In this case, it may be in your best interest to prefer using a bro logging predicate over filtering in your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common) in order to lessen the load of that thread.
* You can also filter IPv6 logs from within your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#is_ip). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.
* It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin.
## Settings
-### `kafka_conf`
+### `logs_to_send`
--- End diff --
Nevermind. You're wanting to preceed this with #2
---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/17#discussion_r232413911
--- Diff: README.md ---
@@ -12,24 +12,72 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec
## Installation
+### `bro-pkg` Installation
+
+`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation.
+
+1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4.
+
+ In order to use this plugin within a kerberized Kafka environment, you will also need `libsasl2` installed and will need to pass `--enable-sasl` to the `configure` script.
+
+ ```
+ $ curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz
+ $ cd librdkafka-0.9.4/
+ $ ./configure --enable-sasl
+ $ make
+ $ sudo make install
+ ```
+
+1. Configure `bro-pkg` by following the quickstart guide [here](https://bro-package-manager.readthedocs.io/en/stable/quickstart.html).
+
+1. Install the plugin using `bro-pkg install`.
+
+ ```
+ $ bro-pkg install apache/metron-bro-plugin-kafka --version master
+ The following packages will be INSTALLED:
+ bro/apache/metron-bro-plugin-kafka (master)
+
+ Verify the following REQUIRED external dependencies:
+ (Ensure their installation on all relevant systems before proceeding):
+ from bro/apache/metron-bro-plugin-kafka (master):
+ librdkafka ~0.9.4
+
+ Proceed? [Y/n]
+ bro/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree) ? [/usr/local/lib]
+ Saved answers to config file: /home/jonzeolla/.bro-pkg/config
+ Running unit tests for "bro/apache/metron-bro-plugin-kafka"
+ all 10 tests successful
+
+
+ Installing "bro/apache/metron-bro-plugin-kafka"........
+ Installed "bro/apache/metron-bro-plugin-kafka" (master)
+ Loaded "bro/apache/metron-bro-plugin-kafka"
+ ```
+
--- End diff --
I think it would be good to add the same sanity check that we have at the end of the manual installation instructions, as the last step here.
```
[root@localhost ~]# bro -N Apache::Kafka
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)
```
---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron-bro-plugin-kafka/pull/17
---
[GitHub] metron-bro-plugin-kafka issue #17: METRON-1866: Improve metron-bro-plugin-ka...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron-bro-plugin-kafka/pull/17
+1
---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/17#discussion_r232435113
--- Diff: README.md ---
@@ -12,24 +12,72 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec
## Installation
+### `bro-pkg` Installation
+
+`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation.
+
+1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4.
+
+ In order to use this plugin within a kerberized Kafka environment, you will also need `libsasl2` installed and will need to pass `--enable-sasl` to the `configure` script.
+
+ ```
+ $ curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz
+ $ cd librdkafka-0.9.4/
+ $ ./configure --enable-sasl
+ $ make
+ $ sudo make install
+ ```
+
+1. Configure `bro-pkg` by following the quickstart guide [here](https://bro-package-manager.readthedocs.io/en/stable/quickstart.html).
+
+1. Install the plugin using `bro-pkg install`.
+
+ ```
+ $ bro-pkg install apache/metron-bro-plugin-kafka --version master
+ The following packages will be INSTALLED:
+ bro/apache/metron-bro-plugin-kafka (master)
+
+ Verify the following REQUIRED external dependencies:
+ (Ensure their installation on all relevant systems before proceeding):
+ from bro/apache/metron-bro-plugin-kafka (master):
+ librdkafka ~0.9.4
+
+ Proceed? [Y/n]
+ bro/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree) ? [/usr/local/lib]
+ Saved answers to config file: /home/jonzeolla/.bro-pkg/config
+ Running unit tests for "bro/apache/metron-bro-plugin-kafka"
+ all 10 tests successful
+
+
+ Installing "bro/apache/metron-bro-plugin-kafka"........
+ Installed "bro/apache/metron-bro-plugin-kafka" (master)
+ Loaded "bro/apache/metron-bro-plugin-kafka"
+ ```
+
--- End diff --
Sounds good, added a final step to validate manually. Initially, I left that out since `bro-pkg` does a very similar test, among others
---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/17#discussion_r232414752
--- Diff: README.md ---
@@ -145,23 +217,35 @@ event bro_init() &priority=-5
#### Notes
* `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`.
+ * In Bro 2.5.x the bro project introduced a [logger function](https://www.bro.org/sphinx/cluster/index.html#logger) which removes the logging functions from the manager thread, and taking advantage of that is highly recommended. If you are running this plugin on Bro 2.4.x, you may encounter issues where the manager thread is taking on too much responsibility and pinning a single CPU core without the ability to spread the load across additional cores. In this case, it may be in your best interest to prefer using a bro logging predicate over filtering in your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common) in order to lessen the load of that thread.
* You can also filter IPv6 logs from within your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#is_ip). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.
* It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin.
## Settings
-### `kafka_conf`
+### `logs_to_send`
--- End diff --
We don't currently have `logs_to_send` that comes later. Some of these are remnants of #2 perhaps.
If it's too much pain to clean-up what should be in #2 versus in this PR, I can be flexible. Just let me know.
---