You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by GitBox <gi...@apache.org> on 2020/05/20 20:04:09 UTC
[GitHub] [metron-bro-plugin-kafka] vikgmdev opened a new pull request #47: METRON-2357: Extends example 4 with a dynamic version
vikgmdev opened a new pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47
## Contributor Comments
[Please place any comments here. A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc.]
- Extends the current [Example 4](https://github.com/apache/metron-bro-plugin-kafka#example-4---send-each-zeek-log-to-a-unique-topic) to include a dynamic version.
It will demonstrate how to automatically send each zeek log to a topic with the same name.
For instance the `CONN::LOG` log to be sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic without defining a `Log::Filter` for each of those.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed via:
```
bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428759769
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
Review comment:
On it, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-705538753
Thanks @vikgmdev looking good. Have you tested with the end to end `docker/` tests? I think after the initial spin-up you'll need to configure the environment using `docker_execute_create_topic_in_kafka.sh` (because the topics don't auto-create on publish yet), update the zeek scripts with what you have in the README, and then re-run to ensure it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-702370870
Hey, @JonZeolla just noticed I had the changes since a few months ago but haven't pushed them before. We'll wait for you re-review. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-705538753
Thanks @vikgmdev looking good. Have you tested with the end to end `docker/` tests? I think after the initial spin-up you'll need to configure the environment using `docker_execute_create_topic_in_kafka.sh` (because the topics don't auto-create on publish yet), update the zeek scripts with what you have in the README, and then re-run to ensure it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428765442
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
Review comment:
Yes, I referred to that but you're right it's more a personal preference, should I remove it from the example?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428631726
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
Review comment:
I assume you are referring to [this](https://github.com/apache/kafka/blob/d63eaaaa0181bb7b9b4f5ed088abc00d7b32aeb0/core/src/main/scala/kafka/admin/TopicCommand.scala#L198-L199)? Because otherwise, _ appears to be an allowed character in a topic name.
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
Review comment:
Nit: Please keep the indentation consistent.
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
Review comment:
naes typo
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
+ const topic_name: string = sub(topic_name_under, /_/, "-");
+
+ if (|Kafka::logs_to_send| == 0 || stream_id in Kafka::logs_to_send)
Review comment:
What is the goal of the `|Kafka::logs_to_send| == 0 || ` portion of this? My read of this is this could get confusing because this example ignores our `logs_to_exclude` and `send_all_active_logs` options, if they also get set in an environment following this example config. Is there a way we can refactor this, potentially leveraging [`send_to_kafka`](https://github.com/apache/metron-bro-plugin-kafka/blob/587e9dac9beeee915f7d0c1116a4432447b5d92e/scripts/Apache/Kafka/logs-to-kafka.zeek#L23-L37) to make it more robust?
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
Review comment:
I'm unclear on what ej. means, should this be [e.g.](https://www.dictionary.com/browse/exempli-gratia)? Same with the below reuse of ej.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-714611136
Hi @vikgmdev yeah unfortunately those are known issues. If you manually download [nitroba](https://github.com/open-nsm/course/blob/master/pcaps/nitroba.pcap?raw=true) and put it in the [pcap directory](https://github.com/apache/metron-bro-plugin-kafka/blob/92b85e5e00cc2fd9023ea7f53466db6592eb6634/docker/run_end_to_end.sh#L193), and change the [zeek version](https://github.com/apache/metron-bro-plugin-kafka/blob/92b85e5e00cc2fd9023ea7f53466db6592eb6634/docker/docker-compose.yml#L32) to `3.1.5` it should be able to get through the tests until we can figure out a workaround for the [change](https://github.com/zeek/zeek/issues/977) which is causing the tests to exit due to pcap issues. Sorry for the hassle with this... We plan to make this a non-issue as a part of future work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-697312918
Hi @JonZeolla hadn't the chance to push the updates. Let me retake the issue and I'll push the updates soon. Thanks a lot for your patience.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428759024
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
Review comment:
Oh, yup, I refer to **e.g.** sorry for the typo and I should improve my grammatic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
ottobackwards commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-632103636
I have added Victor to the jira contributors role and assigned the jira to him
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-706576041
Hey, @JonZeolla I encountered two issues when running the end to end `docker/` tests...
The link to download the [nitroba.pcap](http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap) is no longer available:
```
--2020-10-10 11:26:40-- http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
Resolving downloads.digitalcorpora.org (downloads.digitalcorpora.org)... 129.174.125.204
Connecting to downloads.digitalcorpora.org (downloads.digitalcorpora.org)|129.174.125.204|:80... failed: Connection timed out.
Retrying.
```
I commented that line to unblock me and keep with the test.
But after that, the script is crashing and throwing this error:
```
...
===================================================
Configuring kafka plugin
configured the kafka plugin
MADE /home/vick/Documents/Anubis/metron-bro-plugin-kafka/docker/test_output/sáb_10_oct_2020_11_32_09_CDT/exercise-traffic_pcap
Running docker_execute_process_data_dir with
CONTAINER_NAME = metron-bro-plugin-kafka_zeek_1
PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap
===================================================
executing process_data_file.sh in the zeek docker container
PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap
================================
WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks.
1258563753.320650 fatal error: failed to read a packet from /root/data/example-traffic/exercise-traffic.pcap: truncated dump file; tried to read 1434 captured bytes, only got 415
```
so not sure if it's an actual bug on the test or if it's just a step a missed from my setup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r429223738
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
Review comment:
I'm fine with that but if that's the approach the above :: substitution should go direct to -
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-697271265
Hi @vikgmdev thanks again for the contribution. Do you have an update? If not we will unfortunately need to mark this as abandoned
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-643479496
Hi @vikgmdev any update on this? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
JonZeolla commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r429162878
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
Review comment:
As an example it makes sense to leave but maybe be more clear on the situation. Can you add a version pinned link to Kafka documentation or code into your comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r498486281
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
+ const topic_name: string = sub(topic_name_under, /_/, "-");
+
+ if (|Kafka::logs_to_send| == 0 || stream_id in Kafka::logs_to_send)
Review comment:
Hey, @JonZeolla I just improve this part of the example by removing the condition, it still works the same. I also had some confusions about that line `|Kafka::logs_to_send| == 0 || ` but now is all clear so it isn't need at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
ottobackwards commented on a change in pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r429198652
##########
File path: README.md
##########
@@ -179,6 +179,48 @@ event zeek_init() &priority=-10
}
```
+#### Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() &priority=-10
+{
+ for (stream_id in Log::active_streams) {
+ # Convert stream type enum to string
+ const stream_string: string = fmt("%s", stream_id);
+
+ # replace `::` by `_` from the log string name
+ # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+ const stream_name: string = sub(stream_string, /::/, "_");
+
+ # lowercase the whole string for nomalization
+ const topic_name_lower: string = to_lower(stream_name);
+
+ # remove the _log at the of each topic name
+ const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+ # replace `_` by `-` for compatibility with acceptable Kafka topic naes
Review comment:
I think it complicates things, we want this to be as simple as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] vikgmdev edited a comment on pull request #47: METRON-2357: Extends example 4 with a dynamic version
Posted by GitBox <gi...@apache.org>.
vikgmdev edited a comment on pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-706576041
Hey, @JonZeolla I encountered two issues when running the end to end `docker/` tests...
The link to download the [nitroba.pcap](http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap) is no longer available:
```
--2020-10-10 11:26:40-- http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
Resolving downloads.digitalcorpora.org (downloads.digitalcorpora.org)... 129.174.125.204
Connecting to downloads.digitalcorpora.org (downloads.digitalcorpora.org)|129.174.125.204|:80... failed: Connection timed out.
Retrying.
```
I commented that line to unblock me and keep with the test.
But after that, the script is crashing and throwing this error:
```
...
===================================================
Configuring kafka plugin
configured the kafka plugin
MADE /home/vick/Documents/Anubis/metron-bro-plugin-kafka/docker/test_output/sáb_10_oct_2020_11_32_09_CDT/exercise-traffic_pcap
Running docker_execute_process_data_dir with
CONTAINER_NAME = metron-bro-plugin-kafka_zeek_1
PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap
===================================================
executing process_data_file.sh in the zeek docker container
PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap
================================
WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks.
1258563753.320650 fatal error: failed to read a packet from /root/data/example-traffic/exercise-traffic.pcap: truncated dump file; tried to read 1434 captured bytes, only got 415
```
so not sure if it's an actual bug on the test or if it's just a step I missed from my setup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org