You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by GitBox <gi...@apache.org> on 2020/05/19 18:34:49 UTC

[GitHub] [metron-bro-plugin-kafka] ottobackwards opened a new pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

ottobackwards opened a new pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46


   Add the ability to define static json fields and values to be added to
   the top level object when tagged_json is T.
   
   ### Testing
   - run_end_to_end.sh
   - look at any of the kafka output logs in the testing output directory for:
   "FIRST_STATIC_NAME" : "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME":"SECOND_STATIC_VALUE"
   in the json.
   
   
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin.
   
   In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
   - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
   
   ### For code changes:
   - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
   - [x] Have you included steps or a guide to how the change may be verified and tested manually?
   - [x] Have you ensured that the full suite of tests and checks have been executed via:
     ```
     bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
     ```
   - [x] Have you written or updated unit tests and or integration tests to verify your changes?
   - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [-] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent?
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631134572


   LGTM +1 thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on a change in pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
ottobackwards commented on a change in pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#discussion_r427535179



##########
File path: README.md
##########
@@ -224,6 +224,22 @@ event zeek_init() &priority=-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing Kafka json message when tagged_json is set to true.  Each will be added to the root json object.
+    * the Kafka::additional_message_values table can be configured with each name and value
+    * based on the following configuration, each outgoing message will have "FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": "SECOND_STATIC_VALUE" added.
+```bash
+@load packages
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
+redef Kafka::topic_name = "zeek";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(["metadata.broker.list"] = "kafka-1:9092,kafka-2:9092");
+redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] = "FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE");

Review comment:
       follow on




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
JonZeolla commented on pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631050619


   Looks good.  Noticed we are lacking btests - I'm fine with having that as a follow-on.  +1 pending JIRA creation


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
JonZeolla commented on a change in pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#discussion_r427518350



##########
File path: scripts/init.zeek
##########
@@ -53,6 +53,14 @@ export {
                 ["metadata.broker.list"] = "localhost:9092"
         ) &redef;
 
+##  Key value pairs that will be added to outgoing messages at the root level
+##  for example:          ["bro_server"] = "this_server_name"

Review comment:
       Let's use `zeek` in examples instead of the old `bro`

##########
File path: README.md
##########
@@ -224,6 +224,22 @@ event zeek_init() &priority=-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message

Review comment:
       The notes are specific to the prior example, so please shift this down in the README

##########
File path: scripts/init.zeek
##########
@@ -53,6 +53,14 @@ export {
                 ["metadata.broker.list"] = "localhost:9092"
         ) &redef;
 
+##  Key value pairs that will be added to outgoing messages at the root level
+##  for example:          ["bro_server"] = "this_server_name"
+##  will results in a  "bro_server":"this_server_name" field added to the outgoing

Review comment:
       Same as above comment

##########
File path: README.md
##########
@@ -224,6 +224,22 @@ event zeek_init() &priority=-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing Kafka json message when tagged_json is set to true.  Each will be added to the root json object.
+    * the Kafka::additional_message_values table can be configured with each name and value
+    * based on the following configuration, each outgoing message will have "FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": "SECOND_STATIC_VALUE" added.
+```bash

Review comment:
       This is `zeek` not `bash`

##########
File path: scripts/init.zeek
##########
@@ -53,6 +53,14 @@ export {
                 ["metadata.broker.list"] = "localhost:9092"
         ) &redef;
 
+##  Key value pairs that will be added to outgoing messages at the root level
+##  for example:          ["bro_server"] = "this_server_name"
+##  will results in a  "bro_server":"this_server_name" field added to the outgoing
+##  json
+##  note this depends on tag_json being T
+const additional_message_values: table[string] of string = table(

Review comment:
       Can't we just shrink this to one line via `table() &redef;`?

##########
File path: scripts/init.zeek
##########
@@ -53,6 +53,14 @@ export {
                 ["metadata.broker.list"] = "localhost:9092"
         ) &redef;
 
+##  Key value pairs that will be added to outgoing messages at the root level

Review comment:
       Please align indentation with other items

##########
File path: README.md
##########
@@ -224,6 +224,22 @@ event zeek_init() &priority=-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing Kafka json message when tagged_json is set to true.  Each will be added to the root json object.
+    * the Kafka::additional_message_values table can be configured with each name and value
+    * based on the following configuration, each outgoing message will have "FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": "SECOND_STATIC_VALUE" added.
+```bash
+@load packages
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
+redef Kafka::topic_name = "zeek";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(["metadata.broker.list"] = "kafka-1:9092,kafka-2:9092");
+redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] = "FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE");

Review comment:
       We should consider a more illustrative example, like:
   ```
   @if ( Cluster::is_enabled() )
   redef Kafka::additional_message_values = table(["CLUSTER_ENABLED"] = "True", ["NODE_ID"] = Cluster::node);
   @else
   redef Kafka::additional_message_values = table(["CLUSTER_ENABLED"] = "False");
   @endif
   ```
   
   Warning: did not test.  May need to pull `node$id` for the NODE_ID value




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] asfgit closed pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
ottobackwards commented on pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631089959


   https://issues.apache.org/jira/browse/METRON-2356 <-- @JonZeolla 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

Posted by GitBox <gi...@apache.org>.
ottobackwards commented on pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631090146


   I think that jira captures why the change is beyond this pr


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org