You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/04/10 14:07:24 UTC
[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/521
METRON-835 Use Profiler with Kerberos
## Contributor Comments
* Enhanced the Kerberos documentation to outline additional steps needed to use the Profiler with Kerberos.
* Enhanced some of the steps to help others avoid stupid user mistakes that I made when setting up a Kerberized-environment.
## Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR?
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-835
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/521.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #521
----
commit de24191976c7d7fbcda5a72c5064c006bbaacf00
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-04-10T13:58:10Z
METRON-835 Use Profiler with Kerberos
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/521#discussion_r113203641
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -55,35 +112,245 @@ General Kerberization notes can be found in the metron-deployment [README.md](..
- c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up.
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+
+ 
+
+1. Create a Metron keytab
-## Push Data
-1. Kinit with the metron user
```
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+ cp metron.headless.keytab /etc/security/keytabs
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
+
+Kafka Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `metron` principal.
+
```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for 'yaf' telemetry.
-2. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
- cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
+ ${KAFKA_HOME}/bin/kafka-topics.sh \
+ --zookeeper ${ZOOKEEPER}:2181 \
+ --create \
+ --topic yaf \
+ --partitions 1 \
+ --replication-factor 1
+ ```
+
+1. Setup Kafka ACLs for the `bro`, `snort`, `enrichments`, and `indexing` topics. Run the same command against any additional topics that you might be using; for example `yaf`.
+
```
+ export KERB_USER=metron
+
+ for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --topic ${topic}
+ done
+ ```
+
+1. Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.
-3. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
```
- curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
- curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
+ export KERB_USER=metron
+
+ for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --group ${group}
+ done
+ ```
+
+1. Add the `metron` principal to the `kafka-cluster` ACL.
+
+ ```
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --cluster kafka-cluster
+ ```
+
+HBase Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `hbase` principal
+
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ ```
+
+1. Grant permissions for the HBase tables used in Metron.
+
+ ```
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
+
+1. If you are using the Profiler, do the same for its HBase table.
+
```
+ echo "create 'profiler', 'P'" | hbase shell
+ echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+ ```
-4. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+Storm Authorization
+-------------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
+
+ ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create the directory `/home/metron/.storm` and switch to that directory.
+
+ ```
+ mkdir /home/metron/.storm
+ cd /home/metron/.storm
+ ```
+
+1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
+
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
+
+1. Create a YAML file at `/home/metron/.storm/storm.yaml`. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.
+
+ ```
+ cat << EOF > /home/metron/.storm/storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
+
+1. Create an auxiliary storm configuration file at `/home/metron/storm-config.json`. Note the login config option in the file points to the client JAAS file.
+
+ ```
+ cat << EOF > /home/metron/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
+
+1. Configure the Enrichment, Indexing and the Profiler topologies to use the client JAAS file. Add the following properties to each of the topology properties files.
+
+ ```
+ kafka.security.protocol=PLAINTEXTSASL
+ topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ ```
+
+ * `${METRON_HOME}/config/enrichment.properties`
+ * `${METRON_HOME}/config/elasticsearch.properties`
+ * `${METRON_HOME}/config/profiler.properties`
+
+ Use the following command to automate this step.
+
+ ```
+ for file in enrichment.properties elasticsearch.properties profiler.properties; do
+ echo ${file}
+ sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
+ sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
+ done
+ ```
+
+Start Metron
+------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
+
+ ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Restart the parser topologies. Be sure to pass in the new parameter, `-ksp` or `--kafka_security_protocol`. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example `yaf`.
+
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e /home/metron/storm-config.json;
+ done
+ ```
+
+1. Restart the Enrichment and Indexing topologies.
+
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+
+ ```
+ cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
--- End diff --
Env var list above got rid of HDP_HOME. Maybe we can keep both and build KAFKA_HOME from HDP_HOME? Gives the user quick access to other tools also. e.g. zkCli
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/521
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/521
We would normally want to push format changes to a separate PR because it's hard to follow what has changed here, but the steps still look consistent to me. Were you able to resolve the issues with the Kafka ACLs in the current order? I recall from the mailing list there was some issue there and it wasn't clear if the user needed to be logged in as 'root' instead of 'metron'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
I think the ACL issue was because there was a loop iterating over the topics setting the ACLs. The 'yaf' topic was not in that loop so the ACL was never set. I changed it so none of the validation suggests using 'yaf' since we don't set the ACLs and don't start the parser topology.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/521
I pulled that from our dev guidelines - https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
I'm ok with it and I like the new formatting. What was the end result with the Kafka ACL authorization problem you were seeing before? These instructions still have the user creating the ACL's as the 'metron' user - did that work the second time through?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/521#discussion_r113204204
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -1,48 +1,105 @@
-# Setting Up Kerberos in Vagrant Full Dev
-**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
-General Kerberization notes can be found in the metron-deployment [README.md](../README.md)
+Kerberos Setup
+==============
-## Setup the KDC
+This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
-1. Build full dev and ssh into the machine
- ```
- cd incubator-metron/metron-deployment/vagrant/full-dev-platform
- vagrant up
- vagrant ssh
- ```
+* [Setup](#setup)
+* [Create a KDC](#create-a-kdc)
+* [Enable Kerberos](#enable-kerberos)
+* [Kafka Authorization](#kafka-authorization)
+* [HBase Authorization](#hbase-authorization)
+* [Storm Authorization](#storm-authorization)
+* [Start Metron](#start-metron)
+
+Setup
+-----
+
+1. Deploy a Vagrant development environment; either [Full Dev](full-dev-platform) or [Quick Dev](quick-dev-platform).
+
+1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.
-2. Export env vars. Replace *node1* with the appropriate hosts if running anywhere other than full-dev Vagrant.
```
# execute as root
sudo su -
export ZOOKEEPER=node1
export BROKERLIST=node1
--- End diff --
I've been thinking about this and realized we shouldn't be declaring our zookeeper and kafka broker lists this way because it won't work for anything but a single node environment. The lists are really host:port comma-delimited.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
And you are right @mmiklavc . I should keep formatting changes separate, if not a separate PR, separate commits at least. Will keep that in mind for the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
Bump
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/521
You and @JonZeolla are going to collide - https://github.com/apache/incubator-metron/pull/510
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/521
+1 by inspection. Thanks for this @nickwallen!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
Conflicts resolved again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
Thanks @mmiklavc . I went ahead and addressed all of the items that you mentioned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/521#discussion_r113203115
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -55,35 +112,245 @@ General Kerberization notes can be found in the metron-deployment [README.md](..
- c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up.
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+
+ 
+
+1. Create a Metron keytab
-## Push Data
-1. Kinit with the metron user
```
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+ cp metron.headless.keytab /etc/security/keytabs
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
+
+Kafka Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `metron` principal.
+
```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for 'yaf' telemetry.
-2. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
- cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
+ ${KAFKA_HOME}/bin/kafka-topics.sh \
+ --zookeeper ${ZOOKEEPER}:2181 \
+ --create \
+ --topic yaf \
+ --partitions 1 \
+ --replication-factor 1
+ ```
+
+1. Setup Kafka ACLs for the `bro`, `snort`, `enrichments`, and `indexing` topics. Run the same command against any additional topics that you might be using; for example `yaf`.
+
```
+ export KERB_USER=metron
+
+ for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --topic ${topic}
+ done
+ ```
+
+1. Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.
-3. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
```
- curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
- curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
+ export KERB_USER=metron
+
+ for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --group ${group}
+ done
+ ```
+
+1. Add the `metron` principal to the `kafka-cluster` ACL.
+
+ ```
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --cluster kafka-cluster
+ ```
+
+HBase Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `hbase` principal
+
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ ```
+
+1. Grant permissions for the HBase tables used in Metron.
+
+ ```
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
+
+1. If you are using the Profiler, do the same for its HBase table.
+
```
+ echo "create 'profiler', 'P'" | hbase shell
+ echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+ ```
-4. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+Storm Authorization
+-------------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
+
+ ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create the directory `/home/metron/.storm` and switch to that directory.
+
+ ```
+ mkdir /home/metron/.storm
+ cd /home/metron/.storm
+ ```
+
+1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
+
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
+
+1. Create a YAML file at `/home/metron/.storm/storm.yaml`. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.
+
+ ```
+ cat << EOF > /home/metron/.storm/storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
+
+1. Create an auxiliary storm configuration file at `/home/metron/storm-config.json`. Note the login config option in the file points to the client JAAS file.
+
+ ```
+ cat << EOF > /home/metron/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
+
+1. Configure the Enrichment, Indexing and the Profiler topologies to use the client JAAS file. Add the following properties to each of the topology properties files.
+
+ ```
+ kafka.security.protocol=PLAINTEXTSASL
+ topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ ```
+
+ * `${METRON_HOME}/config/enrichment.properties`
+ * `${METRON_HOME}/config/elasticsearch.properties`
+ * `${METRON_HOME}/config/profiler.properties`
+
+ Use the following command to automate this step.
+
+ ```
+ for file in enrichment.properties elasticsearch.properties profiler.properties; do
+ echo ${file}
+ sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
+ sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
+ done
+ ```
+
+Start Metron
+------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
+
+ ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Restart the parser topologies. Be sure to pass in the new parameter, `-ksp` or `--kafka_security_protocol`. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example `yaf`.
+
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e /home/metron/storm-config.json;
+ done
+ ```
+
+1. Restart the Enrichment and Indexing topologies.
+
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+
+ ```
+ cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
+ ```
+
+1. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you\u2019ve chosen.
+
+ ```
+ curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
--- End diff --
Just noticed this issue that came in with the original PR - should be ELASTICSEARCH, not ZOOKEEPER.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
I have resolved the merge conflicts. Please take a look @mmiklavc and @JonZeolla
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/521
> We would normally want to push format changes to a separate PR because it's hard to follow what has changed here,
I don't know if we have a normal. I've seen many instances go both ways during the course of the project. I can certainly spend the time to split out formatting, if you need me to. If you don't like the formatting, but want the Kerberos instructions that would be one reason for me to do so.
I had a hard time understanding what we were trying to accomplish with each of the steps, which user the commands would be run under, etc which is why I split them under goal oriented headings. It also helped me generalize the instructions to run in environments other than Full/Quick Dev.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/521
Oh wow, didn't expect that. Thanks @nickwallen looks great!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---