You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/04/27 22:13:42 UTC
[07/26] incubator-metron git commit: METRON-821 Minor fixes in full
dev kerberos setup instructions (JonZeolla) closes
apache/incubator-metron#510
METRON-821 Minor fixes in full dev kerberos setup instructions (JonZeolla) closes apache/incubator-metron#510
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/29d7cb37
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/29d7cb37
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/29d7cb37
Branch: refs/heads/Metron_0.4.0
Commit: 29d7cb378835b819b326660ed59aca4becbf7099
Parents: 2ecabaa
Author: JonZeolla <ze...@gmail.com>
Authored: Wed Apr 19 21:32:58 2017 -0400
Committer: jonzeolla <jo...@apache.org>
Committed: Wed Apr 19 21:32:58 2017 -0400
----------------------------------------------------------------------
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
metron-deployment/vagrant/Kerberos-setup.md | 313 ++++++++++++-----------
2 files changed, 166 insertions(+), 149 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/29d7cb37/.github/PULL_REQUEST_TEMPLATE.md
----------------------------------------------------------------------
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 56ca8f3..7d5ce2c 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -40,5 +40,5 @@ In order to streamline the review of the contribution we ask you follow these gu
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
-It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
+It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/29d7cb37/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
index 1fce650..a66da8a 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -2,60 +2,59 @@
**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
1. Build full dev and ssh into the machine
- ```
-cd incubator-metron/metron-deployment/vagrant/full-dev-platform
-vagrant up
-vagrant ssh
- ```
+ ```
+ cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+ vagrant up
+ vagrant ssh
+ ```
2. Export env vars. Replace *node1* with the appropriate hosts if running anywhere other than full-dev Vagrant.
- ```
-# execute as root
-sudo su -
-export ZOOKEEPER=node1
-export BROKERLIST=node1
-export HDP_HOME="/usr/hdp/current"
-export METRON_VERSION="0.4.0"
-export METRON_HOME="/usr/metron/${METRON_VERSION}"
- ```
+ ```
+ # execute as root
+ sudo su -
+ export ZOOKEEPER=node1
+ export BROKERLIST=node1
+ export HDP_HOME="/usr/hdp/current"
+ export METRON_VERSION="0.4.0"
+ export METRON_HOME="/usr/metron/${METRON_VERSION}"
+ ```
3. Stop all topologies - we will restart them again once Kerberos has been enabled.
- ```
-for topology in bro snort enrichment indexing; do storm kill $topology; done
- ```
+ ```
+ for topology in bro snort enrichment indexing; do storm kill $topology; done
+ ```
4. Setup Kerberos
- ```
-# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
-# set 'node1' to the correct host for your kdc
-yum -y install krb5-server krb5-libs krb5-workstation
-sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
-cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
-# This step takes a moment. It creates the kerberos database.
-kdb5_util create -s
-/etc/rc.d/init.d/krb5kdc start
-/etc/rc.d/init.d/kadmin start
-chkconfig krb5kdc on
-chkconfig kadmin on
- ```
+ ```
+ # Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
+ # set 'node1' to the correct host for your kdc
+ yum -y install krb5-server krb5-libs krb5-workstation
+ sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+ /bin/cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+ # This step takes a moment. It creates the kerberos database.
+ kdb5_util create -s
+ /etc/rc.d/init.d/krb5kdc start
+ /etc/rc.d/init.d/kadmin start
+ chkconfig krb5kdc on
+ chkconfig kadmin on
+ ```
5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
- ```
-kadmin.local -q "addprinc admin/admin"
-kadmin.local -q "addprinc metron"
- ```
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
6. Create the metron user HDFS home directory
- ```
-sudo -u hdfs hdfs dfs -mkdir /user/metron && \
-sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
-sudo -u hdfs hdfs dfs -chmod 770 /user/metron
- ```
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
-7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user:
+7. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
a. Add the following properties to custom storm-site:
-
```
topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
@@ -87,147 +86,159 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
9. Setup Metron keytab
- ```
-kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
-cp metron.headless.keytab /etc/security/keytabs && \
-chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
-chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
10. Kinit with the metron user
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
12. Setup Kafka ACLs for the topics
- ```
-export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
-done;
- ```
+ ```
+ export KERB_USER=metron
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
+ done
+ ```
13. Setup Kafka ACLs for the consumer groups
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing;
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
+ ```
14. Add metron user to the Kafka cluster ACL
- ```
-/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ```
15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
- ```
-kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
-echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
-echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
- ```
-su metron && cd ~/
-mkdir .storm
-cd .storm
- ```
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
- ```
-[metron@node1 .storm]$ cat client_jaas.conf
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
-};
-KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
-};
- ```
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
- ```
-[metron@node1 .storm]$ cat storm.yaml
-nimbus.seeds : ['node1']
-java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
-storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- ```
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
- ```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
-{
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
-}
- ```
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
20. Setup enrichment and indexing.
- a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
-
+ a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
```
- b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
-
+ b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ fi
```
21. Kinit with the metron user again
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ ```
+ su metron
+ cd
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
22. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
- ```
-for parser in bro snort; do ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json; done
- ```
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
+ done
+ ```
23. Now restart the enrichment and indexing topologies.
- ```
-${METRON_HOME}/bin/start_enrichment_topology.sh
-${METRON_HOME}/bin/start_elasticsearch_topology.sh
- ```
-
-24. Push some sample data to one of the parser topics. E.g for yaf we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/yaf/raw/YafExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/yaf/raw/YafExampleOutput)
- ```
-cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic yaf
- ```
-
-25. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace yaf with whichever parser type you\u2019ve chosen.
- ```
-curl -XGET "${ZOOKEEPER}:9200/yaf*/_search"
-curl -XGET "${ZOOKEEPER}:9200/yaf*/_count"
- ```
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+24. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+ ```
+ cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
+ ```
+
+25. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
+ ```
+ curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
+ curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
+ ```
26. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
@@ -263,5 +274,11 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
+```
+
#### References
* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)