You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/04/27 22:13:46 UTC
[11/26] incubator-metron git commit: The MPack should function in a
kerberized cluster (justinleet) closes apache/incubator-metron#518
The MPack should function in a kerberized cluster (justinleet) closes apache/incubator-metron#518
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/41b5b105
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/41b5b105
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/41b5b105
Branch: refs/heads/Metron_0.4.0
Commit: 41b5b1050cf29e5dcb5d0c36b5a9dbd1cafa745e
Parents: 55062fb
Author: justinjleet <ju...@gmail.com>
Authored: Mon Apr 17 15:16:49 2017 -0400
Committer: leet <le...@apache.org>
Committed: Thu Apr 20 16:03:29 2017 -0400
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 217 +++++++++++++++++++
metron-deployment/README.md | 15 ++
.../common-services/METRON/CURRENT/metainfo.xml | 4 +-
.../package/scripts/enrichment_commands.py | 11 +
.../CURRENT/package/scripts/metron_client.py | 2 +-
.../CURRENT/package/scripts/metron_service.py | 16 --
.../CURRENT/package/scripts/parser_commands.py | 9 +-
.../package/templates/client_jaas.conf.j2 | 2 +-
.../CURRENT/package/templates/storm.config.j2 | 2 +-
.../ambari-storm-site-properties.png | Bin 0 -> 49605 bytes
.../readme-images/ambari-storm-site.png | Bin 0 -> 134251 bytes
.../readme-images/custom-storm-site-final.png | Bin 0 -> 128117 bytes
.../enable-kerberos-configure-kerberos.png | Bin 0 -> 136196 bytes
.../readme-images/enable-kerberos-started.png | Bin 0 -> 149382 bytes
.../readme-images/enable-kerberos.png | Bin 0 -> 16497 bytes
metron-deployment/vagrant/Kerberos-setup.md | 200 ++---------------
.../ambari-storm-site-properties.png | Bin 49605 -> 0 bytes
.../vagrant/readme-images/ambari-storm-site.png | Bin 134251 -> 0 bytes
.../readme-images/custom-storm-site-final.png | Bin 128117 -> 0 bytes
.../enable-kerberos-configure-kerberos.png | Bin 136196 -> 0 bytes
.../readme-images/enable-kerberos-started.png | Bin 149382 -> 0 bytes
.../vagrant/readme-images/enable-kerberos.png | Bin 16497 -> 0 bytes
22 files changed, 269 insertions(+), 209 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
new file mode 100644
index 0000000..4eaa725
--- /dev/null
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -0,0 +1,217 @@
+# Setting Up Kerberos outside of an Ambari Management Pack
+The Ambari Management pack will manage Kerberization when used.
+**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment [README.md](README.md)
+
+## Setup the KDC
+See [Setup the KDC](vagrant/Kerberos-setup.md)
+
+4. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
+
+## Kerberize Metron
+
+1. Stop all topologies - we will restart them again once Kerberos has been enabled.
+ ```
+ for topology in bro snort enrichment indexing; do storm kill $topology; done
+ ```
+
+2. Create the metron user HDFS home directory
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
+
+3. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
+
+ a. Add the following properties to custom storm-site:
+ ```
+ topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+ nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+ supervisor.run.worker.as.user=true
+ ```
+
+ b. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site:
+
+ 
+
+ c. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values:
+
+ 
+
+4. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+
+ a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
+
+ 
+
+ 
+
+ b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
+
+ 
+
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+
+ 
+
+5. Setup Metron keytab
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
+
+6. Kinit with the metron user
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+7. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
+
+8. Setup Kafka ACLs for the topics
+ ```
+ export KERB_USER=metron
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
+ done
+ ```
+
+9. Setup Kafka ACLs for the consumer groups
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
+ ```
+
+10. Add metron user to the Kafka cluster ACL
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ```
+
+11. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
+
+12. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
+
+13. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
+
+14. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
+
+15. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
+
+16. Setup enrichment and indexing.
+
+ a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
+ ```
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
+ ```
+
+ b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
+ ```
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ fi
+ ```
+
+17. Distribute the custom jaas file and the keytab to each supervisor node, in the same locations as above. This ensures that the worker nodes can authenticate. For a one node cluster, nothing needs to be done.
+
+18. Kinit with the metron user again
+ ```
+ su metron
+ cd
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+19. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
+ done
+ ```
+
+20. Now restart the enrichment and indexing topologies.
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+Metron should be ready to receieve data.
+
+## Push Data
+See [Push Data](vagrant/Kerberos-setup.md)
+
+### Other useful commands
+See [Other useful commands](vagrant/Kerberos-setup.md)
+
+#### References
+* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 66aa5f4..6f1653b 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -146,6 +146,21 @@ Build the Ambari Mpack to get the dashboard updated appropriately.
Once the MPack is installed, run the Kibana service's action "Load Template" to install dashboards. This will completely overwrite the .kibana in Elasticsearch, so use with caution.
+## Kerberos
+The MPack can allow Metron to be installed and then Kerberized, or installed on top of an already Kerberized cluster. This is done through Ambari's standard Kerberization setup.
+
+### Caveats
+* For nodes using a Metron client and a local repo, the repo must exist on all nodes (e.g via createrepo). This repo can be empty; only the main Metron services need the RPMs.
+* A Metron client must be installed on each supervisor node in a secured cluster. This is to ensure that the Metron keytab and client_jaas.conf get distributed in order to allow reading and writing from Kafka.
+ * When Metron is already installed on the cluster, this should be done before Kerberizing.
+ * When addding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client.
+* Storm (and Metron) must be restarted after Metron is installed on an already Kerberized cluster. Several Storm configs get updated, and Metron will be unable to write to Kafka without a restart.
+ * Kerberizing a cluster with an existing Metron already has restarts of all services during Kerberization, so it's unneeded.
+
+Instructions for setup on Full Dev can be found at [Kerberos-setup.md](vagrant/Kerberos-setup.md). These instructions can also be used for setting up KDC and testing.
+
+### Kerberos Without an MPack
+Using the MPack is preferred, but instructions for Kerberizing manually can be found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md)
## TODO
- Support Ubuntu deployments
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index 25237f5..a4e82e2 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -177,9 +177,9 @@
<component>
<name>METRON_CLIENT</name>
- <displayName>METRON Client</displayName>
+ <displayName>Metron Client</displayName>
<category>CLIENT</category>
- <cardinality>1+</cardinality>
+ <cardinality>0+</cardinality>
<versionAdvertised>true</versionAdvertised>
<commandScript>
<script>scripts/metron_client.py</script>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index 4fda07b..136eade 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -21,6 +21,7 @@ from resource_management.core.logger import Logger
from resource_management.core.resources.system import Execute, File
import metron_service
+from metron_security import kinit
# Wrap major operations and functionality in this class
class EnrichmentCommands:
@@ -184,6 +185,11 @@ class EnrichmentCommands:
def create_hbase_tables(self):
Logger.info("Creating HBase Tables")
+ if self.__params.security_enabled:
+ kinit(self.__params.kinit_path_local,
+ self.__params.hbase_keytab_path,
+ self.__params.hbase_principal_name,
+ execute_user=self.__params.hbase_user)
cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
add_enrichment_cmd = cmd.format(self.__params.enrichment_table, self.__params.enrichment_cf)
Execute(add_enrichment_cmd,
@@ -208,6 +214,11 @@ class EnrichmentCommands:
def set_hbase_acls(self):
Logger.info("Setting HBase ACLs")
+ if self.__params.security_enabled:
+ kinit(self.__params.kinit_path_local,
+ self.__params.hbase_keytab_path,
+ self.__params.hbase_principal_name,
+ execute_user=self.__params.hbase_user)
cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
add_enrichment_acl_cmd = cmd.format(self.__params.metron_user, self.__params.enrichment_table)
Execute(add_enrichment_acl_cmd,
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
index 65831c9..32fe0e9 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
@@ -46,4 +46,4 @@ class MetronClient(Script):
raise ClientComponentHasNoStatus()
if __name__ == "__main__":
- MetronClient().execute()
\ No newline at end of file
+ MetronClient().execute()
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
index f2cb88d..be1654f 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
@@ -68,22 +68,6 @@ def get_running_topologies(params):
Logger.info("Topologies: " + str(topologiesDict))
return topologiesDict
- try:
- stormjson = json.loads(stdout)
- except ValueError, e:
- Logger.info('Stdout: ' + str(stdout))
- Logger.info('Stderr: ' + str(stderr))
- Logger.exception(str(e))
- return {}
-
- topologiesDict = {}
-
- for topology in stormjson['topologies']:
- topologiesDict[topology['name']] = topology['status']
-
- Logger.info("Topologies: " + str(topologiesDict))
- return topologiesDict
-
def load_global_config(params):
Logger.info('Create Metron Local Config Directory')
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
index 965502a..e4cae47 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
@@ -113,11 +113,10 @@ class ParserCommands:
if self.__params.security_enabled:
# Append the extra configs needed for secured cluster.
start_cmd_template = start_cmd_template + ' -e ~' + self.__params.metron_user + '/.storm/storm.config'
- if self.__params.security_enabled:
- metron_security.kinit(self.__params.kinit_path_local,
- self.__params.metron_keytab_path,
- self.__params.metron_principal_name,
- execute_user=self.__params.metron_user)
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
for parser in self.get_parser_list():
Logger.info('Starting ' + parser)
Execute(start_cmd_template.format(self.__params.metron_home,
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
index ff2498c..c0a047e 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
@@ -41,4 +41,4 @@ KafkaClient {
useTicketCache=false
serviceName="kafka"
principal="{{metron_principal_name}}";
-};
\ No newline at end of file
+};
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
index caf1221..85fdf5c 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
@@ -19,4 +19,4 @@
{
"topology.worker.childopts" : "-Djava.security.auth.login.config={{metron_home}}/client_jaas.conf",
"topology.auto-credentials" : {{topology_auto_credentials_double_quotes}}
-}
\ No newline at end of file
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/ambari-storm-site-properties.png b/metron-deployment/readme-images/ambari-storm-site-properties.png
new file mode 100755
index 0000000..e0050a2
Binary files /dev/null and b/metron-deployment/readme-images/ambari-storm-site-properties.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/ambari-storm-site.png b/metron-deployment/readme-images/ambari-storm-site.png
new file mode 100755
index 0000000..5ff2d24
Binary files /dev/null and b/metron-deployment/readme-images/ambari-storm-site.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/custom-storm-site-final.png b/metron-deployment/readme-images/custom-storm-site-final.png
new file mode 100755
index 0000000..9b383d5
Binary files /dev/null and b/metron-deployment/readme-images/custom-storm-site-final.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png b/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
new file mode 100755
index 0000000..212c64b
Binary files /dev/null and b/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos-started.png b/metron-deployment/readme-images/enable-kerberos-started.png
new file mode 100755
index 0000000..96adb51
Binary files /dev/null and b/metron-deployment/readme-images/enable-kerberos-started.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos.png b/metron-deployment/readme-images/enable-kerberos.png
new file mode 100755
index 0000000..bb46923
Binary files /dev/null and b/metron-deployment/readme-images/enable-kerberos.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
index 27a56b0..f02cc5f 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -1,5 +1,8 @@
# Setting Up Kerberos in Vagrant Full Dev
**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment [README.md](../README.md)
+
+## Setup the KDC
1. Build full dev and ssh into the machine
```
@@ -19,12 +22,7 @@
export METRON_HOME="/usr/metron/${METRON_VERSION}"
```
-3. Stop all topologies - we will restart them again once Kerberos has been enabled.
- ```
- for topology in bro snort enrichment indexing; do storm kill $topology; done
- ```
-
-4. Setup Kerberos
+3. Setup Kerberos
```
# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
# set 'node1' to the correct host for your kdc
@@ -39,210 +37,46 @@
chkconfig kadmin on
```
-5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
+4. Setup the admin user principal. You'll kinit as the metron user when running topologies. Make sure to remember the password.
```
kadmin.local -q "addprinc admin/admin"
- kadmin.local -q "addprinc metron"
- ```
-
-6. Create the metron user HDFS home directory
- ```
- sudo -u hdfs hdfs dfs -mkdir /user/metron && \
- sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
- sudo -u hdfs hdfs dfs -chmod 770 /user/metron
- ```
-
-7. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
-
- a. Add the following properties to custom storm-site:
```
- topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
- nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
- supervisor.run.worker.as.user=true
- ```
-
- b. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site:
- 
-
- c. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values:
-
- 
-
-8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+## Ambari Setup
+1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
- 
+ 
- 
+ 
b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
- 
-
- c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+ 
- 
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up.
-9. Setup Metron keytab
- ```
- kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
- cp metron.headless.keytab /etc/security/keytabs && \
- chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
- chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
-
-10. Kinit with the metron user
+## Push Data
+1. Kinit with the metron user
```
kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
```
-11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
- ```
-
-12. Setup Kafka ACLs for the topics
- ```
- export KERB_USER=metron
- for topic in bro enrichments indexing snort; do
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
- done
- ```
-
-13. Setup Kafka ACLs for the consumer groups
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
- ```
-
-14. Add metron user to the Kafka cluster ACL
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
- ```
-
-15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
- ```
- kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
- echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
- echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
-
-16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
- ```
- su metron
- mkdir ~/.storm
- cd ~/.storm
- ```
-
-17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
- ```
- cat << EOF > client_jaas.conf
- StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
- };
- Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
- };
- KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
- };
- EOF
- ```
-
-18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
- ```
- cat << EOF > storm.yaml
- nimbus.seeds : ['node1']
- java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
- storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- EOF
- ```
-
-19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
- ```
- cat << EOF > ~/storm-config.json
- {
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
- }
- EOF
- ```
-
-20. Setup enrichment and indexing.
-
- a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
- ```
- if [[ $EUID -ne 0 ]]; then
- echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
- else
- sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
- sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
- fi
- ```
-
- b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
- ```
- if [[ $EUID -ne 0 ]]; then
- echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
- else
- sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
- sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
- fi
- ```
-
-21. Kinit with the metron user again
- ```
- su metron
- cd
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
-
-22. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
- ```
- for parser in bro snort; do
- ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
- done
- ```
-
-23. Now restart the enrichment and indexing topologies.
- ```
- ${METRON_HOME}/bin/start_enrichment_topology.sh
- ${METRON_HOME}/bin/start_elasticsearch_topology.sh
- ```
-
-24. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+2. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
```
-25. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
+3. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
```
curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
```
-26. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+4. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
-### Other useful commands:
+### Other useful commands
#### Kerberos
Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn\u2019t download this list, you can also check the principal manually by running the following against the keytab.
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png b/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
deleted file mode 100755
index e0050a2..0000000
Binary files a/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/ambari-storm-site.png b/metron-deployment/vagrant/readme-images/ambari-storm-site.png
deleted file mode 100755
index 5ff2d24..0000000
Binary files a/metron-deployment/vagrant/readme-images/ambari-storm-site.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/custom-storm-site-final.png b/metron-deployment/vagrant/readme-images/custom-storm-site-final.png
deleted file mode 100755
index 9b383d5..0000000
Binary files a/metron-deployment/vagrant/readme-images/custom-storm-site-final.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png b/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
deleted file mode 100755
index 212c64b..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos-started.png b/metron-deployment/vagrant/readme-images/enable-kerberos-started.png
deleted file mode 100755
index 96adb51..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos-started.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos.png b/metron-deployment/vagrant/readme-images/enable-kerberos.png
deleted file mode 100755
index bb46923..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos.png and /dev/null differ