You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by mmiklavc <gi...@git.apache.org> on 2017/03/29 17:00:45 UTC

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

GitHub user mmiklavc opened a pull request:

    https://github.com/apache/incubator-metron/pull/497

    METRON-804: Create a document to describe kerberizing vagrant

    This work completes https://issues.apache.org/jira/browse/METRON-804
    
    I created a markdown README that provides instructions for kerberizing Metron in the Vagrant full dev VM. I also updated the site-book, however I'm noticing some funniness with how markdown is handling numbered lists with code examples interspersed. If anyone has any ideas how to improve this, feel free to comment.
    
    ## Contributor Comments
    [Please place any comments here.  A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc.]
    
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron (Incubating).  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    
    ### For code changes:
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
      ```
      mvn -q clean integration-test install && build_utils/verify_licenses.sh 
      ```
    
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      bin/generate-md.sh
      mvn site:site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mmiklavc/incubator-metron METRON-804

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/497.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #497
    
----
commit 03ab5c95f5095aada0bfda29ec20c81f4da79c29
Author: Michael Miklavcic <mi...@gmail.com>
Date:   2017-03-29T16:22:28Z

    METRON-804: Create a document to describe kerberizing vagrant

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r108781952
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    +20. Setup enrichment and indexing.
    +    1. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +    2. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +21. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
    +```
    +for parser in bro snort; do ${METRON_HOME}/bin/start_parser_topology.sh -z node1:2181 -s ${parser} -ksp PLAINTEXTSASL -e storm-config.json; done
    --- End diff --
    
    At this step however, I am getting the following error. The full log message with the command is available [here](https://pastebin.com/ynHzkeWy). 
    
    Any idea why I am seeing this error?
    
    `
    3770 [main] WARN  o.a.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the  client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshin
 g your cache, ensure that your KDC host's clock is in sync with this host's clock.
    java.lang.RuntimeException: javax.security.auth.login.LoginException: No password provided
    	at org.apache.storm.security.auth.kerberos.AutoTGT.populateCredentials(AutoTGT.java:103)
    	at org.apache.storm.StormSubmitter.populateCredentials(StormSubmitter.java:94)
    	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:214)
    	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:310)
    	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
    	at org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:318)
    Caused by: javax.security.auth.login.LoginException: No password provided
    	at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:919)
    	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
    	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    	at org.apache.storm.security.auth.kerberos.AutoTGT.populateCredentials(AutoTGT.java:80)
    	... 5 more
    `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    +1 by inspection and via vagrant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r108999637
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    +20. Setup enrichment and indexing.
    +    1. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +    2. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    --- End diff --
    
    change to `SASL_PLAINTEXT`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by kylerichardson <gi...@git.apache.org>.
Github user kylerichardson commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    @mmiklavc Nice document! Planning to run through it myself later this week.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r108997211
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    +20. Setup enrichment and indexing.
    +    1. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +    2. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +21. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
    +```
    +for parser in bro snort; do ${METRON_HOME}/bin/start_parser_topology.sh -z node1:2181 -s ${parser} -ksp PLAINTEXTSASL -e storm-config.json; done
    --- End diff --
    
    Did a round of troubleshooting with @mmiklavc - thank you, Mike!
    
    - Need to do `kinit` as metron user for the above error to go away
    - Further, the `kafka_security_protocol` needs to be changed as `SASL_PLAINTEXT` instead of `PLAINTEXTSASL` in order to avoid `worker died` errors in the storm topology


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r108982409
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    --- End diff --
    
    Need to enclose `enrichment` in single quotes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by Matt Foley <mf...@hortonworks.com>.
Okay, try this: https://github.com/mattf-horton/incubator-metron/blob/METRON-804/metron-deployment/vagrant/Kerberos-setup.md

I wasn’t able to build a PR to your branch, seems there’s a non-ff in the way the previous patch was merged.  Anyway, if you just grab that file and diff against yours, you’ll see the change is small.

Items 7, 8, and 20 needed to be fixed.  The problem is that “a-b-c” paragraphs aren’t actually list-items, as MD only knows roman numerals for list numbering.  Since they are paragraphs, the codeblocks and images under them should be at the SAME indent level, and separated by explicit blank line.

This works in both Github-MD and doxia-markdown.  It looks slightly better in doxia because in Github the “a-b-c” paragraphs are exdented a little.  If you hate it we can try a couple other things, but I thought this was close enough.

Cheers,
--Matt

On 3/30/17, 2:40 PM, "Matt Foley" <mf...@hortonworks.com> wrote:

    That’s weird.  Mine looks fine: https://github.com/mattf-horton/incubator-metron/blob/METRON-804-notes/METRON-804-mf.tiff
    
    But the tooling was exactly that of https://github.com/mattf-horton/incubator-metron/tree/METRON-804/site-book/bin
    
    What additional changes did you make?
    
    Oh, I just looked in github, and it’s broken there!  How ironic.
    On your side, is it broken in Github or in the site-book? 
    --Matt
    
    On 3/30/17, 11:15 AM, "mmiklavc" <gi...@git.apache.org> wrote:
    
        Github user mmiklavc commented on the issue:
        
            https://github.com/apache/incubator-metron/pull/497
          
            @mattf-horton Thanks again for the patch! I made a couple more minor tweaks to get the images and indentation correct for the nested lists. I'm unable to get a nested list code block to format correctly, however. It's not bad, but it's just not quite right. If anyone has any suggestions, please chime in.
            ![image](https://cloud.githubusercontent.com/assets/658443/24519468/7f928ea6-1542-11e7-80c6-0070a1810f5e.png)
        
        
        
        ---
        If your project is set up for it, you can reply to this email and have your
        reply appear on GitHub as well. If your project does not have this feature
        enabled and wishes so, or if the feature is enabled but not working, please
        contact infrastructure at infrastructure@apache.org or file a JIRA ticket
        with INFRA.
        ---
        
        
    
    


Re: [GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by Matt Foley <mf...@hortonworks.com>.
That’s weird.  Mine looks fine: https://github.com/mattf-horton/incubator-metron/blob/METRON-804-notes/METRON-804-mf.tiff

But the tooling was exactly that of https://github.com/mattf-horton/incubator-metron/tree/METRON-804/site-book/bin

What additional changes did you make?

Oh, I just looked in github, and it’s broken there!  How ironic.
On your side, is it broken in Github or in the site-book? 
--Matt

On 3/30/17, 11:15 AM, "mmiklavc" <gi...@git.apache.org> wrote:

    Github user mmiklavc commented on the issue:
    
        https://github.com/apache/incubator-metron/pull/497
      
        @mattf-horton Thanks again for the patch! I made a couple more minor tweaks to get the images and indentation correct for the nested lists. I'm unable to get a nested list code block to format correctly, however. It's not bad, but it's just not quite right. If anyone has any suggestions, please chime in.
        ![image](https://cloud.githubusercontent.com/assets/658443/24519468/7f928ea6-1542-11e7-80c6-0070a1810f5e.png)
    
    
    
    ---
    If your project is set up for it, you can reply to this email and have your
    reply appear on GitHub as well. If your project does not have this feature
    enabled and wishes so, or if the feature is enabled but not working, please
    contact infrastructure at infrastructure@apache.org or file a JIRA ticket
    with INFRA.
    ---
    
    


[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    @mattf-horton Thanks again for the patch! I made a couple more minor tweaks to get the images and indentation correct for the nested lists. I'm unable to get a nested list code block to format correctly, however. It's not bad, but it's just not quite right. If anyone has any suggestions, please chime in.
    ![image](https://cloud.githubusercontent.com/assets/658443/24519468/7f928ea6-1542-11e7-80c6-0070a1810f5e.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r108999605
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    +20. Setup enrichment and indexing.
    +    1. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    --- End diff --
    
    change to `SASL_PLAINTEXT`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    @mattf-horton Thanks! I will pull this in and review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    Just a comment, METRON-797 and METRON-793 are both in master as of the time of this comment, so these docs can be done on current master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r109200716
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    +20. Setup enrichment and indexing.
    +    1. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +    2. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    --- End diff --
    
    Yup.. I'm on full dev


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-metron/pull/497


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r108983684
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    --- End diff --
    
    Need to export `METRON_HOME` variable here again, since we are now `su` into `metron` user.
    Also need to kinit again as metron user before attempting to start the parsers:
    `
    kinit -kt /etc/security/keytabs/metron.headless.keytab  @metron@EXAMPLE.COM
    `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    Hi @anandsubbu - just to be clear, this doc requires METRON-793 and METRON-797. Are you running with those branches?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/497#discussion_r109190138
  
    --- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
    @@ -0,0 +1,238 @@
    +# Setting Up Kerberos in Vagrant Full Dev
    +**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
    +
    +1. Build full dev and ssh into the machine
    +```
    +cd incubator-metron/metron-deployment/vagrant/full-dev-platform
    +vagrant up
    +vagrant ssh
    +```
    +
    +2. Export env vars
    +```
    +# execute as root
    +sudo su -
    +export ZOOKEEPER=node1
    +export BROKERLIST=node1
    +export HDP_HOME="/usr/hdp/current"
    +export METRON_VERSION="0.3.1"
    +export METRON_HOME="/usr/metron/${METRON_VERSION}"
    +```
    +
    +3. Stop all topologies - we will  restart them again once Kerberos has been enabled.
    +```
    +for topology in bro snort enrichment indexing; do storm kill $topology; done
    +```
    +
    +4. Setup Kerberos
    +```
    +# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
    +yum -y install krb5-server krb5-libs krb5-workstation
    +sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
    +cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    +# This step takes a moment. It creates the kerberos database.
    +kdb5_util create -s
    +/etc/rc.d/init.d/krb5kdc start
    +/etc/rc.d/init.d/kadmin start
    +chkconfig krb5kdc on
    +chkconfig kadmin on
    +```
    +
    +5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
    +```
    +kadmin.local -q "addprinc admin/admin"
    +kadmin.local -q "addprinc metron"
    +```
    +
    +6. Create the metron user HDFS home directory
    +```
    +sudo -u hdfs hdfs dfs -mkdir /user/metron && \
    +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
    +sudo -u hdfs hdfs dfs -chmod 770 /user/metron
    +```
    +
    +7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user. Add the following properties to custom storm-site. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values.
    +```
    +topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
    +supervisor.run.worker.as.user=true
    +```
    +
    +![custom storm-site](readme-images/ambari-storm-site.png)
    +
    +![custom storm-site properties](readme-images/ambari-storm-site-properties.png)
    +
    +8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
    +   1. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
    +      ![enable keberos](readme-images/enable-kerberos.png)
    +      ![enable keberos get started](readme-images/enable-kerberos-started.png)
    +   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
    +      ![enable keberos configure](readme-images/enable-kerberos-configure-kerberos.png)
    +   3. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
    +      ![enable keberos configure](readme-images/custom-storm-site-final.png)
    +
    +9. Setup Metron keytab
    +```
    +kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +cp metron.headless.keytab /etc/security/keytabs && \
    +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +chmod 440 /etc/security/keytabs/metron.headless.keytab
    +```
    +
    +10. Kinit with the metron user
    +```
    +kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +```
    +
    +11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic yaf --partitions 1 --replication-factor 1
    +```
    +
    +12. Setup Kafka ACLs for the topics
    +```
    +export KERB_USER=metron;
    +for topic in bro enrichments indexing snort; do
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
    +done;
    +```
    +
    +13. Setup Kafka ACLs for the consumer groups
    +```
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --group indexing;
    +```
    +
    +14. Add metron user to the Kafka cluster ACL
    +```
    +/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
    +```
    +
    +15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
    +```
    +kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +echo "grant 'metron', 'RW', enrichment" | hbase shell
    +```
    +
    +16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
    +```
    +su - metron
    +mkdir .storm
    +cd .storm
    +```
    +
    +17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    +```
    +[metron@node1 .storm]$ cat client_jaas.conf
    +StormClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="nimbus";
    +};
    +Client {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="zookeeper";
    +};
    +KafkaClient {
    +   com.sun.security.auth.module.Krb5LoginModule required
    +   useTicketCache=true
    +   renewTicket=true
    +   serviceName="kafka";
    +};
    +```
    +
    +18. Create a storm.yaml with jaas file info.
    +```
    +[metron@node1 .storm]$ cat storm.yaml
    +nimbus.seeds : ['node1']
    +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +```
    +
    +19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
    +```
    +cd /home/metron
    +[metron@node1 ~]$ cat storm-config.json
    +{
    +  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +}
    +```
    +
    +20. Setup enrichment and indexing.
    +    1. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    +topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +```
    +
    +    2. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
    +```
    +kafka.security.protocol=PLAINTEXTSASL
    --- End diff --
    
    Not sure what's going on here - I ran through this again and it still worked with PLAINTEXTSASL and not SASL_PLAINTEXT. You're running this on full dev?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/497
  
    @mmiklavc , Markdown requires you to indent a codeblock (beginning and ending triple back-ticks) if you wish it to be "under" a list item.  Otherwise it interrupts the list, as you've seen, so the next list item restarts at "1".  I've sent you a PR with fixes.  Also substituted "a b c" for sub-item numbering, and split item 7 into sub-items to make it easier to read.  You may squash without attribution. --Matt


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---