You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by JonZeolla <gi...@git.apache.org> on 2017/04/04 20:44:03 UTC
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/incubator-metron/pull/510
METRON-821 Minor fixes in full dev kerberos setup instructions
## Contributor Comments
As I ran through Kerberos-setup.md I found [one minor issue](https://github.com/apache/incubator-metron/blame/eb209f5c9400b81720a866ef5dd1817f73226f44/metron-deployment/vagrant/Kerberos-setup.md#L138), so I figured that if I was going to make a change I might as well make some other adjustments.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
Note: I only reviewed the relevant `target/site/metron-deployment/vagrant/Kerberos-setup.html`
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/incubator-metron METRON-821
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/510.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #510
----
commit 5dd65bb30eccca25cb5b76a96a38af56f61139b8
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-04T20:37:00Z
Update Kerberos-setup.md documentation
commit f93c8e24aa840d0e61c90c1e88769a3278752108
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-04T20:43:21Z
This typo has been driving me crazy
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/510
+1 by inspection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Confirmed that fixed my prior issue in the storm logs. Thanks @dlyle65535!
I'm still unable to confirm that this is working entirely as expected, though. It appeared to be working for a minute there (ES counts were incrementing). Everything is working fine w/the kafka topics, but I'm having some issues with storm that I need to look into.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r109957525
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -135,71 +135,77 @@ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
```
-su metron && cd ~/
-mkdir .storm
-cd .storm
+su metron
+mkdir ~/.storm
+cd ~/.storm
```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
```
-[metron@node1 .storm]$ cat client_jaas.conf
+cat << EOF > client_jaas.conf
StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
};
Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
};
KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
};
+EOF
```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
```
-[metron@node1 .storm]$ cat storm.yaml
+cat << EOF > storm.yaml
nimbus.seeds : ['node1']
java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+EOF
```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
+cd
+cat << EOF > storm-config.json
{
"topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
}
+EOF
```
20. Setup enrichment and indexing.
a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ [[ $EUID -eq 0 ]] || exit
--- End diff --
Right, like if they are just in the native `vagrant ssh`, which is bad. I will clean it up to alert and quit the code snippet but not the shell.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111182531
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -107,23 +107,23 @@ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --cre
12. Setup Kafka ACLs for the topics
```
export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
+for topic in bro enrichments indexing snort yaf; do
--- End diff --
Yes, that was the reasoning. I'm game with either way, I assumed that there a specific reason why yaf was used. I'll update the instructions to consider limited resources.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/510
I'm comfortable with what you have @JonZeolla . Any other issues that you might be having are likely independent of these instructions.
+1 pending a clean Travis build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Looks like there was a small defect in [PR-522](https://github.com/apache/incubator-metron/pull/522). The build is no longer being built against the HDP-2.5.0.0 profile, so there is no org.apache.kafka.common.protocol.SecurityProtocol.PLAINTEXTSASL.
I've opened [METRON-856](https://issues.apache.org/jira/browse/METRON-856).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla closed the pull request at:
https://github.com/apache/incubator-metron/pull/510
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111161348
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -107,23 +107,23 @@ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --cre
12. Setup Kafka ACLs for the topics
```
export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
+for topic in bro enrichments indexing snort yaf; do
--- End diff --
Did you want to start `yaf` because the instructions use the YAF topology for validation later on?
Another option, is to not start YAF here (as we know resources are constrained in Full/Quick Dev) and simply change the instructions below to validate against Snort or Bro, rather than YAF.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
GitHub user JonZeolla reopened a pull request:
https://github.com/apache/incubator-metron/pull/510
METRON-821 Minor fixes in full dev kerberos setup instructions
## Contributor Comments
As I ran through Kerberos-setup.md I found [one minor issue](https://github.com/apache/incubator-metron/blame/eb209f5c9400b81720a866ef5dd1817f73226f44/metron-deployment/vagrant/Kerberos-setup.md#L138), so I figured that if I was going to make a change I might as well make some other adjustments.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
Note: I only reviewed the relevant `target/site/metron-deployment/vagrant/Kerberos-setup.html`
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/incubator-metron METRON-821
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/510.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #510
----
commit 5dd65bb30eccca25cb5b76a96a38af56f61139b8
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-04T20:37:00Z
Update Kerberos-setup.md documentation
commit f93c8e24aa840d0e61c90c1e88769a3278752108
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-04T20:43:21Z
This typo has been driving me crazy
commit 70a4bfd2b41e9aa688fad82f6e6f257c691943c2
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-05T01:03:06Z
Minor adjustments
commit 624fdf8de43508d350aa73cdb8c1039fd137a448
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-05T21:23:29Z
Merge branch 'master' of https://github.com/apache/incubator-metron into METRON-821
commit f7d2ba50450c311f9fe325450264f7df0ea376f2
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-05T21:29:42Z
Update README according to feedback
commit 9fa4353285a3440a53bfe8e92954a119ffbcc493
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-12T15:32:10Z
Update based on PR feedback, adjust spacing
commit 2491b45466b20995194c765aadd0d71638f3bbde
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-12T15:32:32Z
Fix 'are you sure' due to cp alias
commit d45fc8fe466fcf6b362887b6b294e050ba5b99bb
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-13T15:28:35Z
Minor semantic changes and update to 0.4.0 from 0.3.1
commit 5fe0ed7cc14adbfeb7ac07cce4fd4cb041b7f587
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-13T15:30:53Z
Change from yaf to bro for testing on steps 24 and 25
commit 10502ef1b214abe0e8b0e29ccc284a42aa33eb34
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-13T17:15:44Z
Change to use integration test data so differentiating from sensor-stubs is easier
commit 62c1c9b2de576b57daa7552218ef7c1fb929585c
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-13T17:22:22Z
Fix formatting for GitHub and Doxia docs
commit fa45ab09fc1ca9a3489dcb45f2c24424f6ee11fa
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-13T17:50:33Z
Merge branch 'master' of https://github.com/apache/incubator-metron into METRON-821
commit 030d5951466fe7027068fbca617ce9866eff18fb
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-13T18:20:09Z
Revert to for loop for sensor-stub restarts
commit f400af3b0cc3ccd7cac05c0adc1c4a65b27f1d58
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-04-17T20:07:53Z
Superficial modification
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r109948918
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -263,5 +269,12 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
--- End diff --
Nice addition
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/510
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r109948537
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -135,71 +135,77 @@ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
```
-su metron && cd ~/
-mkdir .storm
-cd .storm
+su metron
+mkdir ~/.storm
+cd ~/.storm
```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
```
-[metron@node1 .storm]$ cat client_jaas.conf
+cat << EOF > client_jaas.conf
StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
};
Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
};
KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
};
+EOF
```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
```
-[metron@node1 .storm]$ cat storm.yaml
+cat << EOF > storm.yaml
nimbus.seeds : ['node1']
java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+EOF
```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
+cd
+cat << EOF > storm-config.json
{
"topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
}
+EOF
```
20. Setup enrichment and indexing.
a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ [[ $EUID -eq 0 ]] || exit
--- End diff --
I wonder if it would be better to print a message about needing to be root? This will log out the user and close their screen altogether in some instances.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111183194
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -167,39 +167,48 @@ KafkaClient {
serviceName="kafka"
principal="metron@EXAMPLE.COM";
};
+EOF
```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
```
-[metron@node1 .storm]$ cat storm.yaml
+cat << EOF > storm.yaml
nimbus.seeds : ['node1']
java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+EOF
```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
+cd
+cat << EOF > storm-config.json
{
"topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
}
+EOF
```
20. Setup enrichment and indexing.
a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo "You must be root to run these commands"
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
```
b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ su metron
--- End diff --
I'll move it down to the next step
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111453498
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -263,5 +272,12 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+# Restart the appropriate sensor-stubs
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
--- End diff --
Oops, Interesting. Yes, we can fix with separate JIRA. Loop is fine with me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
I need to run through this another time to double check my changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111454161
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -263,5 +272,12 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+# Restart the appropriate sensor-stubs
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
--- End diff --
Ok, done. I'm just about done here, just spinning it up to do a final test now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111422687
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -263,5 +272,12 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+# Restart the appropriate sensor-stubs
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
--- End diff --
I'm still seeing an issue with `service sensor-stubs restart bro snort` - it looks like it's restarting all `sensor-stubs`, including `yaf`, even when I pass specific sensor stubs. Not a big issue functionally, but this was why I went back to the for loop. Should probably just fix the `service sensor-stubs` via a separate JIRA. Agree?
```
[root@node1 ~]# service sensor-stubs restart bro snort
Stopping sensor-stubs...
.. bro: Stopped [510]
yaf: Not running
.. snort: Stopped [32669]
Starting sensor-stubs...
bro: Ok [20884]
yaf: Ok [20889]
snort: Ok [20912]
[root@node1 ~]# service sensor-stubs status
Checking sensor-stubs...
bro: Running [20884]
yaf: Running [20889]
snort: Running [20912]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111163138
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -263,5 +272,12 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+# Restart the appropriate sensor-stubs
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
--- End diff --
This can be even simpler; `service sensor-stubs restart bro snort`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111482378
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -87,147 +86,159 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
9. Setup Metron keytab
- ```
-kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
-cp metron.headless.keytab /etc/security/keytabs && \
-chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
-chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
10. Kinit with the metron user
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
12. Setup Kafka ACLs for the topics
- ```
-export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
-done;
- ```
+ ```
+ export KERB_USER=metron;
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
+ done
+ ```
13. Setup Kafka ACLs for the consumer groups
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing;
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
+ ```
14. Add metron user to the Kafka cluster ACL
- ```
-/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ```
15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
- ```
-kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
-echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
-echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
- ```
-su metron && cd ~/
-mkdir .storm
-cd .storm
- ```
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
- ```
-[metron@node1 .storm]$ cat client_jaas.conf
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
-};
-KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
-};
- ```
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
- ```
-[metron@node1 .storm]$ cat storm.yaml
-nimbus.seeds : ['node1']
-java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
-storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- ```
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
- ```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
-{
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
-}
- ```
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
20. Setup enrichment and indexing.
- a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
-
+ a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
```
- b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
-
+ b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ fi
--- End diff --
I kind of prefer just showing the key=value so that it is clear to the user what properties need to change.
I appreciate the fact that you've automated it, but since these are manual instructions, maybe we should opt for making them as clear as possible. I could be convinced otherwise though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111495184
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -87,147 +86,159 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
9. Setup Metron keytab
- ```
-kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
-cp metron.headless.keytab /etc/security/keytabs && \
-chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
-chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
10. Kinit with the metron user
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
12. Setup Kafka ACLs for the topics
- ```
-export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
-done;
- ```
+ ```
+ export KERB_USER=metron;
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
+ done
+ ```
13. Setup Kafka ACLs for the consumer groups
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing;
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
+ ```
14. Add metron user to the Kafka cluster ACL
- ```
-/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ```
15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
- ```
-kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
-echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
-echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
- ```
-su metron && cd ~/
-mkdir .storm
-cd .storm
- ```
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
- ```
-[metron@node1 .storm]$ cat client_jaas.conf
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
-};
-KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
-};
- ```
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
- ```
-[metron@node1 .storm]$ cat storm.yaml
-nimbus.seeds : ['node1']
-java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
-storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- ```
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
- ```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
-{
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
-}
- ```
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
20. Setup enrichment and indexing.
- a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
-
+ a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
```
- b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
-
+ b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ fi
--- End diff --
My main goal with this was to minimize the potential for typos and make it easier to get this up and running (copy/pasta instead of "mirror this setting"), but I can appreciate being more transparent with the changes. I could go for something in between like this (but with better formatting):
20. Setup enrichment and indexing.
b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties` so that the following key/values are set:
```
kafka.security.protocol=PLAINTEXTSASL
topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
```
This could be performed using the following commands:
```
if [[ $EUID -ne 0 ]]; then
echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
else
sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
fi
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/510
You and @nickwallen are going to collide - https://github.com/apache/incubator-metron/pull/521
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Thanks, I will test later today with your updates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Let us know when it is all done and you're happy. I'd like to get your change in before #521 so you don't have to deal with any merge conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Validated this with @mmiklavc today, an issue I found was confirmed as unrelated and I will open a separate JIRA for it soon, if nobody else beats me to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
I think the docs are cleaned up in the way that I intended to do with this PR, but I'm running into an issue proving that they're successful. Perhaps someone can provide some feedback?
```
[metron@node1 ~]$ curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
{"count":1740,"_shards":{"total":1,"successful":1,"failed":0}}[metron@node1 ~]$
[metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
[metron@node1 ~]$ head -1 sample-bro.txt
{"http":{"ts":1402307733.473,"uid":"CTo78A11g7CYbbOHvj","id.orig_h":"192.249.113.37","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
[metron@node1 ~]$ cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
[2017-04-14 20:13:07,290] WARN The TGT cannot be renewed beyond the next expiry date: Sat Apr 15 20:12:58 UTC 2017.This process will not be able to authenticate new SASL connections after that time (for example, it will not be able to authenticate a new connection with a Kafka Broker). Ask your system administrator to either increase the 'renew until' time by doing : 'modprinc -maxrenewlife null ' within kadmin, or instead, to generate a keytab for null. Because the TGT's expiry cannot be further extended by refreshing, exiting refresh thread now. (org.apache.kafka.common.security.kerberos.KerberosLogin)
[metron@node1 ~]$ curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
{"count":1740,"_shards":{"total":1,"successful":1,"failed":0}}[metron@node1 ~]$
[metron@node1 ~]$ date
Fri Apr 14 20:13:16 UTC 2017
[metron@node1 ~]$ date;curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
Fri Apr 14 20:13:49 UTC 2017
{"count":1740,"_shards":{"total":1,"successful":1,"failed":0}}[metron@node1 ~]$
```
In my storm worker.logs I'm seeing things like:
```
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702) ~[stormjar.jar:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:557) ~[stormjar.jar:?]
at org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:350) ~[stormjar.jar:?]
at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:346) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:646) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.kafka.common.protocol.SecurityProtocol.PLAINTEXTSASL
at java.lang.Enum.valueOf(Enum.java:238) ~[?:1.8.0_77]
at org.apache.kafka.common.protocol.SecurityProtocol.valueOf(SecurityProtocol.java:28) ~[stormjar.jar:?]
at org.apache.kafka.common.protocol.SecurityProtocol.forName(SecurityProtocol.java:89) ~[stormjar.jar:?]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79) ~[stormjar.jar:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:623) ~[stormjar.jar:?]
... 7 more
```
More details [here](https://gist.github.com/JonZeolla/1f74e002f96081fe7ab66664f813caa7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r109947841
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -135,71 +135,77 @@ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
```
-su metron && cd ~/
-mkdir .storm
-cd .storm
+su metron
+mkdir ~/.storm
+cd ~/.storm
```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
```
-[metron@node1 .storm]$ cat client_jaas.conf
+cat << EOF > client_jaas.conf
StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
+ com.sun.security.auth.module.Krb5LoginModule required
--- End diff --
Can we keep the indentation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111181738
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -263,5 +272,12 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+# Restart the appropriate sensor-stubs
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
--- End diff --
Thanks, I did try restart earlier and snagged an error so I assumed it needed a stop/start.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r109957825
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -135,71 +135,77 @@ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
```
-su metron && cd ~/
-mkdir .storm
-cd .storm
+su metron
+mkdir ~/.storm
+cd ~/.storm
```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
```
-[metron@node1 .storm]$ cat client_jaas.conf
+cat << EOF > client_jaas.conf
StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
+ com.sun.security.auth.module.Krb5LoginModule required
--- End diff --
Whups, yeah that was an accident.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
I'm planning to put a bow on this tomorrow. I'm happy with how it looks as is, I just haven't been able to successfully see the test messages show up in ES following it and need to figure out why. It also appears that I have one last comment to bring to resolution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Thanks.
I will merge this once I'm a bit more comfortable with it. I had some issues the last time I ran through this and want to make sure it was just a local issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/510
Thanks, did some updates. Still need to adjust the test steps to not send to the `yaf` topic but everything else should be done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/510#discussion_r111162379
--- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
@@ -167,39 +167,48 @@ KafkaClient {
serviceName="kafka"
principal="metron@EXAMPLE.COM";
};
+EOF
```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
```
-[metron@node1 .storm]$ cat storm.yaml
+cat << EOF > storm.yaml
nimbus.seeds : ['node1']
java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+EOF
```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
+cd
+cat << EOF > storm-config.json
{
"topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
}
+EOF
```
20. Setup enrichment and indexing.
a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo "You must be root to run these commands"
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
```
b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ su metron
--- End diff --
Why `su metron; cd` here? We could move them to the step that actually needs them done (maybe the next step) or call them out as a separate step. Their purpose is not very clear to me when we tack them onto the end of this step.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---