You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by GitBox <gi...@apache.org> on 2019/09/25 07:31:53 UTC
[GitHub] [metron] tigerquoll edited a comment on issue #1519: METRON-2265:
Update Kerberos settings
tigerquoll edited a comment on issue #1519: METRON-2265: Update Kerberos settings
URL: https://github.com/apache/metron/pull/1519#issuecomment-534891894
I do a full dev install and followed
https://github.com/apache/metron/blob/429c5eac55554496c967ca9f6e935f6e0b2d4781/metron-deployment/Kerberos-manual-setup.md
Used the following environment:
```
export BROKERLIST=node1:6667
export KAFKA_HOME=/usr/hdp/3.1.4.0-315/kafka
export METRON_PRINCIPAL_NAME=metron@EXAMPLE.COM
export METRON_SERVICE_KEYTAB=/etc/security/keytabs/metron.headless.keytab
export CLIENT_JAAS_ARG=/etc/kafka/conf/kafka_client_jaas.conf
export KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
export ELASTICSEARCH=node1:9200
export KAFKA_OPTS="-Djava.security.auth.login.config=$CLIENT_JAAS_ARG"
```
Please note the KAFKA_OPTS variable suggested in the guide is wrong, it is a copy from the HDP page which is most likely to be a typo.
The check of the count in elasticsearch at the "Push Data" stage is misleading, as the guide does not delete the previous collection before enabling kerberos. Repeating the "add sample-bro.txt" stage does not increase the document count.
Was utilising `${KAFKA_HOME}/bin/kafka-consumer-groups.sh` to monitor Kafka activity by utilising the following recipe:
1. Create file `/home/metron/kafka.command.config` with content of
```
security.protocol=SASL_PLAINTEXT
```
2. Run command
```
${KAFKA_HOME}/bin/kafka-consumer-groups.sh --command-config=/home/metron/kafka.command.config --bootstrap-server ${BROKERLIST} --describe --group bro_parser
```
This showed nothing consuming from the bro topic.
Further investigations reveals that storm workers are unable to communicate with ZK?
```
2019-2019-09-25 00:36:21.805 o.a.k.c.NetworkClient Thread-18-kafkaSpout-bro-executor[4 4] [WARN] [Consumer clientId=consumer-2, groupId=bro_parser] Connection to node 1001 could not be established. Broker may not be available.-
25 00:36:21.906 o.a.k.c.NetworkClient Thread-12-kafkaSpout-yaf-executor[6 6] [WARN] [Consumer clientId=consumer-1, groupId=yaf_parser] Connection to node 1001 could not be established. Broker may not be available.
2019-09-25 00:36:22.012 o.a.k.c.NetworkClient Thread-16-kafkaSpout-snort-executor[5 5] [WARN] [Consumer clientId=consumer-3, groupId=snort_parser] Connection to node 1001 could not be established. Broker may not be available.
2019-09-25 00:36:22.053 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node1:2181) [INFO] Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2019-09-25 00:36:22.053 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node1:2181) [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
...java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$mk_halting_timer$fn__10399.invoke(worker.clj:259) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315]
at org.apache.storm.timer$mk_timer$fn__1639$fn__1640.invoke(timer.clj:71) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315]
at org.apache.storm.timer$mk_timer$fn__1639.invoke(timer.clj:42) [storm-core-1.2.1.3.1.4.0-315.jar:1.2.1.3.1.4.0-315]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
2019-09-25 00:36:22.191 o.a.s.d.worker Thread-21 [INFO] Shutting down worker bro__snort__yaf-4-1569366291 a0046483-4363-4209-b4e0-d1ad9f6deea2 6701
2019-09-25 00:36:22.191 o.a.s.d.worker Thread-21 [INFO] Terminating messaging context
2019-09-25 00:36:22.191 o.a.s.d.worker Thread-21 [INFO] Shutting down executors
```
Tried back tracking through HDP/Storm/Kerberos docs to verify correct setup, but ran out of time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services