You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Alexandre Vermeerbergen <av...@gmail.com> on 2019/06/03 08:52:06 UTC
How to setup Kafka authentication on storm-kafka-monitor ?
Hello,
I am trying to switch my Storm topologies based on Kafka spout from
authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka
Brokers.
So I have changed my Kafka spouts build code to something like this
(actial user & password have been changed to 'foo' and 'bar'):
kafkaSpoutConfig =
KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic")
.setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer")
.setProp("security.protocol", "SASL_SSL")
.setProp("sasl.jaas.config",
"org.apache.kafka.common.security.scram.ScramLoginModule required
username=\"foo\" password=\"bar\";")
.setProp("sasl.mechanism", "SCRAM-SHA-512")
.setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
.setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST)
.setRecordTranslator(new PodSyncPodsKafkaRecordTranslator())
.build();
My problem is that in Storm UI, I see that the usual Kafka Lag
monitoring fails with these errors:
Topology spouts lag error
IdTypeMessage
eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason:
org.apache.kafka.shaded.common.KafkaException: Failed to construct
kafka consumer at
org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
Caused by: org.apache.kafka.shaded.common.KafkaException:
java.lang.IllegalArgumentException: Could not find a 'KafkaClient'
entry in the JAAS configuration. System property
'java.security.auth.login.config' is not set at
org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
at org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
... 4 more Caused by: java.lang.IllegalArgumentException: Could not
find a 'KafkaClient' entry in the JAAS configuration. System property
'java.security.auth.login.config' is not set at
org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65)
at org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 7 more
Question:
I guess that this issue comes from the fact that storm-kafka-monitor
is not aware of the security setup which have been applied to the
Spout.
=> How can I configure storm-kafka-monitor to pass it the Kafka
consumer properties that will allow it to query the Kafka lag without
failing on Kafka authentication?
I have not found a clue about this in documentation...
Also, is it normal that Kafka spouts is not automatically inherited by
storm-kafka-monitor?
Kind regards,
Alexandre Vermeerbergen
Re: How to setup Kafka authentication on storm-kafka-monitor ?
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Sandip,
Thanks for the suggestion, I have created a storm_jaas.conf file with
the following content:
KafkaClient {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="foo"
password="bar";
};
and i got a different exception on Storm UI at the place where Kafka
lag is supposed to be displayed:
Unable to get offset lags for kafka. Reason:
org.apache.kafka.shaded.common.KafkaException: Failed to construct
kafka consumer at
org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
Caused by: org.apache.kafka.shaded.common.KafkaException:
javax.security.auth.login.LoginException: No LoginModule found for
org.apache.kafka.common.security.scram.ScramLoginModule at
org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
at org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
... 4 more Caused by: javax.security.auth.login.LoginException: No
LoginModule found for
org.apache.kafka.common.security.scram.ScramLoginModule at
java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:710)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
at java.base/java.security.AccessController.doPrivileged(Native
Method) at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
at org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 7 more
moreover, the storm-kafka-monitoring processes are eating a huge CPU.
Maybe I'm missing a way to setup storm-kafka-monitoring's consumers
with something equivalent to my Kafka Spout's setup (which works fine
with my Kafka SSL + auth brokers):
.setProp("security.protocol", "SASL_SSL")
.setProp("sasl.mechanism", "SCRAM-SHA-512")
.setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
What would be the right way to setup these properties on
storm-kafka-client's consumers ?
Would the pull requested suggested by Stig help?
Note: i'm running with Apache Storm 1.2.3 latest commits, my
topologies run fine with Kafka Brokers with SSL + Auth, but I need
storm-kafka-monitor to be run & running to track Kafka lag of my Kafka
spouts....
Kind regards,
Alexandre
Le lun. 3 juin 2019 à 17:33, Sandeep Nemuri <nh...@gmail.com> a écrit :
>
> I guess you are missing the KafkaClient section in ${STORM_CONF_DIR}/storm_jaas.conf in UI node.
>
> On Mon, Jun 3, 2019 at 8:32 PM Stig Rohde Døssing <st...@gmail.com> wrote:
>>
>> There was a recent PR to partially address this at https://github.com/apache/storm/pull/3016, it probably doesn't fix your exact case, but fixing your case would probably require doing something similar.
>>
>> The reason this is broken is that storm-kafka-monitor is a workaround to us not having generic metrics delivery from spouts to Storm UI. The way storm-kafka-monitor works, is that your spout reports some of its configuration to Nimbus. When someone requests a Storm UI page, Nimbus then boots up a new JVM running storm-kafka-monitor, which just starts a KafkaConsumer with the config from Nimbus, checks the current consumer position, and exits.
>>
>> We want to get rid of storm-kafka-monitor eventually. I've been looking at it in https://issues.apache.org/jira/browse/STORM-3202, but it will probably take a while to get done. If you'd like to take a look at implementing that issue, I'd be happy to share the code (it does not currently compile).
>>
>> Den man. 3. jun. 2019 kl. 10.52 skrev Alexandre Vermeerbergen <av...@gmail.com>:
>>>
>>> Hello,
>>>
>>> I am trying to switch my Storm topologies based on Kafka spout from
>>> authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka
>>> Brokers.
>>>
>>> So I have changed my Kafka spouts build code to something like this
>>> (actial user & password have been changed to 'foo' and 'bar'):
>>>
>>> kafkaSpoutConfig =
>>> KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic")
>>> .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer")
>>> .setProp("security.protocol", "SASL_SSL")
>>> .setProp("sasl.jaas.config",
>>> "org.apache.kafka.common.security.scram.ScramLoginModule required
>>> username=\"foo\" password=\"bar\";")
>>> .setProp("sasl.mechanism", "SCRAM-SHA-512")
>>>
>>> .setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
>>> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST)
>>> .setRecordTranslator(new PodSyncPodsKafkaRecordTranslator())
>>> .build();
>>>
>>>
>>> My problem is that in Storm UI, I see that the usual Kafka Lag
>>> monitoring fails with these errors:
>>>
>>> Topology spouts lag error
>>>
>>> IdTypeMessage
>>> eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason:
>>> org.apache.kafka.shaded.common.KafkaException: Failed to construct
>>> kafka consumer at
>>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
>>> at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
>>> at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
>>> at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
>>> at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
>>> Caused by: org.apache.kafka.shaded.common.KafkaException:
>>> java.lang.IllegalArgumentException: Could not find a 'KafkaClient'
>>> entry in the JAAS configuration. System property
>>> 'java.security.auth.login.config' is not set at
>>> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
>>> at org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
>>> at org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
>>> at org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
>>> ... 4 more Caused by: java.lang.IllegalArgumentException: Could not
>>> find a 'KafkaClient' entry in the JAAS configuration. System property
>>> 'java.security.auth.login.config' is not set at
>>> org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65)
>>> at org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
>>> at org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
>>> at org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
>>> ... 7 more
>>>
>>> Question:
>>>
>>> I guess that this issue comes from the fact that storm-kafka-monitor
>>> is not aware of the security setup which have been applied to the
>>> Spout.
>>> => How can I configure storm-kafka-monitor to pass it the Kafka
>>> consumer properties that will allow it to query the Kafka lag without
>>> failing on Kafka authentication?
>>>
>>> I have not found a clue about this in documentation...
>>>
>>> Also, is it normal that Kafka spouts is not automatically inherited by
>>> storm-kafka-monitor?
>>>
>>> Kind regards,
>>> Alexandre Vermeerbergen
>
>
>
> --
> Regards
> Sandeep Nemuri
Re: How to setup Kafka authentication on storm-kafka-monitor ?
Posted by Sandeep Nemuri <nh...@gmail.com>.
I guess you are missing the KafkaClient section in
*${STORM_CONF_DIR}/storm_jaas.conf* in UI node.
On Mon, Jun 3, 2019 at 8:32 PM Stig Rohde Døssing <st...@gmail.com>
wrote:
> There was a recent PR to partially address this at
> https://github.com/apache/storm/pull/3016, it probably doesn't fix your
> exact case, but fixing your case would probably require doing something
> similar.
>
> The reason this is broken is that storm-kafka-monitor is a workaround to
> us not having generic metrics delivery from spouts to Storm UI. The way
> storm-kafka-monitor works, is that your spout reports some of its
> configuration to Nimbus. When someone requests a Storm UI page, Nimbus then
> boots up a new JVM running storm-kafka-monitor, which just starts a
> KafkaConsumer with the config from Nimbus, checks the current consumer
> position, and exits.
>
> We want to get rid of storm-kafka-monitor eventually. I've been looking at
> it in https://issues.apache.org/jira/browse/STORM-3202, but it will
> probably take a while to get done. If you'd like to take a look at
> implementing that issue, I'd be happy to share the code (it does not
> currently compile).
>
> Den man. 3. jun. 2019 kl. 10.52 skrev Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>:
>
>> Hello,
>>
>> I am trying to switch my Storm topologies based on Kafka spout from
>> authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka
>> Brokers.
>>
>> So I have changed my Kafka spouts build code to something like this
>> (actial user & password have been changed to 'foo' and 'bar'):
>>
>> kafkaSpoutConfig =
>> KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic")
>> .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer")
>> .setProp("security.protocol", "SASL_SSL")
>> .setProp("sasl.jaas.config",
>> "org.apache.kafka.common.security.scram.ScramLoginModule required
>> username=\"foo\" password=\"bar\";")
>> .setProp("sasl.mechanism", "SCRAM-SHA-512")
>>
>> .setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
>>
>> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST)
>> .setRecordTranslator(new
>> PodSyncPodsKafkaRecordTranslator())
>> .build();
>>
>>
>> My problem is that in Storm UI, I see that the usual Kafka Lag
>> monitoring fails with these errors:
>>
>> Topology spouts lag error
>>
>> IdTypeMessage
>> eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason:
>> org.apache.kafka.shaded.common.KafkaException: Failed to construct
>> kafka consumer at
>>
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
>> at
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
>> at
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
>> at
>> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
>> at
>> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
>> Caused by: org.apache.kafka.shaded.common.KafkaException:
>> java.lang.IllegalArgumentException: Could not find a 'KafkaClient'
>> entry in the JAAS configuration. System property
>> 'java.security.auth.login.config' is not set at
>>
>> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
>> at
>> org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
>> at
>> org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
>> at
>> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
>> ... 4 more Caused by: java.lang.IllegalArgumentException: Could not
>> find a 'KafkaClient' entry in the JAAS configuration. System property
>> 'java.security.auth.login.config' is not set at
>>
>> org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65)
>> at
>> org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
>> at
>> org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
>> at
>> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
>> ... 7 more
>>
>> Question:
>>
>> I guess that this issue comes from the fact that storm-kafka-monitor
>> is not aware of the security setup which have been applied to the
>> Spout.
>> => How can I configure storm-kafka-monitor to pass it the Kafka
>> consumer properties that will allow it to query the Kafka lag without
>> failing on Kafka authentication?
>>
>> I have not found a clue about this in documentation...
>>
>> Also, is it normal that Kafka spouts is not automatically inherited by
>> storm-kafka-monitor?
>>
>> Kind regards,
>> Alexandre Vermeerbergen
>>
>
--
* Regards*
* Sandeep Nemuri*
Re: How to setup Kafka authentication on storm-kafka-monitor ?
Posted by Stig Rohde Døssing <st...@gmail.com>.
There was a recent PR to partially address this at
https://github.com/apache/storm/pull/3016, it probably doesn't fix your
exact case, but fixing your case would probably require doing something
similar.
The reason this is broken is that storm-kafka-monitor is a workaround to us
not having generic metrics delivery from spouts to Storm UI. The way
storm-kafka-monitor works, is that your spout reports some of its
configuration to Nimbus. When someone requests a Storm UI page, Nimbus then
boots up a new JVM running storm-kafka-monitor, which just starts a
KafkaConsumer with the config from Nimbus, checks the current consumer
position, and exits.
We want to get rid of storm-kafka-monitor eventually. I've been looking at
it in https://issues.apache.org/jira/browse/STORM-3202, but it will
probably take a while to get done. If you'd like to take a look at
implementing that issue, I'd be happy to share the code (it does not
currently compile).
Den man. 3. jun. 2019 kl. 10.52 skrev Alexandre Vermeerbergen <
avermeerbergen@gmail.com>:
> Hello,
>
> I am trying to switch my Storm topologies based on Kafka spout from
> authenticated & unencrypted Kafka brokers to SSL + Authenticated Kafka
> Brokers.
>
> So I have changed my Kafka spouts build code to something like this
> (actial user & password have been changed to 'foo' and 'bar'):
>
> kafkaSpoutConfig =
> KafkaSpoutConfig.builder(supConfig.kafka_broker_hosts_str, "myTopic")
> .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myConsumer")
> .setProp("security.protocol", "SASL_SSL")
> .setProp("sasl.jaas.config",
> "org.apache.kafka.common.security.scram.ScramLoginModule required
> username=\"foo\" password=\"bar\";")
> .setProp("sasl.mechanism", "SCRAM-SHA-512")
>
> .setProp(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "")
>
> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.LATEST)
> .setRecordTranslator(new
> PodSyncPodsKafkaRecordTranslator())
> .build();
>
>
> My problem is that in Storm UI, I see that the usual Kafka Lag
> monitoring fails with these errors:
>
> Topology spouts lag error
>
> IdTypeMessage
> eventFromAdminTopicKAFKAUnable to get offset lags for kafka. Reason:
> org.apache.kafka.shaded.common.KafkaException: Failed to construct
> kafka consumer at
>
> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:703)
> at
> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:584)
> at
> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:566)
> at
> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:230)
> at
> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:144)
> Caused by: org.apache.kafka.shaded.common.KafkaException:
> java.lang.IllegalArgumentException: Could not find a 'KafkaClient'
> entry in the JAAS configuration. System property
> 'java.security.auth.login.config' is not set at
>
> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
> at
> org.apache.kafka.shaded.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
> at
> org.apache.kafka.shaded.clients.ClientUtils.createChannelBuilder(ClientUtils.java:85)
> at
> org.apache.kafka.shaded.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
> ... 4 more Caused by: java.lang.IllegalArgumentException: Could not
> find a 'KafkaClient' entry in the JAAS configuration. System property
> 'java.security.auth.login.config' is not set at
>
> org.apache.kafka.shaded.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:65)
> at
> org.apache.kafka.shaded.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
> at
> org.apache.kafka.shaded.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
> at
> org.apache.kafka.shaded.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
> ... 7 more
>
> Question:
>
> I guess that this issue comes from the fact that storm-kafka-monitor
> is not aware of the security setup which have been applied to the
> Spout.
> => How can I configure storm-kafka-monitor to pass it the Kafka
> consumer properties that will allow it to query the Kafka lag without
> failing on Kafka authentication?
>
> I have not found a clue about this in documentation...
>
> Also, is it normal that Kafka spouts is not automatically inherited by
> storm-kafka-monitor?
>
> Kind regards,
> Alexandre Vermeerbergen
>