You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "jackson.dickson" <ja...@gmail.com> on 2018/02/07 04:49:07 UTC
Enable Kerberos on Ignite Kafka streamer / support for new consumer
config
Hi,
We are trying to enable Kerberos in our Ignite-Kafka env. Kerberos is
supported in Kafka consumers using the newer consumer configs. As per Kafka
documentation, Kerberos is only supported on the newer KAfka consumers.
It seems the Ignite KafkaStreamer only supports using the old consumer
config.
Is there any way to setup Kerberos auth on KafkaStreamer?
Thanks,
Jackson
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by "jackson.dickson" <ja...@gmail.com>.
Hi Andrew,
Yes, "bootstrap.server" option is not working for you?
-Jackson
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by "jackson.dickson" <ja...@gmail.com>.
Thanks Andrew.
The new consumer config enable Kafka clients to connect directly to the
Kafka brokers. kafka-clients 0.10.x support both style. I believe you would
need another ConsumerConfig type like ZKConfig.
-Jackson
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Jackson,
Got it. I've created a ticket for this [1].
I'm not too close familiar with Kafka. Please, let me know if I missed smth.
Seems, the fix is trivial, we've just create a KafkaStreamer directly
instead of using zookeeper kafka connector.
[1] https://issues.apache.org/jira/browse/IGNITE-7752
On Mon, Feb 19, 2018 at 2:04 PM, jackson.dickson <ja...@gmail.com>
wrote:
> Hi Andrew,
>
> Yes, "bootstrap.server" option is not working for me.
>
> -Jackson
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by "jackson.dickson" <ja...@gmail.com>.
Hi Andrew,
Yes, "bootstrap.server" option is not working for me.
-Jackson
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Jackson,
I can't figure out how to switch to new style config.
Seem, "bootstrap.servers" option should be used instead of
"zookeeper.connect".
Do you mean using "bootstrap.server" option is not working for you?
On Wed, Feb 7, 2018 at 1:18 PM, jackson.dickson <ja...@gmail.com>
wrote:
> Thanks Andrew for your response.
>
> As per this documentation (
> https://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-
> configuration-for-kafka-clients
> <https://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-
> configuration-for-kafka-clients>
> ), SASL auth is supported only for new Kafka consumers. The new kafka
> consumers connect to the Kafka brokers directly instead of zookeeper
>
>
> Below two security config are required to be specified in the consumers
> configs which are only supported in the new Kafka consumer config.
> security.protocol
> sasl.kerberos.service.name
>
> Refer - https://kafka.apache.org/0100/documentation.html#
> oldconsumerconfigs
> <https://kafka.apache.org/0100/documentation.html#oldconsumerconfigs>
>
> The ignite kafka streamer uses old consumer config
> (kafka.consumer.ConsumerConfig). I get below errors in the log file if I
> specify these new consumer configs.
>
> 2018-02-07 05:11:42.432|myapp|ip-x-x-x-x.myserver|[main]|INFO
> |||||||kafka.utils.VerifiableProperties:info|68|Verifying properties
> 2018-02-07 05:11:42.454|myapp|ip-x-x-x-x.myserver|[main]|WARN
> |||||||kafka.utils.VerifiableProperties:warn|83|Property auto.reset.offset
> is not valid
> 2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|INFO
> |||||||kafka.utils.VerifiableProperties:info|68|Property group.id is
> overridden to ignio-kafka-streamer
> 2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|WARN
> |||||||kafka.utils.VerifiableProperties:warn|83|Property
> sasl.kerberos.service.name is not valid
> 2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|WARN
> |||||||kafka.utils.VerifiableProperties:warn|83|Property security.protocol
> is not valid
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by "jackson.dickson" <ja...@gmail.com>.
Thanks Andrew for your response.
As per this documentation (
https://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-configuration-for-kafka-clients
<https://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-configuration-for-kafka-clients>
), SASL auth is supported only for new Kafka consumers. The new kafka
consumers connect to the Kafka brokers directly instead of zookeeper
Below two security config are required to be specified in the consumers
configs which are only supported in the new Kafka consumer config.
security.protocol
sasl.kerberos.service.name
Refer - https://kafka.apache.org/0100/documentation.html#oldconsumerconfigs
<https://kafka.apache.org/0100/documentation.html#oldconsumerconfigs>
The ignite kafka streamer uses old consumer config
(kafka.consumer.ConsumerConfig). I get below errors in the log file if I
specify these new consumer configs.
2018-02-07 05:11:42.432|myapp|ip-x-x-x-x.myserver|[main]|INFO
|||||||kafka.utils.VerifiableProperties:info|68|Verifying properties
2018-02-07 05:11:42.454|myapp|ip-x-x-x-x.myserver|[main]|WARN
|||||||kafka.utils.VerifiableProperties:warn|83|Property auto.reset.offset
is not valid
2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|INFO
|||||||kafka.utils.VerifiableProperties:info|68|Property group.id is
overridden to ignio-kafka-streamer
2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|WARN
|||||||kafka.utils.VerifiableProperties:warn|83|Property
sasl.kerberos.service.name is not valid
2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|WARN
|||||||kafka.utils.VerifiableProperties:warn|83|Property security.protocol
is not valid
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Enable Kerberos on Ignite Kafka streamer / support for new
consumer config
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Jackson,
Ignite supports kafka-0.10.0.1 version.
Kafka official docs has support some kerberos options [1].
So, it should be possible to enable Kerberos with certain options in
ConsumerConfig.
[1] https://kafka.apache.org/0100/documentation.html#brokerconfigs
On Wed, Feb 7, 2018 at 7:49 AM, jackson.dickson <ja...@gmail.com>
wrote:
> Hi,
>
> We are trying to enable Kerberos in our Ignite-Kafka env. Kerberos is
> supported in Kafka consumers using the newer consumer configs. As per Kafka
> documentation, Kerberos is only supported on the newer KAfka consumers.
>
> It seems the Ignite KafkaStreamer only supports using the old consumer
> config.
>
> Is there any way to setup Kerberos auth on KafkaStreamer?
>
> Thanks,
> Jackson
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov