You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Meghana Narasimhan <mn...@bandwidth.com> on 2017/07/24 20:55:18 UTC
kafka-consumer-groups tool with SASL_PLAINTEXT
Hi,
What is the correct way to use the kafka-consumer-groups tool with
SASL_PLAINTEXT security enabled ?
The tool seems to work fine with PLAINTEXT port but not with
SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
what permissions have to enabled for it ?
Thanks,
Meghana
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
Thanks, Vahid ! Nice documentation. All the tools were working fine except
for the kafka-consumer-groups --list which is what I was struggling to get
working. Realized I had missed the cluster permissions for the user. It
looks good now.
Thanks,
Meghana
On Mon, Jul 24, 2017 at 5:14 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Meghana,
>
> I did some experiments with SASL_PLAINTEXT and documented the results
> here:
> https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
> I think it covers what you'd like to achieve. If not, please advise.
>
> Thanks.
> --Vahid
>
>
>
>
> From: Meghana Narasimhan <mn...@bandwidth.com>
> To: users@kafka.apache.org
> Date: 07/24/2017 01:56 PM
> Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
>
>
>
> Hi,
> What is the correct way to use the kafka-consumer-groups tool with
> SASL_PLAINTEXT security enabled ?
>
> The tool seems to work fine with PLAINTEXT port but not with
> SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
> what permissions have to enabled for it ?
>
> Thanks,
> Meghana
>
>
>
>
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Gabriel,
I have yet to experiment with enabling SSL for Kafka.
However, there are some good documents out there that seem to cover it.
Examples:
*
https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/
*
http://coheigea.blogspot.com/2016/09/securing-apache-kafka-broker-part-i.html
Is there anything specific about the SSL and consumer groups that you are
having issues with?
Thanks.
--Vahid
From: Gabriel Machado <gm...@gmail.com>
To: users@kafka.apache.org
Date: 07/28/2017 08:40 AM
Subject: Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Hi Vahid,
Do you know how to use consumer-group tool with ssl only (without sasl) ?
Gabriel.
Le 24 juil. 2017 11:15 PM, "Vahid S Hashemian" <va...@us.ibm.com>
a écrit :
Hi Meghana,
I did some experiments with SASL_PLAINTEXT and documented the results
here:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
I think it covers what you'd like to achieve. If not, please advise.
Thanks.
--Vahid
From: Meghana Narasimhan <mn...@bandwidth.com>
To: users@kafka.apache.org
Date: 07/24/2017 01:56 PM
Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
Hi,
What is the correct way to use the kafka-consumer-groups tool with
SASL_PLAINTEXT security enabled ?
The tool seems to work fine with PLAINTEXT port but not with
SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
what permissions have to enabled for it ?
Thanks,
Meghana
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Gabriel Machado <gm...@gmail.com>.
Sorry for the delayed reply, i was on holidays.
Manikumar, it works :).
Thank you very much for your help.
Gabriel.
2017-07-31 10:27 GMT+02:00 Manikumar <ma...@gmail.com>:
> We should pass necessary ssl configs using --command-config command-line
> option.
>
> >>security.protocol=SSL
> >>ssl.truststore.location=/var/private/ssl/client.truststore.jks
> >>ssl.truststore.password=test1234
>
> http://kafka.apache.org/documentation.html#security_configclients
>
> On Mon, Jul 31, 2017 at 1:22 PM, Gabriel Machado <gm...@gmail.com>
> wrote:
>
> > Thank you for your help Vahid.
> >
> > I use kafka_2.11-0.10.0.1 with ssl.
> > kafka-consumer-groups.sh script fails with a java heap space out of
> memory.
> > Am i doing something wrong ?
> >
> > #bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server
> > myserver:9092 --list
> > Error while executing consumer group command Java heap space
> > java.lang.OutOfMemoryError: Java heap space
> > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> > at org.apache.kafka.common.network.NetworkReceive.
> > readFromReadableChannel(NetworkReceive.java:93)
> > at org.apache.kafka.common.network.NetworkReceive.
> > readFrom(NetworkReceive.java:71)
> > at org.apache.kafka.common.network.KafkaChannel.receive(
> > KafkaChannel.java:154)
> > at org.apache.kafka.common.network.KafkaChannel.read(
> > KafkaChannel.java:135)
> > at org.apache.kafka.common.network.Selector.
> > pollSelectionKeys(Selector.
> > java:323)
> > at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
> > at org.apache.kafka.clients.NetworkClient.poll(
> NetworkClient.java:260)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.
> > clientPoll(ConsumerNetworkClient.java:360)
> > at org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> > at org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> > at org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> > at kafka.admin.AdminClient.kafka$admin$AdminClient$$send(
> > AdminClient.scala:49)
> > at kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(
> > AdminClient.scala:61)
> > at kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(
> > AdminClient.scala:58)
> > at scala.collection.immutable.List.foreach(List.scala:381)
> > at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:58)
> > at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:87)
> > at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:96)
> > at kafka.admin.AdminClient.listAllGroupsFlattened(
> > AdminClient.scala:117)
> > at kafka.admin.AdminClient.listAllConsumerGroupsFlattened
> > (AdminClient.scala:121)
> > at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.
> > list(ConsumerGroupCommand.scala:311)
> > at kafka.admin.ConsumerGroupCommand$.main(
> > ConsumerGroupCommand.scala:63)
> > at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> >
> > Gabriel.
> >
> > 2017-07-28 18:28 GMT+02:00 Vahid S Hashemian <vahidhashemian@us.ibm.com
> >:
> >
> > > Hi Gabriel,
> > >
> > > I have yet to experiment with enabling SSL for Kafka.
> > > However, there are some good documents out there that seem to cover it.
> > > Examples:
> > > *
> > > https://www.confluent.io/blog/apache-kafka-security-
> > > authorization-authentication-encryption/
> > > *
> > > http://coheigea.blogspot.com/2016/09/securing-apache-kafka-
> > > broker-part-i.html
> > >
> > > Is there anything specific about the SSL and consumer groups that you
> are
> > > having issues with?
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Gabriel Machado <gm...@gmail.com>
> > > To: users@kafka.apache.org
> > > Date: 07/28/2017 08:40 AM
> > > Subject: Re: kafka-consumer-groups tool with SASL_PLAINTEXT
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > Do you know how to use consumer-group tool with ssl only (without
> sasl) ?
> > >
> > > Gabriel.
> > >
> > >
> > > Le 24 juil. 2017 11:15 PM, "Vahid S Hashemian" <
> > vahidhashemian@us.ibm.com>
> > > a écrit :
> > >
> > > Hi Meghana,
> > >
> > > I did some experiments with SASL_PLAINTEXT and documented the results
> > > here:
> > > https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
> > > I think it covers what you'd like to achieve. If not, please advise.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Meghana Narasimhan <mn...@bandwidth.com>
> > > To: users@kafka.apache.org
> > > Date: 07/24/2017 01:56 PM
> > > Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
> > >
> > >
> > >
> > > Hi,
> > > What is the correct way to use the kafka-consumer-groups tool with
> > > SASL_PLAINTEXT security enabled ?
> > >
> > > The tool seems to work fine with PLAINTEXT port but not with
> > > SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If
> so
> > > what permissions have to enabled for it ?
> > >
> > > Thanks,
> > > Meghana
> > >
> > >
> > >
> > >
> > >
> >
>
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Manikumar <ma...@gmail.com>.
We should pass necessary ssl configs using --command-config command-line
option.
>>security.protocol=SSL
>>ssl.truststore.location=/var/private/ssl/client.truststore.jks
>>ssl.truststore.password=test1234
http://kafka.apache.org/documentation.html#security_configclients
On Mon, Jul 31, 2017 at 1:22 PM, Gabriel Machado <gm...@gmail.com>
wrote:
> Thank you for your help Vahid.
>
> I use kafka_2.11-0.10.0.1 with ssl.
> kafka-consumer-groups.sh script fails with a java heap space out of memory.
> Am i doing something wrong ?
>
> #bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server
> myserver:9092 --list
> Error while executing consumer group command Java heap space
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.apache.kafka.common.network.NetworkReceive.
> readFromReadableChannel(NetworkReceive.java:93)
> at org.apache.kafka.common.network.NetworkReceive.
> readFrom(NetworkReceive.java:71)
> at org.apache.kafka.common.network.KafkaChannel.receive(
> KafkaChannel.java:154)
> at org.apache.kafka.common.network.KafkaChannel.read(
> KafkaChannel.java:135)
> at org.apache.kafka.common.network.Selector.
> pollSelectionKeys(Selector.
> java:323)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.
> clientPoll(ConsumerNetworkClient.java:360)
> at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> at kafka.admin.AdminClient.kafka$admin$AdminClient$$send(
> AdminClient.scala:49)
> at kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(
> AdminClient.scala:61)
> at kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(
> AdminClient.scala:58)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:58)
> at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:87)
> at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:96)
> at kafka.admin.AdminClient.listAllGroupsFlattened(
> AdminClient.scala:117)
> at kafka.admin.AdminClient.listAllConsumerGroupsFlattened
> (AdminClient.scala:121)
> at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.
> list(ConsumerGroupCommand.scala:311)
> at kafka.admin.ConsumerGroupCommand$.main(
> ConsumerGroupCommand.scala:63)
> at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
>
> Gabriel.
>
> 2017-07-28 18:28 GMT+02:00 Vahid S Hashemian <va...@us.ibm.com>:
>
> > Hi Gabriel,
> >
> > I have yet to experiment with enabling SSL for Kafka.
> > However, there are some good documents out there that seem to cover it.
> > Examples:
> > *
> > https://www.confluent.io/blog/apache-kafka-security-
> > authorization-authentication-encryption/
> > *
> > http://coheigea.blogspot.com/2016/09/securing-apache-kafka-
> > broker-part-i.html
> >
> > Is there anything specific about the SSL and consumer groups that you are
> > having issues with?
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Gabriel Machado <gm...@gmail.com>
> > To: users@kafka.apache.org
> > Date: 07/28/2017 08:40 AM
> > Subject: Re: kafka-consumer-groups tool with SASL_PLAINTEXT
> >
> >
> >
> > Hi Vahid,
> >
> > Do you know how to use consumer-group tool with ssl only (without sasl) ?
> >
> > Gabriel.
> >
> >
> > Le 24 juil. 2017 11:15 PM, "Vahid S Hashemian" <
> vahidhashemian@us.ibm.com>
> > a écrit :
> >
> > Hi Meghana,
> >
> > I did some experiments with SASL_PLAINTEXT and documented the results
> > here:
> > https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
> > I think it covers what you'd like to achieve. If not, please advise.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Meghana Narasimhan <mn...@bandwidth.com>
> > To: users@kafka.apache.org
> > Date: 07/24/2017 01:56 PM
> > Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
> >
> >
> >
> > Hi,
> > What is the correct way to use the kafka-consumer-groups tool with
> > SASL_PLAINTEXT security enabled ?
> >
> > The tool seems to work fine with PLAINTEXT port but not with
> > SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
> > what permissions have to enabled for it ?
> >
> > Thanks,
> > Meghana
> >
> >
> >
> >
> >
>
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Gabriel Machado <gm...@gmail.com>.
Thank you for your help Vahid.
I use kafka_2.11-0.10.0.1 with ssl.
kafka-consumer-groups.sh script fails with a java heap space out of memory.
Am i doing something wrong ?
#bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server
myserver:9092 --list
Error while executing consumer group command Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.kafka.common.network.NetworkReceive.
readFromReadableChannel(NetworkReceive.java:93)
at org.apache.kafka.common.network.NetworkReceive.
readFrom(NetworkReceive.java:71)
at org.apache.kafka.common.network.KafkaChannel.receive(
KafkaChannel.java:154)
at org.apache.kafka.common.network.KafkaChannel.read(
KafkaChannel.java:135)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.
java:323)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.
clientPoll(ConsumerNetworkClient.java:360)
at org.apache.kafka.clients.consumer.internals.
ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.
ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
at org.apache.kafka.clients.consumer.internals.
ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
at kafka.admin.AdminClient.kafka$admin$AdminClient$$send(
AdminClient.scala:49)
at kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(
AdminClient.scala:61)
at kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(
AdminClient.scala:58)
at scala.collection.immutable.List.foreach(List.scala:381)
at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:58)
at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:87)
at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:96)
at kafka.admin.AdminClient.listAllGroupsFlattened(AdminClient.scala:117)
at kafka.admin.AdminClient.listAllConsumerGroupsFlattened
(AdminClient.scala:121)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.
list(ConsumerGroupCommand.scala:311)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Gabriel.
2017-07-28 18:28 GMT+02:00 Vahid S Hashemian <va...@us.ibm.com>:
> Hi Gabriel,
>
> I have yet to experiment with enabling SSL for Kafka.
> However, there are some good documents out there that seem to cover it.
> Examples:
> *
> https://www.confluent.io/blog/apache-kafka-security-
> authorization-authentication-encryption/
> *
> http://coheigea.blogspot.com/2016/09/securing-apache-kafka-
> broker-part-i.html
>
> Is there anything specific about the SSL and consumer groups that you are
> having issues with?
>
> Thanks.
> --Vahid
>
>
>
>
> From: Gabriel Machado <gm...@gmail.com>
> To: users@kafka.apache.org
> Date: 07/28/2017 08:40 AM
> Subject: Re: kafka-consumer-groups tool with SASL_PLAINTEXT
>
>
>
> Hi Vahid,
>
> Do you know how to use consumer-group tool with ssl only (without sasl) ?
>
> Gabriel.
>
>
> Le 24 juil. 2017 11:15 PM, "Vahid S Hashemian" <va...@us.ibm.com>
> a écrit :
>
> Hi Meghana,
>
> I did some experiments with SASL_PLAINTEXT and documented the results
> here:
> https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
> I think it covers what you'd like to achieve. If not, please advise.
>
> Thanks.
> --Vahid
>
>
>
>
> From: Meghana Narasimhan <mn...@bandwidth.com>
> To: users@kafka.apache.org
> Date: 07/24/2017 01:56 PM
> Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
>
>
>
> Hi,
> What is the correct way to use the kafka-consumer-groups tool with
> SASL_PLAINTEXT security enabled ?
>
> The tool seems to work fine with PLAINTEXT port but not with
> SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
> what permissions have to enabled for it ?
>
> Thanks,
> Meghana
>
>
>
>
>
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Gabriel Machado <gm...@gmail.com>.
Hi Vahid,
Do you know how to use consumer-group tool with ssl only (without sasl) ?
Gabriel.
Le 24 juil. 2017 11:15 PM, "Vahid S Hashemian" <va...@us.ibm.com>
a écrit :
Hi Meghana,
I did some experiments with SASL_PLAINTEXT and documented the results
here:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
I think it covers what you'd like to achieve. If not, please advise.
Thanks.
--Vahid
From: Meghana Narasimhan <mn...@bandwidth.com>
To: users@kafka.apache.org
Date: 07/24/2017 01:56 PM
Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
Hi,
What is the correct way to use the kafka-consumer-groups tool with
SASL_PLAINTEXT security enabled ?
The tool seems to work fine with PLAINTEXT port but not with
SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
what permissions have to enabled for it ?
Thanks,
Meghana
Re: kafka-consumer-groups tool with SASL_PLAINTEXT
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Meghana,
I did some experiments with SASL_PLAINTEXT and documented the results
here:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
I think it covers what you'd like to achieve. If not, please advise.
Thanks.
--Vahid
From: Meghana Narasimhan <mn...@bandwidth.com>
To: users@kafka.apache.org
Date: 07/24/2017 01:56 PM
Subject: kafka-consumer-groups tool with SASL_PLAINTEXT
Hi,
What is the correct way to use the kafka-consumer-groups tool with
SASL_PLAINTEXT security enabled ?
The tool seems to work fine with PLAINTEXT port but not with
SASL_PLAINTEXT. Can it be configured to work with SASL_PLAINTEXT ? If so
what permissions have to enabled for it ?
Thanks,
Meghana