You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Michal Klempa (JIRA)" <ji...@apache.org> on 2017/11/07 10:54:00 UTC
[jira] [Commented] (KAFKA-3355) GetOffsetShell command doesn't work
with SASL enabled Kafka
[ https://issues.apache.org/jira/browse/KAFKA-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241823#comment-16241823 ]
Michal Klempa commented on KAFKA-3355:
--------------------------------------
Hi, anything new on this?
I am running into same issue with Kafka 0.10.2.1. My environment follows.
Command:
{code}
KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf" kafka-run-class kafka.tools.GetOffsetShell --broker-list metis-worker5.metis.ideata:9092 --topic test_09
{code}
Output:
{code}
[2017-11-07 11:47:57,682] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2017-11-07 11:47:57,702] INFO Property client.id is overridden to GetOffsetShell (kafka.utils.VerifiableProperties)
[2017-11-07 11:47:57,702] INFO Property metadata.broker.list is overridden to metis-worker5.metis.ideata:9092 (kafka.utils.VerifiableProperties)
[2017-11-07 11:47:57,702] INFO Property request.timeout.ms is overridden to 1000 (kafka.utils.VerifiableProperties)
[2017-11-07 11:47:57,755] TRACE Instantiating Scala Sync Producer with properties: {metadata.broker.list=metis-worker5.metis.ideata:9092, request.timeout.ms=1000, port=9092, host=metis-worker5.metis.ideata, client.id=GetOffsetShell} (kafka.producer.SyncProducer)
[2017-11-07 11:47:57,757] INFO Fetching metadata from broker BrokerEndPoint(0,metis-worker5.metis.ideata,9092) with correlation id 0 for 1 topic(s) Set(test_09) (kafka.client.ClientUtils$)
[2017-11-07 11:47:57,761] TRACE verifying sendbuffer of size 61 (kafka.producer.SyncProducer)
[2017-11-07 11:47:57,771] DEBUG Created socket with SO_TIMEOUT = 1000 (requested 1000), SO_RCVBUF = 87379 (requested -1), SO_SNDBUF = 102400 (requested 102400), connectTimeoutMs = 1000. (kafka.network.BlockingChannel)
[2017-11-07 11:47:57,771] INFO Connected to metis-worker5.metis.ideata:9092 for producing (kafka.producer.SyncProducer)
[2017-11-07 11:47:57,775] TRACE 65 bytes written. (kafka.network.RequestOrResponseSend)
[2017-11-07 11:47:57,776] INFO Disconnecting from metis-worker5.metis.ideata:9092 (kafka.producer.SyncProducer)
[2017-11-07 11:47:57,780] WARN Fetching topic metadata with correlation id 0 for topics [Set(test_09)] from broker [BrokerEndPoint(0,metis-worker5.metis.ideata,9092)] failed (kafka.client.ClientUtils$)
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:60)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:95)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
[2017-11-07 11:47:57,782] INFO Disconnecting from metis-worker5.metis.ideata:9092 (kafka.producer.SyncProducer)
Exception in thread "main" kafka.common.KafkaException: fetching topic metadata for topics [Set(test_09)] from broker [ArrayBuffer(BrokerEndPoint(0,metis-worker5.metis.ideata,9092))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:74)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:95)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
Caused by: java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:60)
... 3 more
{code}
Configuration Kerberos:
{code}
cat /etc/kafka/kafka_client_jaas.conf
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
{code}
and
{code}
cat /etc/krb5.conf
[libdefaults]
default_realm = METIS.IDEATA
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac
default_tkt_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac
permitted_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac
udp_preference_limit = 1
kdc_timeout = 3000
dns_lookup_realm = false
rdns = false
[realms]
METIS.IDEATA = {
kdc = metis-master1.metis.ideata
admin_server = metis-master1.metis.ideata
}
[domain_realm]
.metis.ideata = METIS.IDEATA
metis.ideata = METIS.IDEATA
{code}
I have ticket:
{code}
klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: kafka@METIS.IDEATA
Valid starting Expires Service principal
11/07/17 11:38:50 11/08/17 11:38:50 krbtgt/METIS.IDEATA@METIS.IDEATA
renew until 11/14/17 11:38:50
{code}
Which is UPN, also super.user in Kafka, but even if not, I have ACLs:
{code}
Current ACLs for resource `Topic:test_09`:
User:nifi has Allow permission for operations: Write from hosts: *
User:nifi has Allow permission for operations: Describe from hosts: *
User:nifi has Allow permission for operations: Read from hosts: *
User:kafka has Allow permission for operations: All from hosts: *
{code}
> GetOffsetShell command doesn't work with SASL enabled Kafka
> -----------------------------------------------------------
>
> Key: KAFKA-3355
> URL: https://issues.apache.org/jira/browse/KAFKA-3355
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Affects Versions: 0.9.0.1
> Environment: Kafka 0.9.0.1
> Reporter: TAO XIAO
> Assignee: Ashish Singh
>
> I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe this is due to old producer being used in GetOffsetShell.
> Kafka version 0.9.0.1
> Exception
> % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test --time -1
> [2016-03-04 21:43:56,597] INFO Verifying properties (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property client.id is overridden to GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden to localhost:9092 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to 1000 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,674] INFO Fetching metadata from broker BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s) Set(test) (kafka.client.ClientUtils$)
> [2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing (kafka.producer.SyncProducer)
> [2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)] failed (kafka.client.ClientUtils$)
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:498)
> at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304)
> at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36)
> at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52)
> at kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
> at kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at scala.collection.immutable.Range.foreach(Range.scala:166)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:120)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
> at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78)
> at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)