[jira] [Commented] (FLUME-2792) Flume Kafka Kerberos Support

["Roshan Naik (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990763#comment-14990763 ] 

Roshan Naik commented on FLUME-2792:
------------------------------------

Updates - 
 - PLAINTESTSAL  is being renamed to  SASL_PLAINTEXT in Apache Kafka.
 - Kafka has decided to support security only for the new Producer APIs. Apache Flume uses the old API today. So until Flume code is updated for the new APIs, this wont work.
- The Kafka that is shipped as part of HDP had secure support for the old API too. 

> Flume Kafka Kerberos Support
> ----------------------------
>
>                 Key: FLUME-2792
>                 URL: https://issues.apache.org/jira/browse/FLUME-2792
>             Project: Flume
>          Issue Type: Bug
>          Components: Configuration, Docs, Sinks+Sources
>    Affects Versions: v1.6.0, v1.5.2
>         Environment: HDP 2.3 fully kerberized including Kafka 0.8.2.2 + Flume 1.5.2 or Apache Flume 1.6 downloaded from apache.org
>            Reporter: Hari Sekhon
>            Priority: Blocker
>
> Following on from FLUME-2790 it appears as though Flume doesn't yet have support for Kafka + Kerberos as there are is no setting documented in the Flume 1.6.0 user guide under the Kafka source section to tell Flume to use plaintextsasl as the connection mechanism to Kafka and Kafka rejects unauthenticated plaintext mechanism:
> {code}15/09/10 16:51:22 INFO consumer.ConsumerFetcherManager: [ConsumerFetcherManager-1441903874830] Added fetcher for partitions ArrayBuffer()
> 15/09/10 16:51:22 WARN consumer.ConsumerFetcherManager$LeaderFinderThread: [flume_<custom_scrubbed>-1441903874763-abdc98ec-leader-finder-thread], Failed to find leader for Set([<custom_scrubbed>,0], [<custom_scrubbed>,1])
> kafka.common.BrokerEndPointNotAvailableException: End point PLAINTEXT not found for broker 0
>         at kafka.cluster.Broker.getBrokerEndPoint(Broker.scala:140)
>         at kafka.utils.ZkUtils$$anonfun$getAllBrokerEndPointsForChannel$1.apply(ZkUtils.scala:124)
>         at kafka.utils.ZkUtils$$anonfun$getAllBrokerEndPointsForChannel$1.apply(ZkUtils.scala:124)
>         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>         at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>         at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>         at kafka.utils.ZkUtils$.getAllBrokerEndPointsForChannel(ZkUtils.scala:124)
>         at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)