[jira] [Created] (SPARK-18779) Upgrade Spark Streaming integration for Kafka 0.10 to use Kafka client library at 0.10.1

["Pranav Nakhe (JIRA)" <ji...@apache.org>]

Pranav Nakhe created SPARK-18779:
------------------------------------

             Summary: Upgrade Spark Streaming integration for Kafka 0.10 to use Kafka client library at 0.10.1
                 Key: SPARK-18779
                 URL: https://issues.apache.org/jira/browse/SPARK-18779
             Project: Spark
          Issue Type: Improvement
          Components: DStreams
    Affects Versions: 2.0.2
            Reporter: Pranav Nakhe


The Spark Streaming integration for Kafka 0.10 currently pulls the kafka client 10.0.1 as a maven dependency. There is a newer kafka client available 10.1.0.  Would it be possible to have Spark Kafka integration use kafka client 10.1.0.

The reason we need the later Kafka client is because the later one (10.1.0) has introduced a new API 'offsetsForTimes' for the KafkaConsumer which look up the offsets for the given partitions by timestamp. We are using that to find the offset that needs to be fed to the createDirectStream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org