[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.1.0

["Ofir Manor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653415#comment-15653415 ] 

Ofir Manor commented on SPARK-18057:
------------------------------------

This is great, especially if we follow up by adding reading topics from a timestamp...
Do you suggest to replace the existing 0.10.0 client with 0.10.1 client, or switch to having multiple implementations? (similar to how dstream has 0.8 and 0.10 implementation).
Due to Kafka architecture, older clients can talk to newer brokers, but newer clients can not talk to older brokers (therefore, during upgrades, you always upgrade the brokers first).
So, bumping up Spark's Kafka client version practically means de-supporting Kafka broker 0.10.0 by Structured Streaming, which is why I prefer adding a second implementation...
For context, 0.10.0 was released in May 2016, 0.10.1 was released this October.

> Update structured streaming kafka from 10.0.1 to 10.1.0
> -------------------------------------------------------
>
>                 Key: SPARK-18057
>                 URL: https://issues.apache.org/jira/browse/SPARK-18057
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>            Reporter: Cody Koeninger
>
> There are a couple of relevant KIPs here, https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org