[GitHub] zsxwing edited a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

[GitBox <gi...@apache.org>]

zsxwing edited a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-463008868
 
 
   I'm thinking about what can be broken by this change (not block this PR, just to figure out what we should document in the release note). Here are two things on my list:
   
   - Add a new column to the Kafka source schema.
     - "select *" returns different schemas and results.
     - Streaming queries using "dropDuplicates(<empty parameters>)" cannot upgrade directly as the state store assumes no schema changes. The worse thing here is executor JVM will probably just crash. I think we need a better state store to maintain the binary compatibility between versions but that's out of scope and cannot be resolved by this PR.
   - The minimum required version of Kafka client changes from 0.10.x to 0.11.x.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org