[GitHub] [spark] HyukjinKwon commented on pull request #34089: [SPARK-36837][BUILD] Upgrade Kafka to 3.1.0

[GitBox <gi...@apache.org>]

HyukjinKwon commented on pull request #34089:
URL: https://github.com/apache/spark/pull/34089#issuecomment-1035871660


   I don't have enough background on Kafka but here are the summary from what I understood by reading comments here:
   
   - This PR was reviewed by multiple committers.
   - Kafka is one of the main dependency in Structured Streaming so it might need some more extra care and reviews. For example, issues such as KAFKA-13322 might cause a non-trivial impact.
   
   For the former, this PR was reviewed by multiple committers when it was merged so I don't think there is a particular problem.
   For the latter, I agree that we might need some more extra care for major dependency changes such as Parquet. In my experience at the Spark community, we put extra care of the major version upgrade when a library has many issues after last few upgrades. Might be good to document and clarify this somewhere.
   
   Since there were a couple of post-reviews and concerns here (https://github.com/apache/spark/pull/34089#issuecomment-1033397906 and https://github.com/apache/spark/pull/34089#issuecomment-1033406639), it might be great to have some feedback from Kafka maintainer(s) about the upgrade, stability and safety.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org