You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/18 18:46:55 UTC
[GitHub] [spark] tedyu opened a new pull request, #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
tedyu opened a new pull request, #38715:
URL: https://github.com/apache/spark/pull/38715
### What changes were proposed in this pull request?
This PR upgrades Kafka to 3.3.0 release.
### Why are the changes needed?
Kafka 3.3.0 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing test suite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tedyu commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
tedyu commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1320568475
@HeartSaVioR
Can you take a look ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #38715: [SPARK-41197][BUILD] Upgrade Kafka to 3.3.1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #38715: [SPARK-41197][BUILD] Upgrade Kafka to 3.3.1
URL: https://github.com/apache/spark/pull/38715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1323204958
cc @dongjoon-hyun and @HeartSaVioR FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tedyu commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
tedyu commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1323744490
@HyukjinKwon @dongjoon-hyun @HeartSaVioR
Please take a look - tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38715: [SPARK-41197][BUILD] Upgrade Kafka to 3.3.1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1329753258
This is merged via https://github.com/apache/spark/commit/0ff201cc219884d3cbb6844732c681546e53f4d4 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tedyu commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
tedyu commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1320977594
@bjornjorgensen
I have updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bjornjorgensen commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
bjornjorgensen commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1322547054
Can it have something to do with https://github.com/apache/kafka/pull/12794 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tedyu commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
tedyu commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1323743712
Thanks @dengziming for the information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bjornjorgensen commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
bjornjorgensen commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1320967123
Hi, 3.3.0 A significant bug was found in the 3.3.0 release after artifacts were pushed to Apache and Maven central but prior to the release announcement. As a result, the decision was made to not announce 3.3.0 and instead release 3.3.1 with the fix. It is recommended that 3.3.0 not be used.
Can you try with 3.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tedyu commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
tedyu commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1322164554
I noticed the test failures.
Let me analyze the test output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tedyu commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
tedyu commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1322550518
Possible.
When would the next Kafka be released ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dengziming commented on pull request #38715: [SPARK-41197] Upgrade Kafka version to 3.3 release
Posted by GitBox <gi...@apache.org>.
dengziming commented on PR #38715:
URL: https://github.com/apache/spark/pull/38715#issuecomment-1323037097
These failures comes from [apache/kafka#12049](https://github.com/apache/kafka/pull/12049) and is described here: https://kafka.apache.org/documentation/#upgrade_33_notable
The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds batch.size, switches to the next partition. In spark kafka tests, this will result in records being sent to one partition in some tests.
One simplest solution is add `props.put("partitioner.class",classOf[org.apache.kafka.clients.producer.internals.DefaultPartitioner].getName)` in `KafkaTestUtils.producerConfiguration`, or we can implement our own partitioner, or set a small`batch.size` config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org