You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2023/10/16 01:01:14 UTC
[PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
LuciferYang opened a new pull request, #43379:
URL: https://github.com/apache/spark/pull/43379
This reverts commit d1bd21a2a219ebe6c5ac3fcb1e17db75af3c670c.
### What changes were proposed in this pull request?
This pr aims to revert SPARK-45502 to make the test case `KafkaSourceStressSuite` stable.
### Why are the changes needed?
The test case `KafkaSourceStressSuite` has become very unstable after the merger of SPARK-45502, with 10 out of the recent 22 tests failing because of it. Revert it for now, and we can upgrade Kafka again after resolving the test issues.
- https://github.com/apache/spark/actions/runs/6497999347/job/17648385705
- https://github.com/apache/spark/actions/runs/6502219014/job/17660900989
- https://github.com/apache/spark/actions/runs/6502591917/job/17661861797
- https://github.com/apache/spark/actions/runs/6503144598/job/17663199041
- https://github.com/apache/spark/actions/runs/6503233514/job/17663413817
- https://github.com/apache/spark/actions/runs/6504416528/job/17666334238
- https://github.com/apache/spark/actions/runs/6509796846/job/17682130466
- https://github.com/apache/spark/actions/runs/6510877112/job/17685502094
- https://github.com/apache/spark/actions/runs/6512948316/job/17691625228
- https://github.com/apache/spark/actions/runs/6516366232/job/17699813649
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1764793654
Thank you, @LuciferYang and all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1764103763
Merged into master. Thanks @HyukjinKwon @HeartSaVioR @dengziming ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "dengziming (via GitHub)" <gi...@apache.org>.
dengziming commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1765575257
> Could we please be conservative and pick bugfix versions unless the minor release unblocks us/users for some specific functionalities?
Thank you @HeartSaVioR for mention this, I agree with you about picking bugfix versions, however, this bug is introduced by me when I haven't tested it carefully and the last build accidentally succeed, so it was merged by mistake. lets wait and upgrade it to 3.6.1 later.
BTW, I have found the root cause is https://github.com/apache/kafka/pull/12590, we will send a fetch request when closing `FetchSession` which contains a deleted topic, and we are setting "auto.create.topics.enable"=true so deleted topics will come back again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1764101408
https://github.com/LuciferYang/spark/runs/17721718456
GA Passed
<img width="1244" alt="image" src="https://github.com/apache/spark/assets/1475305/b6ef7769-8d42-4c36-b691-d60e65c7fb6b">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1766862823
Thank you for the info, @dengziming .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1763581146
https://github.com/apache/spark/pull/43362
I see that the author is trying to fix it, we can also wait a little longer. But since the sbt test is in a fail-fast mode, once `KafkaSourceStressSuite ` fails, the modules to be tested later will not run, making it hard to confirm whether the subsequent tests are healthy. This is a risk point.
also cc @dongjoon-hyun @dengziming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #43379: Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0"
URL: https://github.com/apache/spark/pull/43379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org