You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dengziming (via GitHub)" <gi...@apache.org> on 2023/10/12 05:08:20 UTC

[PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

dengziming opened a new pull request, #43348:
URL: https://github.com/apache/spark/pull/43348

   ### What changes were proposed in this pull request?
   Upgrade Apache Kafka from 3.4.1 to 3.6.0
   
   ### Why are the changes needed?
   A bunch of improvements.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   
   ### How was this patch tested?
   GitHub CI.
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1763584923

   > this failed test KafkaSourceStressSuite is also flaky in my local env, I'm investigating this.
   
   Let's be clear about this. Did you mean KafkaSourceStressSuite is also flaky with Kafka 3.4.1 or Kafka 3.6.0 in your local env? If the case is latter then we shouldn't ignore the flakiness.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #43348: [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0
URL: https://github.com/apache/spark/pull/43348


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1759292669

   3-times consecutive failures? Does all test cases pass in your local environment, @dengziming ? Or, do you think this PR may introduce a new flakiness?
   
   <img width="322" alt="Screenshot 2023-10-12 at 2 49 15 AM" src="https://github.com/apache/spark/assets/9700541/30646ade-dc47-4a3f-af76-5beb0544800d">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1760842784

   In the 10 builds after this pr was merged, the success rate of `KafkaSourceStressSuite` is quite low. Is there any progress in the investigation? @dengziming 
   
   - https://github.com/apache/spark/actions/runs/6497942586/job/17648192318
   - https://github.com/apache/spark/actions/runs/6497999347/job/17648385705
   - https://github.com/apache/spark/actions/runs/6502219014/job/17660900989
   - https://github.com/apache/spark/actions/runs/6502591917/job/17661861797
   - https://github.com/apache/spark/actions/runs/6503144598/job/17663199041
   - https://github.com/apache/spark/actions/runs/6503233514/job/17663413817


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1760850197

   Thank you @dengziming 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1759887309

   Merged to master for Apache Spark 4.0.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dengziming (via GitHub)" <gi...@apache.org>.
dengziming commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1760848673

   Hello @LuciferYang,  I have reproduced this locally and investigating, it seems there are more than one reason for this problem and I need some time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dengziming (via GitHub)" <gi...@apache.org>.
dengziming commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1759471002

   Thank you  @dongjoon-hyun for mention this, The first two failures are related to checkstyle problems, and this failed test `KafkaSourceStressSuite` passed in my local environment. I'm investigate this flakiness currently.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1759885295

   Thank you, @dengziming . Let me merge this your contribution. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1763308899

   If it can't be fixed quickly, can we revert this patch or ignore `KafkaSourceStressSuite` first? The recent CI is almost impossible to succeed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0 [spark]

Posted by "dengziming (via GitHub)" <gi...@apache.org>.
dengziming commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1763761255

   > is also flaky with Kafka 3.4.1 or Kafka 3.6.0 in your local env
   
   After some attempt, I find that there is a weird situation: deleted topic will com back again. this is the root cause for the flakiness, I will make some deeper debug.     


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org