You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "JoshRosen (via GitHub)" <gi...@apache.org> on 2023/05/08 21:25:36 UTC
[GitHub] [spark] JoshRosen opened a new pull request, #41095: [SPARK-43414] Fix flakiness in Kafka RDD suites due to port binding configuration issue
JoshRosen opened a new pull request, #41095:
URL: https://github.com/apache/spark/pull/41095
### What changes were proposed in this pull request?
This PR addresses a test flakiness issue in Kafka connector RDD suites.
https://github.com/apache/spark/pull/34089#pullrequestreview-872739182 upgraded Spark to Kafka 3.1.0, which requires a different configuration key for configuring the broker listening port. That PR updated the `KafkaTestUtils.scala` used in SQL tests, but there's a near-duplicate of that code in a different `KafkaTestUtils.scala` used by RDD API suites which wasn't updated. As a result, the RDD suites began using Kafka's default port 9092 and this results in flakiness as multiple concurrent suites hit port conflicts when trying to bind to that default port.
This PR fixes that by simply copying the updated configuration from the SQL copy of `KafkaTestUtils.scala`.
### Why are the changes needed?
Fix test flakiness due to port conflicts.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran 20 concurrent copies of `org.apache.spark.streaming.kafka010.JavaKafkaRDDSuite` in my CI environment and confirmed that this PR's changes resolve the test flakiness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #41095: [SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due to port binding configuration issue
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #41095: [SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due to port binding configuration issue
URL: https://github.com/apache/spark/pull/41095
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org