[spark] branch master updated: [SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due to port binding configuration issue

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 175fcfd2106 [SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due to port binding configuration issue
175fcfd2106 is described below

commit 175fcfd2106652baff53fdfb1be84638a4f6a9c9
Author: Josh Rosen <jo...@databricks.com>
AuthorDate: Mon May 8 16:06:15 2023 -0700

    [SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due to port binding configuration issue
    
    ### What changes were proposed in this pull request?
    
    This PR addresses a test flakiness issue in Kafka connector RDD suites
    
    https://github.com/apache/spark/pull/34089#pullrequestreview-872739182 (Spark 3.4.0) upgraded Spark to Kafka 3.1.0, which requires a different configuration key for configuring the broker listening port. That PR updated the `KafkaTestUtils.scala` used in SQL tests, but there's a near-duplicate of that code in a different `KafkaTestUtils.scala` used by RDD API suites which wasn't updated. As a result, the RDD suites began using Kafka's default port 9092 and this results in flakiness as [...]
    
    This PR fixes that by simply copying the updated configuration from the SQL copy of `KafkaTestUtils.scala`.
    
    ### Why are the changes needed?
    
    Fix test flakiness due to port conflicts.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    Ran 20 concurrent copies of `org.apache.spark.streaming.kafka010.JavaKafkaRDDSuite` in my CI environment and confirmed that this PR's changes resolve the test flakiness.
    
    Closes #41095 from JoshRosen/update-kafka-test-utils-to-fix-port-binding-flakiness.
    
    Authored-by: Josh Rosen <jo...@databricks.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala    | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala b/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
index 91fecacb6e7..1a7a6e9676a 100644
--- a/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
+++ b/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
@@ -240,9 +240,7 @@ private[kafka010] class KafkaTestUtils extends Logging {
   private def brokerConfiguration: Properties = {
     val props = new Properties()
     props.put("broker.id", "0")
-    props.put("host.name", localHostNameForURI)
-    props.put("advertised.host.name", localHostNameForURI)
-    props.put("port", brokerPort.toString)
+    props.put("listeners", s"PLAINTEXT://$localHostNameForURI:$brokerPort")
     props.put("log.dir", brokerLogDir)
     props.put("zookeeper.connect", zkAddress)
     props.put("zookeeper.connection.timeout.ms", "60000")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org