You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "kete1987 (via GitHub)" <gi...@apache.org> on 2026/06/10 09:25:35 UTC

[PR] [SPARK-57365][SS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

kete1987 opened a new pull request, #56427:
URL: https://github.com/apache/spark/pull/56427

   ### What changes were proposed in this pull request?
   
   Make the test \\ esolved start offset greater than end offset (without latest)\\ in \KafkaRelationSuite\ deterministic. The two messages per partition are now produced with explicit, increasing CreateTime timestamps (\ase\, \ase + 1\) via \RecordBuilder\, so the second message (offset 1) is deterministically resolved by \offsetsForTimes\.
   
   ### Why are the changes needed?
   
   The test is flaky. It produces two messages per partition with no explicit timestamp and uses the second message's CreateTime as the ending-offset timestamp, expecting \offsetsForTimes\ to resolve to offset 1. When both messages land in the same millisecond they share a timestamp, \offsetsForTimes\ resolves to offset 0, and the hardcoded \\ esolved end offset 1\\ assertion fails with \\ esolved end offset 0\\. The existing \eventually(60.seconds)\ does not help: produced timestamps are fixed at produce time, so retrying always resolves the same offset (~3800 retries before timeout). Introduced by SPARK-52096.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Test-only change.
   
   ### How was this patch tested?
   
   A/B, 50 runs x 5 suite variants (V1 / V2 / WithAdminV1 / WithAdminV2) = 250 test instances per arm:
   - With the fix: 250/250 passed.
   - Forcing the race (both messages produced with the same timestamp) reproduces the failure deterministically: \\The resolved start offset 3 is greater than the resolved end offset 0 ...\\ vs the expected \\... resolved end offset 1 ...\\, with \eventually\ exhausting ~3800 retries.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Co-authored with Claude (Anthropic), used for analysis, code generation and review assistance.
   Generated-by: Claude Sonnet 4.6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on PR #56427:
URL: https://github.com/apache/spark/pull/56427#issuecomment-4754783581

   Thanks again @LuciferYang for the careful review! All three suggestions are applied in the latest push:
   - Removed the `eventually(60.seconds)` wrapper in favor of a flat `intercept`, matching the `(with latest)` twin.
   - `sendTwoWithDistinctTs` now returns `base + 1` directly, with a one-line note about the CreateTime assumption.
   - Replaced the three `val` assignments with `(0 to 2).map(sendTwoWithDistinctTs)`.
   
   Also corrected the PR description's test budget (50 × 4 = 200, not 250). Could you take another look when you have a chance?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #56427: [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps
URL: https://github.com/apache/spark/pull/56427


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on PR #56427:
URL: https://github.com/apache/spark/pull/56427#issuecomment-4702122167

   @sarutak @JungtaekLim would you mind taking a look when you have a chance? This fixes a flaky test in the Kafka connector introduced by SPARK-52096. Test-only change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TEST-ONLY] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on PR #56427:
URL: https://github.com/apache/spark/pull/56427#issuecomment-4753299987

   > Thank you @kete1987, please add [TEST-ONLY] to the PR title. Adding @LuciferYang @HeartSaVioR.
   
   Done, thank you @uros-b!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on code in PR #56427:
URL: https://github.com/apache/spark/pull/56427#discussion_r3444604070


##########
connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala:
##########
@@ -675,12 +675,21 @@ abstract class KafkaRelationSuiteBase extends SharedSparkSession with KafkaTest
   test("resolved start offset greater than end offset (without latest)") {
     val topic = newTopic()
     testUtils.createTopic(topic, partitions = 3)
-    val timestamp1 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(0))(1)._2.timestamp()
-    val timestamp2 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(1))(1)._2.timestamp()
-    val timestamp3 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(2))(1)._2.timestamp()
+    // Two messages per partition with explicit, increasing timestamps so the second message
+    // (offset 1) is deterministically resolved by offsetsForTimes. Without explicit timestamps
+    // both messages can share a CreateTime millisecond and resolve to offset 0, making the test
+    // flaky (the `eventually` below does not help: produced timestamps are fixed at produce time,

Review Comment:
   Agreed — removed the `eventually(60.seconds)` wrapper and replaced it with a flat `intercept`, matching the `(with latest)` twin. As you noted, the resolved-offset check fires synchronously on the driver and the produced timestamps are fixed, so retrying could never change the outcome.



##########
connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala:
##########
@@ -675,12 +675,21 @@ abstract class KafkaRelationSuiteBase extends SharedSparkSession with KafkaTest
   test("resolved start offset greater than end offset (without latest)") {
     val topic = newTopic()
     testUtils.createTopic(topic, partitions = 3)
-    val timestamp1 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(0))(1)._2.timestamp()
-    val timestamp2 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(1))(1)._2.timestamp()
-    val timestamp3 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(2))(1)._2.timestamp()
+    // Two messages per partition with explicit, increasing timestamps so the second message
+    // (offset 1) is deterministically resolved by offsetsForTimes. Without explicit timestamps
+    // both messages can share a CreateTime millisecond and resolve to offset 0, making the test
+    // flaky (the `eventually` below does not help: produced timestamps are fixed at produce time,
+    // so retrying always resolves the same wrong offset).
+    def sendTwoWithDistinctTs(part: Int): Long = {

Review Comment:
   Done — `sendTwoWithDistinctTs` now returns `base + 1` directly instead of reading back the metadata timestamp. Added a one-line note in the comment that this relies on the default CreateTime semantics (broker echoes the producer timestamp unchanged).



##########
connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala:
##########
@@ -675,12 +675,21 @@ abstract class KafkaRelationSuiteBase extends SharedSparkSession with KafkaTest
   test("resolved start offset greater than end offset (without latest)") {
     val topic = newTopic()
     testUtils.createTopic(topic, partitions = 3)
-    val timestamp1 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(0))(1)._2.timestamp()
-    val timestamp2 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(1))(1)._2.timestamp()
-    val timestamp3 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(2))(1)._2.timestamp()
+    // Two messages per partition with explicit, increasing timestamps so the second message
+    // (offset 1) is deterministically resolved by offsetsForTimes. Without explicit timestamps
+    // both messages can share a CreateTime millisecond and resolve to offset 0, making the test
+    // flaky (the `eventually` below does not help: produced timestamps are fixed at produce time,
+    // so retrying always resolves the same wrong offset).
+    def sendTwoWithDistinctTs(part: Int): Long = {
+      val base = System.currentTimeMillis()
+      testUtils.sendMessages(Seq(
+        new RecordBuilder(topic, "0").partition(part).timestamp(base).build(),
+        new RecordBuilder(topic, "0").partition(part).timestamp(base + 1).build()
+      ))(1)._2.timestamp() // = base + 1, resolves to offset 1
+    }
+    val timestamp1 = sendTwoWithDistinctTs(0)

Review Comment:
   Done — switched to `(0 to 2).map(sendTwoWithDistinctTs)` to match `prepareTimestampRelatedUnitTest`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #56427:
URL: https://github.com/apache/spark/pull/56427#issuecomment-4753660075

   > Thank you @kete1987, please add [TEST-ONLY] to the PR title. Adding @LuciferYang @HeartSaVioR.
   
   `[TESTS]` is likely more commonly used than `[TEST-ONLY]`, with a usage count ratio of 3:2773. I have reverted it back to `[TESTS]`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #56427:
URL: https://github.com/apache/spark/pull/56427#discussion_r3443883464


##########
connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala:
##########
@@ -675,12 +675,21 @@ abstract class KafkaRelationSuiteBase extends SharedSparkSession with KafkaTest
   test("resolved start offset greater than end offset (without latest)") {
     val topic = newTopic()
     testUtils.createTopic(topic, partitions = 3)
-    val timestamp1 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(0))(1)._2.timestamp()
-    val timestamp2 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(1))(1)._2.timestamp()
-    val timestamp3 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(2))(1)._2.timestamp()
+    // Two messages per partition with explicit, increasing timestamps so the second message
+    // (offset 1) is deterministically resolved by offsetsForTimes. Without explicit timestamps
+    // both messages can share a CreateTime millisecond and resolve to offset 0, making the test
+    // flaky (the `eventually` below does not help: produced timestamps are fixed at produce time,

Review Comment:
   The new comment at `KafkaRelationSuite.scala:681` admits: *"the `eventually` below does not help: produced timestamps are fixed at produce time, so retrying always resolves the same wrong offset."* Once that's said, leaving the wrapper in is dead code.
   
   No other source of intermittency exists on the path either: `createTopic` already does `waitUntilMetadataIsPropagated` (`KafkaTestUtils.scala:362-380`); `producer.send().get(10s)` waits on `ack=all` (`KafkaTestUtils.scala:434`); and the `RESOLVED_START_OFFSET_GREATER_THAN_END_OFFSET` check fires synchronously on the driver before any task runs — V1 via `KafkaRelation.buildScan` → `getOffsetRangesFromUnresolvedOffsets`, V2 via `KafkaBatch.planInputPartitions` → same — so `df.collect()` gets the exception immediately on all four concrete suites (V1, V2, WithAdminV1, WithAdminV2). Keeping `eventually` only adds up to 60 s of silent retries for any future unrelated flake, masking the very kind of issue this PR aims to surface. Twin test `(with latest)` at `KafkaRelationSuite.scala:760` already does a plain `intercept` without `eventually`; symmetry argues for the same here.
   
   Suggested change: replace the `eventually(timeout(60.seconds)) { val e = intercept[...] { df.collect() }; assert(...) }` block with a flat `val e = intercept[...] { df.collect() }; assert(...)`.
   



##########
connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala:
##########
@@ -675,12 +675,21 @@ abstract class KafkaRelationSuiteBase extends SharedSparkSession with KafkaTest
   test("resolved start offset greater than end offset (without latest)") {
     val topic = newTopic()
     testUtils.createTopic(topic, partitions = 3)
-    val timestamp1 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(0))(1)._2.timestamp()
-    val timestamp2 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(1))(1)._2.timestamp()
-    val timestamp3 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(2))(1)._2.timestamp()
+    // Two messages per partition with explicit, increasing timestamps so the second message
+    // (offset 1) is deterministically resolved by offsetsForTimes. Without explicit timestamps
+    // both messages can share a CreateTime millisecond and resolve to offset 0, making the test
+    // flaky (the `eventually` below does not help: produced timestamps are fixed at produce time,
+    // so retrying always resolves the same wrong offset).
+    def sendTwoWithDistinctTs(part: Int): Long = {

Review Comment:
   this function ends with `testUtils.sendMessages(...)((1)._2.timestamp())` to read back the metadata timestamp of the second record. With explicit producer timestamps and a CreateTime topic (Spark's `KafkaTestUtils` doesn't override `log.message.timestamp.type`, so the broker default `CreateTime` applies), `metadata.timestamp()` returns exactly `base + 1`. Skipping the round-trip and returning `base + 1` directly makes the intent unambiguous and removes an implicit dependency on broker-side timestamp semantics. If the round-trip is intentionally defensive (e.g. for a hypothetical future LogAppendTime override), say so in a one-line comment.
   
   Suggested change:
   
   ```scala
   def sendTwoWithDistinctTs(part: Int): Long = {
   val base = System.currentTimeMillis()
   testUtils.sendMessages(Seq(
    new RecordBuilder(topic, "0").partition(part).timestamp(base).build(),
    new RecordBuilder(topic, "0").partition(part).timestamp(base + 1).build()
   ))
   base + 1
   }
   ```



##########
connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala:
##########
@@ -675,12 +675,21 @@ abstract class KafkaRelationSuiteBase extends SharedSparkSession with KafkaTest
   test("resolved start offset greater than end offset (without latest)") {
     val topic = newTopic()
     testUtils.createTopic(topic, partitions = 3)
-    val timestamp1 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(0))(1)._2.timestamp()
-    val timestamp2 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(1))(1)._2.timestamp()
-    val timestamp3 =
-      testUtils.sendMessages(topic, Seq("0", "0").toArray, Some(2))(1)._2.timestamp()
+    // Two messages per partition with explicit, increasing timestamps so the second message
+    // (offset 1) is deterministically resolved by offsetsForTimes. Without explicit timestamps
+    // both messages can share a CreateTime millisecond and resolve to offset 0, making the test
+    // flaky (the `eventually` below does not help: produced timestamps are fixed at produce time,
+    // so retrying always resolves the same wrong offset).
+    def sendTwoWithDistinctTs(part: Int): Long = {
+      val base = System.currentTimeMillis()
+      testUtils.sendMessages(Seq(
+        new RecordBuilder(topic, "0").partition(part).timestamp(base).build(),
+        new RecordBuilder(topic, "0").partition(part).timestamp(base + 1).build()
+      ))(1)._2.timestamp() // = base + 1, resolves to offset 1
+    }
+    val timestamp1 = sendTwoWithDistinctTs(0)

Review Comment:
   Pure style: `(0 to 2).map(sendTwoWithDistinctTs)` matches the surrounding helper `prepareTimestampRelatedUnitTest` (`KafkaRelationSuite.scala:366-369`). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57365][SS][TESTS] Fix flaky KafkaRelationSuite resolved-offset test by using distinct message timestamps [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #56427:
URL: https://github.com/apache/spark/pull/56427#issuecomment-4760225917

   Thank you @kete1987 and @LuciferYang! Merged to master/4.x.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org