You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Boyang Jerry Peng (Jira)" <ji...@apache.org> on 2022/01/27 20:30:00 UTC

[jira] [Created] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

Boyang Jerry Peng created SPARK-38046:
-----------------------------------------

             Summary: Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing
                 Key: SPARK-38046
                 URL: https://issues.apache.org/jira/browse/SPARK-38046
             Project: Spark
          Issue Type: Improvement
          Components: Tests
    Affects Versions: 3.2.0
            Reporter: Boyang Jerry Peng


There is a test call "compositeReadLimit"

 

[https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala#L460]

 

that is flaky.  The problem is because the Kakfa connector is always getting the actual system time and not advancing it manually, thus leaving room for non-deterministic behaviors especially since the source determines if "maxTriggerDelayMs" is satisfied by comparing the last trigger time with the current system time.  One can simply "sleep" at points in the test to generate different outcomes.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org