You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/01/27 20:51:00 UTC

[jira] [Commented] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

    [ https://issues.apache.org/jira/browse/SPARK-38046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483432#comment-17483432 ] 

Apache Spark commented on SPARK-38046:
--------------------------------------

User 'jerrypeng' has created a pull request for this issue:
https://github.com/apache/spark/pull/35343

> Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing
> --------------------------------------------------------------------------
>
>                 Key: SPARK-38046
>                 URL: https://issues.apache.org/jira/browse/SPARK-38046
>             Project: Spark
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 3.2.0
>            Reporter: Boyang Jerry Peng
>            Priority: Major
>
> There is a test call "compositeReadLimit"
>  
> [https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala#L460]
>  
> that is flaky.  The problem is because the Kakfa connector is always getting the actual system time and not advancing it manually, thus leaving room for non-deterministic behaviors especially since the source determines if "maxTriggerDelayMs" is satisfied by comparing the last trigger time with the current system time.  One can simply "sleep" at points in the test to generate different outcomes.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org