[jira] [Resolved] (SPARK-59784) Fix racy waits for SQL execution metrics in tests

["Uroš Bojanić (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uroš Bojanić resolved SPARK-59784.
----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 59043
[https://github.com/apache/spark/pull/59043]

> Fix racy waits for SQL execution metrics in tests
> -------------------------------------------------
>
>                 Key: SPARK-59784
>                 URL: https://issues.apache.org/jira/browse/SPARK-59784
>             Project: Spark
>          Issue Type: Test
>          Components: SQL, Tests
>    Affects Versions: 4.4.0
>            Reporter: sepuri sai krishna
>            Assignee: sepuri sai krishna
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> {{SQLAppStatusListenerSuite}} test "driver side SQL metrics" fails intermittently in CI. It has been seen repeatedly in the scheduled master and branch-4.x builds, in both the {{SQLAppStatusListenerWithInMemoryStoreSuite}} and {{SQLAppStatusListenerWithRocksDBBackendSuite}} variants, and in unrelated PR builds. Example failure:
> {noformat}
> - driver side SQL metrics *** FAILED *** (21 milliseconds)
>   Map(3105033 -> "2", 3105032 -> "1", 3105031 -> "total (min, med, max (stageId: taskId))
>   0 ms (0 ms, 0 ms, 0 ms (stage 5.0: task 10))") did not contain key 3105125 (SQLAppStatusListenerSuite.scala:607)
> {noformat}
> The wait logic in the test is racy:
> # {{while (statusStore.executionsCount() < oldCount)}} can never be true, because the count only grows, so it never waits.
> # {{SQLAppStatusListener}} processes events asynchronously. If it has not yet processed the new execution, {{statusStore.executionsList().last}} is an execution left over from a previous test. That execution already has metric values, so the second wait loop exits immediately and the test reads the wrong execution's metrics. The map in the failure above is exactly such a previous execution's metrics.
> This can be reproduced deterministically by stalling the status listener queue for a couple of seconds while the test runs.
> Proposed fix: capture the execution id inside {{SQLExecution.withNewExecutionId}} and use {{eventually}} to wait until that specific execution has its metrics, instead of looking at the latest execution in the store.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org