[jira] [Created] (SPARK-55666) [PYTHON][TEST] test_join_ambiguous_cols is flaky due to non-deterministic row order

["Yicong Huang (Jira)" <ji...@apache.org>]

Yicong Huang created SPARK-55666:
------------------------------------

             Summary: [PYTHON][TEST] test_join_ambiguous_cols is flaky due to non-deterministic row order
                 Key: SPARK-55666
                 URL: https://issues.apache.org/jira/browse/SPARK-55666
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Yicong Huang


The test {{test_join_ambiguous_cols}} in {{python/pyspark/sql/tests/connect/test_connect_basic.py}} uses {{assertEqual(cdf.collect(), sdf.collect())}} to compare join results without sorting, which can fail non-deterministically because join does not guarantee row order.

Affected assertions (lines 324, 330, 340, 346, 352): all use {{assertEqual(x.collect(), y.collect())}} directly.

Fix: replace with {{assertEqual(sorted(x.collect()), sorted(y.collect()))}} for all 5 assertions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org