(spark) branch master updated: [SPARK-59555][PYTHON][DOC] Fix flaky DataFrameReader.json doctest by sorting multi-file read

[yi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 50dab9ebdad0 [SPARK-59555][PYTHON][DOC] Fix flaky DataFrameReader.json doctest by sorting multi-file read
50dab9ebdad0 is described below

commit 50dab9ebdad0ce432e58bd71ecfcaff78bc73a86
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Wed Sep 16 15:28:17 2026 +0000

    [SPARK-59555][PYTHON][DOC] Fix flaky DataFrameReader.json doctest by sorting multi-file read
    
    ### What changes were proposed in this pull request?
    
    Example 2 in the `DataFrameReader.json` docstring writes two rows into two separate directories (`d1`, `d2`) and then reads them back with `spark.read.json([d1, d2]).show()` without any ordering. This PR adds `.sort("name")` before `.show()` so the output is deterministic. The expected output block is unchanged, since `Alice`/`Bob` is already in sorted order.
    
    ### Why are the changes needed?
    
    Reading multiple files/directories back produces rows in a nondeterministic order, so the doctest fails intermittently (~50% of the time). This makes the `pyspark.sql.connect.readwriter` doctest module (which reuses these docstrings) flaky. Sorting makes the example deterministic, consistent with Example 4 and the parquet/text examples in the same file that already call `.sort(...)`/`.orderBy(...)`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Docstring example only.
    
    ### How was this patch tested?
    
    Existing doctests. The expected output is unchanged because the sorted order matches the previously expected rows.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No
    
    Closes #58840 from Yicong-Huang/flaky-readwriter-json-doctest.
    
    Authored-by: Yicong Huang <17...@users.noreply.github.com>
    Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
---
 python/pyspark/sql/readwriter.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 73ac96701b82..238ac766912c 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -423,7 +423,7 @@ class DataFrameReader(OptionUtils):
         ...     spark.createDataFrame(
         ...         [{"age": 25, "name": "Alice"}]
         ...     ).write.mode("overwrite").format("json").save(d2)
-        ...     spark.read.json([d1, d2]).show()
+        ...     spark.read.json([d1, d2]).sort("name").show()
         +---+-----+
         |age| name|
         +---+-----+


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org