(spark) branch master updated: [SPARK-52215][PYTHON][TESTS][FOLLOW-UP] Fix `test_arrow_udf_output_nested_arrays`

[xi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

xinrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 84858f3d8be0 [SPARK-52215][PYTHON][TESTS][FOLLOW-UP] Fix `test_arrow_udf_output_nested_arrays`
84858f3d8be0 is described below

commit 84858f3d8be029dcfd3410d1edce9117969557bb
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Thu Aug 7 11:44:14 2025 -0700

    [SPARK-52215][PYTHON][TESTS][FOLLOW-UP] Fix `test_arrow_udf_output_nested_arrays`
    
    ### What changes were proposed in this pull request?
    this test is incorrect, but it happened to pass with special partitioning
    
    ### Why are the changes needed?
    to correct a test, it fails with different envs
    
    ### Does this PR introduce _any_ user-facing change?
    no, test-only
    
    ### How was this patch tested?
    ci and manually check with pyspark shell
    
    ### Was this patch authored or co-authored using generative AI tooling?
    no
    
    Closes #51898 from zhengruifeng/fix_nest_array_test.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Xinrong Meng <xi...@apache.org>
---
 python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py b/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
index 97db29ae6590..15e5123b65ed 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
@@ -97,7 +97,7 @@ class ScalarArrowUDFTestsMixin:
         df = self.spark.createDataFrame([("hi boo",), ("bye boo",)], ["vals"])
 
         tokenize = arrow_udf(
-            lambda s: pa.array([pa.compute.ascii_split_whitespace(s).to_pylist()]),
+            lambda s: pa.array([[v] for v in pa.compute.ascii_split_whitespace(s).to_pylist()]),
             ArrayType(ArrayType(StringType())),
         )
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org