You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yi...@apache.org on 2026/08/26 06:59:13 UTC
(spark) branch branch-4.0 updated: [SPARK-58601][4.0][PYTHON][TEST] Guard mapInPandas/mapInArrow list-return tests with the legacy acceptAnyIterable flag
This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new c0f17e821255 [SPARK-58601][4.0][PYTHON][TEST] Guard mapInPandas/mapInArrow list-return tests with the legacy acceptAnyIterable flag
c0f17e821255 is described below
commit c0f17e8212556d28eebea59431bad7dff63de07b
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Wed Aug 26 06:58:59 2026 +0000
[SPARK-58601][4.0][PYTHON][TEST] Guard mapInPandas/mapInArrow list-return tests with the legacy acceptAnyIterable flag
### What changes were proposed in this pull request?
This moves the `mapInPandas` and `mapInArrow` list-return (non-`Iterator` iterable) test cases under the `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=true` flag, and leaves the default `test_map_in_pandas` / `test_map_in_arrow` cases returning a strict iterator. This mirrors the return-value contract introduced on master by SPARK-58601 (strict `Iterator` by default; any iterable accepted only under the legacy flag).
### Why are the changes needed?
The `pyspark-connect-old-client` CI job runs `branch-4.0`'s Python tests against a `master` Spark Connect server. On master, SPARK-58601 tightened `mapInPandas` / `mapInArrow` to require a strict `Iterator` return by default, so `branch-4.0`'s `test_map_in_pandas`, which returned a bare `list`, now fails against that server with `UDF_RETURN_TYPE`. Setting the legacy flag makes the server accept the list, keeping the old-client job green once this lands. The change is also safe on `bra [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing tests, restructured. `test_map_in_pandas` / `test_map_in_arrow` assert the default iterator path; the new `test_map_in_pandas_legacy_accept_any_iterable` / `test_map_in_arrow_legacy_accept_any_iterable` assert the list path under the legacy flag.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58286 from Yicong-Huang/SPARK-58601-oldclient-test.
Authored-by: Yicong Huang <17...@users.noreply.github.com>
Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
---
python/pyspark/sql/tests/arrow/test_arrow_map.py | 13 +++++++++++++
python/pyspark/sql/tests/pandas/test_pandas_map.py | 14 +++++++++-----
2 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_map.py b/python/pyspark/sql/tests/arrow/test_arrow_map.py
index 86dca7ed92d0..62b129cde982 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_map.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_map.py
@@ -51,6 +51,19 @@ class MapInArrowTestsMixin(object):
expected = df.collect()
self.assertEqual(actual, expected)
+ def test_map_in_arrow_legacy_accept_any_iterable(self):
+ # With the legacy flag enabled, returning a non-Iterator iterable (e.g. list) is accepted.
+ def list_not_iter(iterator):
+ return [batch for batch in iterator]
+
+ with self.sql_conf(
+ {"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled": True}
+ ):
+ df = self.spark.range(10)
+ actual = df.mapInArrow(list_not_iter, "id long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
+
def test_multiple_columns(self):
data = [(1, "foo"), (2, None), (3, "bar"), (4, "bar")]
df = self.spark.createDataFrame(data, "a int, b string")
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_map.py b/python/pyspark/sql/tests/pandas/test_pandas_map.py
index 777d6d167048..c003fcd32433 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_map.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_map.py
@@ -79,11 +79,15 @@ class MapInPandasTestsMixin:
expected = df.collect()
self.assertEqual(actual, expected)
- # test returning list of DataFrames
- df = self.spark.range(10, numPartitions=3)
- actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id long").collect()
- expected = df.collect()
- self.assertEqual(actual, expected)
+ def test_map_in_pandas_legacy_accept_any_iterable(self):
+ # With the legacy flag enabled, returning a non-Iterator iterable (e.g. list) is accepted.
+ with self.sql_conf(
+ {"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled": True}
+ ):
+ df = self.spark.range(10, numPartitions=3)
+ actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
def test_multiple_columns(self):
data = [(1, "foo"), (2, None), (3, "bar"), (4, "bar")]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org