You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/09/30 20:31:49 UTC
[PR] [SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the mapInBatch acceptAnyIterable default in mapInArrow/mapInPandas [spark]
Yicong-Huang opened a new pull request, #59168:
URL: https://github.com/apache/spark/pull/59168
### What changes were proposed in this pull request?
Add positive tests for `mapInArrow` and `mapInPandas` that return a non-`Iterator` iterable (a `list`) *without* setting `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled`, asserting the value is accepted under the config's default. The tests are added to the shared `MapInArrowTestsMixin` / `MapInPandasTestsMixin`, so they run in both the classic and Spark Connect suites.
### Why are the changes needed?
The follow-up #58436 flipped `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled` to default `true`, restoring the pre-4.3.0 lenient behavior where a `mapInPandas`/`mapInArrow` UDF may return any iterable. The existing `*_legacy_accept_any_iterable` tests set that flag to `true` explicitly, so they would still pass if the default were flipped back to `false` and therefore do not protect the default from regression. A test that relies on the default (sets no config) closes that gap; if the default regresses to `false`, the returned `list` would raise `UDF_RETURN_TYPE` and these tests would fail.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New tests pass in both classic and Spark Connect modes for `mapInArrow` and `mapInPandas`:
```
python/run-tests --testnames \
"pyspark.sql.tests.arrow.test_arrow_map MapInArrowTests.test_map_in_arrow_accept_any_iterable_by_default,\
pyspark.sql.tests.pandas.test_pandas_map MapInPandasTests.test_map_in_pandas_accept_any_iterable_by_default,\
pyspark.sql.tests.connect.arrow.test_parity_arrow_map ArrowMapParityTests.test_map_in_arrow_accept_any_iterable_by_default,\
pyspark.sql.tests.connect.pandas.test_parity_pandas_map MapInPandasParityTests.test_map_in_pandas_accept_any_iterable_by_default"
```
### Was this patch authored or co-authored using generative AI tooling?
No
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the mapInBatch acceptAnyIterable default in mapInArrow/mapInPandas [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #59168: [SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the mapInBatch acceptAnyIterable default in mapInArrow/mapInPandas
URL: https://github.com/apache/spark/pull/59168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the mapInBatch acceptAnyIterable default in mapInArrow/mapInPandas [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #59168:
URL: https://github.com/apache/spark/pull/59168#issuecomment-5936506361
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/a363a856de83c58074b8534e2ff743d4e6664c9e
- merged into branch-4.x https://github.com/apache/spark/commit/302b0847a491be486e1b2f0993d1d1fa02b11927
- merged into branch-4.3 https://github.com/apache/spark/commit/e4e57ed064b04473ac259087648ca2afa339f823
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org