You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/05/28 01:57:33 UTC
[PR] [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type [spark]
viirya opened a new pull request, #56168:
URL: https://github.com/apache/spark/pull/56168
### What changes were proposed in this pull request?
Fix the default `ret_type` resolution in `_MapArrowIterBenchMixin._write_scenario` in `python/benchmarks/bench_eval_type.py`. The three benchmark UDFs (`identity_udf`, `sort_udf`, `filter_udf`) all return whole `pa.RecordBatch`es with the input row schema, so their declared return type should be the inner row `StructType`, not the first nested field's data type.
Before:
```python
ret_type = schema.fields[0].dataType.fields[0].dataType # first column's type
```
After:
```python
ret_type = schema.fields[0].dataType # the row's StructType
```
### Why are the changes needed?
`mapInArrow` UDFs are contractually `(Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]`, with the user-supplied `schema` describing the output rows as a whole. The benchmark UDFs return full batches but were declaring just one column's type, which is semantically inconsistent with the API.
This did not surface as an error because `worker.py` discards `return_type` for `SQL_MAP_ARROW_ITER_UDF` in `read_single_udf` (returns `None`) and only checks `Iterator[pa.RecordBatch]` structurally via `verify_return_type`. Schema-level validation is currently absent, so the mismatched type was tolerated. If the worker ever adds schema validation for this eval type, the previous declaration would break the benchmark.
### Does this PR introduce _any_ user-facing change?
No. Test-only change in the benchmark module.
### How was this patch tested?
- Confirmed the new default resolves to a 5-field `StructType` matching the input row schema for `sm_batch_few_col`.
- Ran `MapArrowIterUDFTimeBench.setup` + `time_worker` for `(sm_batch_few_col, pure_ints) x (identity_udf, sort_udf, filter_udf)`.
- Ran `MapArrowIterUDFPeakmemBench.setup` + `peakmem_worker` for `sm_batch_few_col/identity_udf`.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Code (claude-opus-4-7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #56168: [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type
URL: https://github.com/apache/spark/pull/56168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #56168:
URL: https://github.com/apache/spark/pull/56168#issuecomment-4561483189
Merged to master/branch-4.x/branch-4.2.
Thanks @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org