You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/05/28 01:57:33 UTC

[PR] [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type [spark]

viirya opened a new pull request, #56168:
URL: https://github.com/apache/spark/pull/56168

   ### What changes were proposed in this pull request?
   
   Fix the default `ret_type` resolution in `_MapArrowIterBenchMixin._write_scenario` in `python/benchmarks/bench_eval_type.py`. The three benchmark UDFs (`identity_udf`, `sort_udf`, `filter_udf`) all return whole `pa.RecordBatch`es with the input row schema, so their declared return type should be the inner row `StructType`, not the first nested field's data type.
   
   Before:
   
   ```python
   ret_type = schema.fields[0].dataType.fields[0].dataType  # first column's type
   ```
   
   After:
   
   ```python
   ret_type = schema.fields[0].dataType  # the row's StructType
   ```
   
   ### Why are the changes needed?
   
   `mapInArrow` UDFs are contractually `(Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]`, with the user-supplied `schema` describing the output rows as a whole. The benchmark UDFs return full batches but were declaring just one column's type, which is semantically inconsistent with the API.
   
   This did not surface as an error because `worker.py` discards `return_type` for `SQL_MAP_ARROW_ITER_UDF` in `read_single_udf` (returns `None`) and only checks `Iterator[pa.RecordBatch]` structurally via `verify_return_type`. Schema-level validation is currently absent, so the mismatched type was tolerated. If the worker ever adds schema validation for this eval type, the previous declaration would break the benchmark.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Test-only change in the benchmark module.
   
   ### How was this patch tested?
   
   - Confirmed the new default resolves to a 5-field `StructType` matching the input row schema for `sm_batch_few_col`.
   - Ran `MapArrowIterUDFTimeBench.setup` + `time_worker` for `(sm_batch_few_col, pure_ints) x (identity_udf, sort_udf, filter_udf)`.
   - Ran `MapArrowIterUDFPeakmemBench.setup` + `peakmem_worker` for `sm_batch_few_col/identity_udf`.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Yes. Generated-by: Claude Code (claude-opus-4-7)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #56168: [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type
URL: https://github.com/apache/spark/pull/56168


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55934][PYTHON][TESTS][FOLLOWUP] Fix MAP_ARROW_ITER bench UDF return type [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #56168:
URL: https://github.com/apache/spark/pull/56168#issuecomment-4561483189

   Merged to master/branch-4.x/branch-4.2.
   
   Thanks @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org