You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/07/08 19:50:00 UTC
[jira] [Updated] (SPARK-58050) Fuse per-row converters into bulk Arrow-to-rows conversion and bulk-assemble Arrow Python UDF results
[ https://issues.apache.org/jira/browse/SPARK-58050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-58050:
-----------------------------------
Labels: pull-request-available (was: )
> Fuse per-row converters into bulk Arrow-to-rows conversion and bulk-assemble Arrow Python UDF results
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-58050
> URL: https://issues.apache.org/jira/browse/SPARK-58050
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> Follow-up of SPARK-58019/SPARK-58023/SPARK-58024. Even with bulk {{_to_pylist}}, the Arrow Python UDF worker still runs per-row Python converters on both sides: on input, {{_create_converter}} wraps every map row into a dict and every struct row into a Row one value at a time; on output, {{LocalDataToArrowConversion}} converters copy every returned list, convert every dict to an entry list and every Row to a dict before {{pa.array}}. Worker profiling shows these per-row converters dominate the remaining gap vs pickled UDFs on nested types (pickle currently beats arrow by 1.4-3.1x on array/map/struct).
> This change:
> * adds {{ArrowTableToRowsConversion._to_rows_column}}, fusing the per-row input converter into the bulk conversion (map rows become dicts and struct rows become Rows built from flattened child columns; child converters are applied per flattened column),
> * bulk-assembles UDF results when element/field/value converters are identity: arrays pass the returned lists to {{pa.array}} directly, map results pass dicts directly (PyArrow accepts dicts for map types), struct results are transposed and assembled via {{StructArray.from_arrays}}; anything else falls back to the existing per-row path.
> Microbenchmark (400k rows): input map->dict 3.0x, input struct->Row 1.8x, output array 7.5x, output map 4.2x, output struct 4.0x — outputs identical. End-to-end UDF benchmark (6.4M rows): array_string 2.39s -> 1.77s, map<string,int> 3.65s -> 2.40s, struct 6.03s -> 4.87s on top of SPARK-58019/58023/58024; arrow-vs-pickle outputs verified identical with injected nulls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org