You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/07/08 19:50:00 UTC

[jira] [Updated] (SPARK-58050) Fuse per-row converters into bulk Arrow-to-rows conversion and bulk-assemble Arrow Python UDF results

     [ https://issues.apache.org/jira/browse/SPARK-58050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58050:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fuse per-row converters into bulk Arrow-to-rows conversion and bulk-assemble Arrow Python UDF results
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58050
>                 URL: https://issues.apache.org/jira/browse/SPARK-58050
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 5.0.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> Follow-up of SPARK-58019/SPARK-58023/SPARK-58024. Even with bulk {{_to_pylist}}, the Arrow Python UDF worker still runs per-row Python converters on both sides: on input, {{_create_converter}} wraps every map row into a dict and every struct row into a Row one value at a time; on output, {{LocalDataToArrowConversion}} converters copy every returned list, convert every dict to an entry list and every Row to a dict before {{pa.array}}. Worker profiling shows these per-row converters dominate the remaining gap vs pickled UDFs on nested types (pickle currently beats arrow by 1.4-3.1x on array/map/struct).
> This change:
> * adds {{ArrowTableToRowsConversion._to_rows_column}}, fusing the per-row input converter into the bulk conversion (map rows become dicts and struct rows become Rows built from flattened child columns; child converters are applied per flattened column),
> * bulk-assembles UDF results when element/field/value converters are identity: arrays pass the returned lists to {{pa.array}} directly, map results pass dicts directly (PyArrow accepts dicts for map types), struct results are transposed and assembled via {{StructArray.from_arrays}}; anything else falls back to the existing per-row path.
> Microbenchmark (400k rows): input map->dict 3.0x, input struct->Row 1.8x, output array 7.5x, output map 4.2x, output struct 4.0x — outputs identical. End-to-end UDF benchmark (6.4M rows): array_string 2.39s -> 1.77s, map<string,int> 3.65s -> 2.40s, struct 6.03s -> 4.87s on top of SPARK-58019/58023/58024; arrow-vs-pickle outputs verified identical with injected nulls.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org