[jira] [Updated] (SPARK-58019) Convert Arrow list columns to Python rows in bulk

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58019:
-----------------------------------
    Labels: pull-request-available  (was: )

> Convert Arrow list columns to Python rows in bulk
> -------------------------------------------------
>
>                 Key: SPARK-58019
>                 URL: https://issues.apache.org/jira/browse/SPARK-58019
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 5.0.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> PyArrow's {{Array.to_pylist()}} materializes one Scalar per element; for list-typed columns each row additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array wrapper before converting elements one by one. This makes Arrow-optimized Python UDF inputs and Spark Connect {{collect()}} several times slower on array columns than necessary. This is the root cause of the performance regression observed when Arrow-serialized Python UDFs read array columns.
> Upstream Arrow issue: https://github.com/apache/arrow/issues/50326 (fix proposed in https://github.com/apache/arrow/pull/50327), but that will only be available in a future PyArrow release.
> Until the minimum supported PyArrow version includes the fix, PySpark can convert list columns in bulk itself using only public PyArrow APIs: convert the flattened child values of a list column in a single pass, then slice the resulting Python list per row using the offsets and validity bitmap. Leaf values are still converted by Arrow's own {{to_pylist}}, so results are exactly identical (unlike a pandas round trip, which coerces {{[1, None, 3]}} in an int list to floats/NaN).
> ASV microbenchmark (1M rows): {{list<string>}} 769ms -> 507ms (1.5x); {{list<list<int32>>}} with nulls 1.86s -> 537ms (3.5x); peak memory unchanged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org