[jira] (SPARK-52214) Python Arrow UDF

["Ruifeng Zheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-52214 ]


    Ruifeng Zheng deleted comment on SPARK-52214:
    ---------------------------------------

was (Author: podongfeng):
Issue resolved by pull request 51292
[https://github.com/apache/spark/pull/51292]

> Python Arrow UDF
> ----------------
>
>                 Key: SPARK-52214
>                 URL: https://issues.apache.org/jira/browse/SPARK-52214
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Connect, PySpark
>    Affects Versions: 4.1.0
>            Reporter: Ruifeng Zheng
>            Assignee: Ruifeng Zheng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> Pandas UDFs (and Pandas Functions like MapInPandas) have a pandas <> arrow conversion, but:
>  * This conversion is not stable, and gets broken from time to time
> -- [The Arrow 13 upgrade|https://github.com/apache/spark/pull/42920], pandas UDFs with data/time types are all broken;
> -- [Weird behavior|https://github.com/apache/spark/commit/9d88020f246ff4af1afba5a4023e139643fb3f54] when the dataset is empty
>  * Pandas <> Arrow conversion is pretty expensive. Zero-copy is only possible [in certain narrow cases|https://arrow.apache.org/docs/python/pandas.html#zero-copy-series-conversions], e.g. StringType is not supported;
>  * The support of complex type is not good, e.g. to support {{StructType}} series, we needs to use {{pd.DataFrame}} as a workaround;
> Arrow UDF is designed to resolve above issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org