[jira] [Resolved] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-40414.
----------------------------------
    Fix Version/s: 3.4.0
       Resolution: Fixed

Issue resolved by pull request 37864
[https://github.com/apache/spark/pull/37864]

> Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-40414
>                 URL: https://issues.apache.org/jira/browse/SPARK-40414
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 3.4.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>             Fix For: 3.4.0
>
>
> During the work of flatMapGroupsWithState in PySpark, we figured out that we are unable to reuse PythonArrowInput and PythonArrowOutput, as PythonArrowInput and PythonArrowOutput are too specific to the strict input data (row) and output data.
> To reuse the implementations we should make these traits more general to handle more generic type of data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org