You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2022/09/13 12:54:00 UTC

[jira] [Created] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

Jungtaek Lim created SPARK-40414:
------------------------------------

             Summary: Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data
                 Key: SPARK-40414
                 URL: https://issues.apache.org/jira/browse/SPARK-40414
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 3.4.0
            Reporter: Jungtaek Lim


During the work of flatMapGroupsWithState in PySpark, we figured out that we are unable to reuse PythonArrowInput and PythonArrowOutput, as PythonArrowInput and PythonArrowOutput are too specific to the strict input data (row) and output data.

To reuse the implementations we should make these traits more general to handle more generic type of data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org