You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Yicong Huang (Jira)" <ji...@apache.org> on 2026/01/26 22:15:00 UTC

[jira] [Created] (SPARK-55221) Add PandasBatchTransformer.to_arrow and remove _create_struct_array

Yicong Huang created SPARK-55221:
------------------------------------

             Summary: Add PandasBatchTransformer.to_arrow and remove _create_struct_array
                 Key: SPARK-55221
                 URL: https://issues.apache.org/jira/browse/SPARK-55221
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Yicong Huang


Add a `to_arrow` method to PandasBatchTransformer that converts a pandas DataFrame directly to Arrow format.

This consolidates the pandas-to-arrow conversion logic currently spread across serializer methods like `_create_struct_array` and `_create_array`, enabling simpler composition with existing ArrowBatchTransformer methods.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org