[jira] [Resolved] (SPARK-56166) Use ArrowBatchTransformer.enforce_schema to replace manual type coercion logic

["Ruifeng Zheng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-56166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruifeng Zheng resolved SPARK-56166.
-----------------------------------
    Fix Version/s: 4.2.0
       Resolution: Fixed

Issue resolved by pull request 54967
[https://github.com/apache/spark/pull/54967]

> Use ArrowBatchTransformer.enforce_schema to replace manual type coercion logic
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-56166
>                 URL: https://issues.apache.org/jira/browse/SPARK-56166
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.2.0
>            Reporter: Yicong Huang
>            Assignee: Yicong Huang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0
>
>
> There are several places in PySpark that manually iterate over columns to coerce types using coerce_arrow_array and reassemble RecordBatches. This duplicates the logic already provided by ArrowBatchTransformer.enforce_schema.
> We should consolidate these to use enforce_schema for consistency and reduced code duplication.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org