[jira] [Updated] (SPARK-58699) Short-circuit identity conversion for string and binary in LocalDataToArrowConversion

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58699:
-----------------------------------
    Labels: pull-request-available  (was: )

> Short-circuit identity conversion for string and binary in LocalDataToArrowConversion
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-58699
>                 URL: https://issues.apache.org/jira/browse/SPARK-58699
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.4.0
>            Reporter: Yicong Huang
>            Priority: Major
>              Labels: pull-request-available
>
> LocalDataToArrowConversion converts local Python rows to Arrow element by element. For string and binary leaf types the per-element converter is effectively a no-op when the element is already the target Python type: convert_string returns str inputs unchanged (str(s) is s), and convert_binary copies bytes inputs even though bytes is immutable.
> This adds a type-check fast path to convert_string and convert_binary, and inlines it into convert_array for string/binary element types so already-target-typed elements skip the per-element converter call. Non-target elements (e.g. a bool coerced to string, or a bytearray) still fall back to the existing converter, so the behavior is unchanged.
> This is orthogonal to SPARK-52796, which optimized the outer convert() assembly rather than the element-level conversion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org