You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2025/06/20 15:02:00 UTC

[jira] [Assigned] (SPARK-52450) Improve performance of schema deepcopy

     [ https://issues.apache.org/jira/browse/SPARK-52450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Herman van Hövell reassigned SPARK-52450:
-----------------------------------------

    Assignee: Xi Lyu

> Improve performance of schema deepcopy
> --------------------------------------
>
>                 Key: SPARK-52450
>                 URL: https://issues.apache.org/jira/browse/SPARK-52450
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 4.1.0, 4.0.0, 4.0.1
>            Reporter: Xi Lyu
>            Assignee: Xi Lyu
>            Priority: Major
>              Labels: pull-request-available
>
> In Spark Connect, `DataFrame.schema` returns a deep copy of the schema to prevent unexpected behavior caused by user modifications to the returned schema object. However, if a user accesses `df.schema` repeatedly on a DataFrame with a complex schema, it can lead to noticeable performance degradation.
> Since `deepcopy` is already known for its performance overhead, we should consider using an alternative serialization/deserialization (ser/de) method to improve the efficiency of accessing `df.schema`.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org