[jira] [Commented] (SPARK-58208) Deep-copy stateful expressions in QueryExecution.optimizedPlan and ConvertToLocalRelation to prevent concurrent optimizer races

["Vinod KC (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-58208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102118#comment-18102118 ] 

Vinod KC commented on SPARK-58208:
----------------------------------

[~marcuslin], Yes, please go ahead

> Deep-copy stateful expressions in QueryExecution.optimizedPlan and ConvertToLocalRelation to prevent concurrent optimizer races
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58208
>                 URL: https://issues.apache.org/jira/browse/SPARK-58208
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.5.0, 4.1.0, 4.0.0, 4.2.0, 4.3.0
>            Reporter: Vinod KC
>            Priority: Major
>
> When two DataFrames are derived from the same base (e.g. val t = base.select(...); val a = t.select(...); val b = t.select(...)), their logical plans share the same expression objects. If a.collect() and b.collect() run concurrently, both QueryExecution.optimizedPlan calls enter the optimizer with the same expression instances. Expressions that hold mutable state (e.g. NamedLambdaVariable with an AtomicReference) are then written by two threads simultaneously, silently corrupting results.
> ConvertToLocalRelation had the same gap: it evaluated InterpretedMutableProjection directly over the shared projectList without making independent copies first.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org