(spark) branch branch-3.5 updated: [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.5 by this push:
     new e693e18c7d0e [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
e693e18c7d0e is described below

commit e693e18c7d0e9a495dcb6e9b31dac9ce2b98428c
Author: Wenchen Fan <we...@databricks.com>
AuthorDate: Fri Sep 13 15:00:53 2024 +0800

    [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    
    ### What changes were proposed in this pull request?
    
    It's possible that a logical plan instance is being shared by multiple DFs and these DFs are executed in parallel. Spark always copy stateful expressions before evaluating them, but one place is missed: `ConstantFolding` can also execute expressions. This PR fixes it.
    
    ### Why are the changes needed?
    
    avoid concurrency issues.
    
    ### Does this PR introduce _any_ user-facing change?
    
    no
    
    ### How was this patch tested?
    
    Not able to write a test for it, but this concurrency issue is quite obvious
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    no
    
    Closes #48104 from cloud-fan/constant.
    
    Authored-by: Wenchen Fan <we...@databricks.com>
    Signed-off-by: Kent Yao <ya...@apache.org>
    (cherry picked from commit 319e7cc7d0e7ba9a99f808d51a8d635a6159ce8f)
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 .../scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala     | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 456d0da54de5..c0995b273bd0 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -78,7 +78,7 @@ object ConstantFolding extends Rule[LogicalPlan] {
     // Fold expressions that are foldable.
     case e if e.foldable =>
       try {
-        Literal.create(e.eval(EmptyRow), e.dataType)
+        Literal.create(e.freshCopyIfContainsStatefulExpression().eval(EmptyRow), e.dataType)
       } catch {
         case NonFatal(_) if isConditionalBranch =>
           // When doing constant folding inside conditional expressions, we should not fail


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org