You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org> on 2026/08/19 11:55:04 UTC

[PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

matvei-zamiatin-db opened a new pull request, #58119:
URL: https://github.com/apache/spark/pull/58119

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This is a behavior-preserving refactor that exposes expression-level entry points on several `FinishAnalysis` rules so they can rewrite a single `Expression` tree, not just a whole `LogicalPlan`. Each rule's rewrite logic is factored into a reusable partial function, the existing plan-level apply delegates to it, and a new `applyForExpression` (or widened visibility) is added.
   
   No plan-level rule changes its output, the extracted partial functions are identical to the original inline bodies.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   The single-pass analyzer needs to apply these `FinishAnalysis` rewrites to individual, already-resolved expression trees rather than by running a full plan-wide rule pass. Today the logic is reachable only through each rule's plan-level apply, so it can't be invoked on a single `Expression`. Extracting shared entry points makes the rewrites reusable.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   No. This is an internal refactor. All plan-level rules produce the same output as before. 
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   
   No new tests were added - the change is a behavior-preserving refactor with no new callers.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   Generated-by: Claude Opus 4.8
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3872870504


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }
+    // `current_time()` and a TIME -> TIMESTAMP cast are not leaves and their only leaf is a
+    // literal, so the tree walk accepts them and only these patterns reject them.
+    !e.containsAnyPattern(CURRENT_LIKE, CAST_TO_TIMESTAMP) && isLiteralTree(e)

Review Comment:
   CAST_TO_TIMESTAMP rejects every timestamp-target cast, although only TIME -> TIMESTAMP is unsafe. A repeated CAST('1970-01-01' AS TIMESTAMP) incorrectly throws.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3862336331


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,23 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }

Review Comment:
   applyForExpression duplicates every With definition, violating its evaluate-once contract for non-foldable/stateful expressions. Procedure arguments are foldable, so the intended caller is safe, but that precondition is neither enforced nor documented. Restrict the helper to foldable definitions or reject unsupported expressions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3893831607


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }

Review Comment:
   One residual minor concern (fail-safe, currently callerless). The ComputeCurrentTime → RewriteWithExpression.applyForExpression pipeline spuriously throws on an already-stabilized TIME→TIMESTAMP StaticInvoke referenced >1 time unless ConstantFolding runs between the steps. You addressed the earlier flag only via comment wording, but not a code guard.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58119:
URL: https://github.com/apache/spark/pull/58119#issuecomment-5512322157

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/09515e6cb2a4ba2ab400ad597966a5e6688cc1c8
   - merged into branch-4.x https://github.com/apache/spark/commit/766db3c655bcdac369290f2c2752d39bdea42733
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3862542059


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,23 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }

Review Comment:
   Fixed



##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceExpressionsSuite.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Add, Coalesce, Literal, Nvl, RuntimeReplaceable}
+import org.apache.spark.sql.catalyst.plans.PlanTest
+
+class ReplaceExpressionsSuite extends PlanTest {

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3894759830


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }

Review Comment:
   Added branch which checks `StaticInvoke` 
   ```scala
         // A TIME -> TIMESTAMP cast stabilized by ComputeCurrentTime is a pure builder StaticInvoke;
         // accept it here before the blanket NonSQLExpression rejection below.
         case _ if ComputeCurrentTime.isStabilizedTimeToTimestamp(e) => e.children.forall(isLiteralTree)
   ```
   
   ```scala
     /** Whether `e` is a `DateTimeUtils.makeTimestamp*` builder that `expressionTransform` emits. */
     private[optimizer] def isStabilizedTimeToTimestamp(e: Expression): Boolean = e match {
       case si: StaticInvoke if si.staticObject == classOf[DateTimeUtils.type] =>
         si.functionName.startsWith("makeTimestamp")
       case _ => false
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814855087


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -112,106 +112,128 @@ object EvalInlineTables extends Rule[LogicalPlan] with CastSupport {
  */
 object ComputeCurrentTime extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = {
-    val instant = Instant.now()
-    val currentTimestampMicros = instantToMicros(instant)
-    val currentTime = Literal.create(currentTimestampMicros, TimestampType)
-    val currentTimeOfDayNanos = instantToNanosOfDay(instant, conf.sessionLocalTimeZone)
-    val timezone = Literal.create(conf.sessionLocalTimeZone, StringType)
+    val snapshot = new TimeSnapshot(Instant.now())
+    plan.transformDownWithSubqueriesAndPruning(transformCondition) {
+      case subQuery =>
+        subQuery.transformAllExpressionsWithPruning(transformCondition)(
+          expressionTransform(snapshot))
+    }
+  }
+
+  /** Rewrites the current date/time functions in a single expression tree. */
+  def applyForExpression(expression: Expression): Expression =
+    applyForExpression(expression, Instant.now())
+
+  /**
+   * Rewrites the current date/time functions in a single expression tree using `instant`. Callers
+   * reducing several expressions that must observe the same wall clock pass one shared instant.
+   */
+  def applyForExpression(expression: Expression, instant: Instant): Expression = {
+    val snapshot = new TimeSnapshot(instant)
+    expression.transformWithPruning(transformCondition)(expressionTransform(snapshot))

Review Comment:
   Added comment for every `applyForExpression`:
   ```
   Does not descend into subquery plans (e.g. `ScalarSubquery`); a caller whose expression
   may contain a subquery must rewrite those plans separately.
   ```
   
   Since we want to use it for procedure's arguments reducing, where ScalarSubquery is prohibited, it should work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58119:
URL: https://github.com/apache/spark/pull/58119#issuecomment-5508858887

   Just two more minor nits:
   
   - DateTimeUtils.makeTimestampX reading a clock would slip past the literal-children gate. Worth either softening the doc wording or matching the specific builder types if strict scoping was the intent.
   
   - Only the NTZ path is directly tested. The other three variants take the identical helper path (I verified each passes the gate), so coverage is logically equivalent, a one-line LTZ case would just make that a bit explicit.
   
   Otherwise looks good, thank you @matvei-zamiatin-db for addressing all previous comments!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58119:
URL: https://github.com/apache/spark/pull/58119#issuecomment-5510370734

   Thank you for working on this @matvei-zamiatin-db, let's also loop in @cloud-fan who is also very familiar with core SQL / catalyst optimizer code, for additional review on the FinishAnalysis batch refactoring and subquery traversal semantics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3879948403


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }
+    // `current_time()` and a TIME -> TIMESTAMP cast are not leaves and their only leaf is a
+    // literal, so the tree walk accepts them and only these patterns reject them.
+    !e.containsAnyPattern(CURRENT_LIKE, CAST_TO_TIMESTAMP) && isLiteralTree(e)

Review Comment:
   Fixed with rejecting only
   ```scala
         case c: Cast
             if Cast.isTimeToTimestampNTZ(c.child.dataType, c.dataType) ||
               Cast.isTimeToTimestampLTZ(c.child.dataType, c.dataType) => false
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }

Review Comment:
   Fixed comments



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {

Review Comment:
   Added rejection for canonicalized ids



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -53,7 +53,7 @@ object ReplaceExpressions extends Rule[LogicalPlan] {
     case p => p.mapExpressions(replace)
   }
 
-  private def replace(e: Expression): Expression = e match {
+  def replace(e: Expression): Expression = e match {

Review Comment:
   Yes, it should've been in previous commit but was reverted for some reason



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3872696535


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,34 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`. Inlining duplicates a definition at
+   * every reference, so this rejects non-foldable definitions: inlining one referenced more than
+   * once would evaluate it more than once, breaking `With`'s evaluate-once contract.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression =

Review Comment:
   Yes, replaced on every `applyForExpression`



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,34 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`. Inlining duplicates a definition at
+   * every reference, so this rejects non-foldable definitions: inlining one referenced more than
+   * once would evaluate it more than once, breaking `With`'s evaluate-once contract.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, rejectNonFoldableDefs = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to allow non-foldable defs.
+  private def inlineWith(expression: Expression, rejectNonFoldableDefs: Boolean): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {

Review Comment:
   For this case we need to do two traversals to understand which defs are some combination of literals, because With is resolved bottom up and defs deciding is top down. Added `isPureLiteralTree` which check if expr is some combination of literals and top down traversal to memorize ids that are safe to inline



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,34 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`. Inlining duplicates a definition at
+   * every reference, so this rejects non-foldable definitions: inlining one referenced more than
+   * once would evaluate it more than once, breaking `With`'s evaluate-once contract.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, rejectNonFoldableDefs = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to allow non-foldable defs.
+  private def inlineWith(expression: Expression, rejectNonFoldableDefs: Boolean): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (rejectNonFoldableDefs && !defs.forall(_.child.foldable)) {

Review Comment:
   Replaced with check of ref count for definition:
   - ref count <= 1 -> inline
   - ref count > 1   -> throw exception if it's not a `Literal` tree



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3872876043


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }

Review Comment:
   The documented preprocessing still rejects TIME -> TIMESTAMP. ComputeCurrentTime produces a pure StaticInvoke, which line 160 rejects as NonSQLExpression. Constant folding would mask both issues, but it is neither documented nor invoked before this helper.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814855087


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -112,106 +112,128 @@ object EvalInlineTables extends Rule[LogicalPlan] with CastSupport {
  */
 object ComputeCurrentTime extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = {
-    val instant = Instant.now()
-    val currentTimestampMicros = instantToMicros(instant)
-    val currentTime = Literal.create(currentTimestampMicros, TimestampType)
-    val currentTimeOfDayNanos = instantToNanosOfDay(instant, conf.sessionLocalTimeZone)
-    val timezone = Literal.create(conf.sessionLocalTimeZone, StringType)
+    val snapshot = new TimeSnapshot(Instant.now())
+    plan.transformDownWithSubqueriesAndPruning(transformCondition) {
+      case subQuery =>
+        subQuery.transformAllExpressionsWithPruning(transformCondition)(
+          expressionTransform(snapshot))
+    }
+  }
+
+  /** Rewrites the current date/time functions in a single expression tree. */
+  def applyForExpression(expression: Expression): Expression =
+    applyForExpression(expression, Instant.now())
+
+  /**
+   * Rewrites the current date/time functions in a single expression tree using `instant`. Callers
+   * reducing several expressions that must observe the same wall clock pass one shared instant.
+   */
+  def applyForExpression(expression: Expression, instant: Instant): Expression = {
+    val snapshot = new TimeSnapshot(instant)
+    expression.transformWithPruning(transformCondition)(expressionTransform(snapshot))

Review Comment:
   Added comment for every `applyForExpression`:
   ```
   Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
   may contain a subquery must rewrite those plans separately.
   ```
   
   Since we want to use it only for procedure's arguments reducing, where ScalarSubquery is prohibited, it should work fine



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3865189605


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,34 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`. Inlining duplicates a definition at
+   * every reference, so this rejects non-foldable definitions: inlining one referenced more than
+   * once would evaluate it more than once, breaking `With`'s evaluate-once contract.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression =

Review Comment:
   Can this be private[sql]?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3872880106


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {

Review Comment:
   safeIds assumes globally unique IDs, but sibling canonicalized With expressions both start numbering at 1. A safe sibling can therefore authorize duplication of an unsafe definition. Reject canonicalized IDs as the plan-level path does, or scope safety per With. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814855087


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -112,106 +112,128 @@ object EvalInlineTables extends Rule[LogicalPlan] with CastSupport {
  */
 object ComputeCurrentTime extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = {
-    val instant = Instant.now()
-    val currentTimestampMicros = instantToMicros(instant)
-    val currentTime = Literal.create(currentTimestampMicros, TimestampType)
-    val currentTimeOfDayNanos = instantToNanosOfDay(instant, conf.sessionLocalTimeZone)
-    val timezone = Literal.create(conf.sessionLocalTimeZone, StringType)
+    val snapshot = new TimeSnapshot(Instant.now())
+    plan.transformDownWithSubqueriesAndPruning(transformCondition) {
+      case subQuery =>
+        subQuery.transformAllExpressionsWithPruning(transformCondition)(
+          expressionTransform(snapshot))
+    }
+  }
+
+  /** Rewrites the current date/time functions in a single expression tree. */
+  def applyForExpression(expression: Expression): Expression =
+    applyForExpression(expression, Instant.now())
+
+  /**
+   * Rewrites the current date/time functions in a single expression tree using `instant`. Callers
+   * reducing several expressions that must observe the same wall clock pass one shared instant.
+   */
+  def applyForExpression(expression: Expression, instant: Instant): Expression = {
+    val snapshot = new TimeSnapshot(instant)
+    expression.transformWithPruning(transformCondition)(expressionTransform(snapshot))

Review Comment:
   Added comment for every `applyForExpression`:
   ```
   Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
   may contain a subquery must rewrite those plans separately, reusing the same `instant`.
   ```
   
   Since we want to use it only for procedure's arguments reducing, where ScalarSubquery is prohibited, it should work fine



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3865178866


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,34 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`. Inlining duplicates a definition at
+   * every reference, so this rejects non-foldable definitions: inlining one referenced more than
+   * once would evaluate it more than once, breaking `With`'s evaluate-once contract.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, rejectNonFoldableDefs = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to allow non-foldable defs.
+  private def inlineWith(expression: Expression, rejectNonFoldableDefs: Boolean): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (rejectNonFoldableDefs && !defs.forall(_.child.foldable)) {

Review Comment:
   CurrentTimestampNanos is foldable but evaluates Instant.now(). Inlining two references can produce different values, violating With’s evaluate-once contract.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58119:
URL: https://github.com/apache/spark/pull/58119#issuecomment-5512256986

   Verdict: 0 blocking, 0 non-blocking, 0 nits — clean, ready to merge.
   
   What this PR does
   
   It factors several FinishAnalysis optimizer rules so their rewrites can run on a single Expression, not just a whole plan — prep work for the single-pass analyzer. Each rule's inline body becomes a reusable PartialFunction with a new applyForExpression entry point (ComputeCurrentTime, ReplaceExpressions, ReplaceCurrentLike, SpecialDatetimeValues, RewriteWithExpression). TimeTravelSpec is converted to the new entry points, dropping a throwaway Project/OneRowRelation wrapper and two casts.
   
   What I verified independently
   
   - Behavior preservation: the extracted partial functions match the old inline arms one-for-one; ComputeCurrentTime.apply still shares one TimeSnapshot across subqueries; ReplaceCurrentLike.currentLikeRewrite is correctly a def (a val would stale the lazy current-catalog/user vals).
   - The one intentional plan-level change (the refToExpr.contains guard) only turns a NoSuchElementException crash into the correct deferred inline — never changes an existing result — and mirrors the pre-existing sibling guard. Traced the new plan-level test end-to-end.
   - TimeTravelSpec one-liner is equivalent to the old fake-Project approach; its no-subquery/no-reference preconditions hold.
   - isSafeToDuplicate is conservative (errs toward throwing), the makeTimestamp whitelist is correctly ordered before the NonSQLExpression rejection, and the entry points beyond the two TimeTravelSpec uses have no production caller yet.
   - Rule order: the test's finishAnalysisAndInline composition matches the real FinishAnalysis batch order in Optimizer.scala.
   
   Existing feedback
   
   All 17 discussion surfaces adjudicated (14 resolved, 3 non-findings). Every concern from the earlier uros-b review rounds — including the guard suggestion, the safe-to-duplicate machinery, the private[sql] scoping, and the two final nits (makeTimestamp clock wording, LTZ test) — is resolved in the current tree.
   
   One forward-looking note carried in the verdict (not a finding): when the single-pass analyzer wires these in, a caller reducing multiple expressions of one query should use the shared-instant two-arg ComputeCurrentTime.applyForExpression(expr, instant) overload so current_timestamp() stays stable within the query.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814855087


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -112,106 +112,128 @@ object EvalInlineTables extends Rule[LogicalPlan] with CastSupport {
  */
 object ComputeCurrentTime extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = {
-    val instant = Instant.now()
-    val currentTimestampMicros = instantToMicros(instant)
-    val currentTime = Literal.create(currentTimestampMicros, TimestampType)
-    val currentTimeOfDayNanos = instantToNanosOfDay(instant, conf.sessionLocalTimeZone)
-    val timezone = Literal.create(conf.sessionLocalTimeZone, StringType)
+    val snapshot = new TimeSnapshot(Instant.now())
+    plan.transformDownWithSubqueriesAndPruning(transformCondition) {
+      case subQuery =>
+        subQuery.transformAllExpressionsWithPruning(transformCondition)(
+          expressionTransform(snapshot))
+    }
+  }
+
+  /** Rewrites the current date/time functions in a single expression tree. */
+  def applyForExpression(expression: Expression): Expression =
+    applyForExpression(expression, Instant.now())
+
+  /**
+   * Rewrites the current date/time functions in a single expression tree using `instant`. Callers
+   * reducing several expressions that must observe the same wall clock pass one shared instant.
+   */
+  def applyForExpression(expression: Expression, instant: Instant): Expression = {
+    val snapshot = new TimeSnapshot(instant)
+    expression.transformWithPruning(transformCondition)(expressionTransform(snapshot))

Review Comment:
   Added comment for every `applyForExpression`:
   ```
   Does not descend into subquery plans (e.g. `ScalarSubquery`); a caller whose expression
   may contain a subquery must rewrite those plans separately.
   ```
   
   Since we want to use it only for procedure's arguments reducing, where ScalarSubquery is prohibited, it should work fine



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3894759830


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,105 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Inlining duplicates a definition at every reference, so a definition referenced more than once
+   * must be safe to duplicate (see `isSafeToDuplicate`); anything else throws, as there is no plan
+   * here to pre-evaluate it into. Run [[ComputeCurrentTime]], [[ReplaceCurrentLike]] and
+   * [[SpecialDatetimeValues]] first to fold the current date/time and catalog expressions to
+   * literals.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  private[sql] def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, checkDuplication = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to inline unconditionally.
+  private def inlineWith(expression: Expression, checkDuplication: Boolean): Expression = {
+    // Which definitions are safe to duplicate can only be decided outer-first, so it is collected
+    // in a separate pass before inlining bottom-up.
+    val safeIds = if (checkDuplication) {
+      safeToDuplicateIds(expression)
+    } else {
+      Set.empty[CommonExpressionId]
+    }
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        if (checkDuplication) {
+          // Nested `With` is already rewritten, so a ref left in `child` belongs to `defs` or to
+          // an enclosing `With`. Only the ids defined here are checked.
+          val refCounts = child.collect { case ref: CommonExpressionRef => ref.id }
+            .groupBy(identity)
+            .transform((_, refs) => refs.size)
+          defs.foreach { commonExprDef =>
+            if (refCounts.getOrElse(commonExprDef.id, 0) > 1 &&
+              !safeIds.contains(commonExprDef.id)) {
+              throw SparkException.internalError(
+                "Cannot inline a common expression definition that is referenced more than " +
+                  s"once and is not safe to duplicate: ${commonExprDef.child.sql}")
+            }
+          }
+        }
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }
+
+  /**
+   * Ids of the definitions that are safe to duplicate. Walks outer-first, so a definition that
+   * references an enclosing one is decided after it, and visits a definition's own nested `With`
+   * before the definition itself. Ids are globally unique, so one flat set is enough.
+   */
+  private def safeToDuplicateIds(expression: Expression): Set[CommonExpressionId] = {
+    val safeIds = mutable.Set.empty[CommonExpressionId]
+    def visit(e: Expression): Unit = if (e.containsPattern(WITH_EXPRESSION)) {
+      e match {
+        case With(child, defs) =>
+          defs.foreach { commonExprDef =>
+            visit(commonExprDef.child)
+            if (isSafeToDuplicate(commonExprDef.child, safeIds)) {
+              safeIds += commonExprDef.id
+            }
+          }
+          visit(child)
+        case other => other.children.foreach(visit)
+      }
+    }
+    visit(expression)
+    safeIds.toSet
+  }
+
+  /**
+   * Whether a copy of `e` at every reference always evaluates to the same value. `foldable` is not
+   * enough: `current_timestamp()` re-reads the clock, a TIME -> TIMESTAMP cast derives the current
+   * date, `aes_encrypt` is a foldable `StaticInvoke` with a random IV, and a Hive UDF is foldable
+   * on a user-declared determinism flag.
+   */
+  private def isSafeToDuplicate(
+      e: Expression,
+      safeIds: collection.Set[CommonExpressionId]): Boolean = {
+    // A whitelist: an unrecognized expression is rejected rather than assumed safe.
+    def isLiteralTree(e: Expression): Boolean = e match {
+      case _: Literal => true
+      // A ref is safe exactly when the definition it points to is.
+      case ref: CommonExpressionRef => safeIds.contains(ref.id)
+      // Any other leaf varies per row (attribute) or reads a clock.
+      case _: LeafExpression => false
+      // Arbitrary Java methods and UDFs can be foldable without being pure.
+      case _: NonSQLExpression | _: UserDefinedExpression => false
+      case _ => e.deterministic && e.children.forall(isLiteralTree)
+    }

Review Comment:
   Added branch
   ```scala
         // A TIME -> TIMESTAMP cast stabilized by ComputeCurrentTime is a pure builder StaticInvoke;
         // accept it here before the blanket NonSQLExpression rejection below.
         case _ if ComputeCurrentTime.isStabilizedTimeToTimestamp(e) => e.children.forall(isLiteralTree)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on PR #58119:
URL: https://github.com/apache/spark/pull/58119#issuecomment-5509359976

   > Just two more minor nits:
   > 
   > * DateTimeUtils.makeTimestampX reading a clock would slip past the literal-children gate. Worth either softening the doc wording or matching the specific builder types if strict scoping was the intent.
   > * Only the NTZ path is directly tested. The other three variants take the identical helper path (I verified each passes the gate), so coverage is logically equivalent, a one-line LTZ case would just make that a bit explicit.
   > 
   > Otherwise looks good, thank you @matvei-zamiatin-db for addressing all previous comments!
   
   Softened comment a and change func name to `isMakeTimestampBuilder`, also added test for `LTZ` case


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3862339924


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceExpressionsSuite.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Add, Coalesce, Literal, Nvl, RuntimeReplaceable}
+import org.apache.spark.sql.catalyst.plans.PlanTest
+
+class ReplaceExpressionsSuite extends PlanTest {

Review Comment:
   The two new expression-only suites should extend SparkFunSuite, not PlanTest.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3862542059


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,23 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
+        }
+    }
+  }

Review Comment:
   Fixed, added non-foldable definitions restriction



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3872892981


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -53,7 +53,7 @@ object ReplaceExpressions extends Rule[LogicalPlan] {
     case p => p.mapExpressions(replace)
   }
 
-  private def replace(e: Expression): Expression = e match {
+  def replace(e: Expression): Expression = e match {

Review Comment:
   Can this be private[sql]? We need to be careful with exposing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814832311


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,20 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   */
+  def applyForExpression(expression: Expression): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef => refToExpr(ref.id)

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3865181330


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,34 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`. Inlining duplicates a definition at
+   * every reference, so this rejects non-foldable definitions: inlining one referenced more than
+   * once would evaluate it more than once, breaking `With`'s evaluate-once contract.
+   *
+   * Does not descend into subquery plans (e.g. `ScalarSubquery`). A caller whose expression
+   * may contain a subquery must rewrite those plans separately.
+   */
+  def applyForExpression(expression: Expression): Expression =
+    inlineWith(expression, rejectNonFoldableDefs = true)
+
+  // The plan-level rewrite shares this to inline `With` in conditional branches, which may not be
+  // evaluated and so can't be pulled into a Project; it passes false to allow non-foldable defs.
+  private def inlineWith(expression: Expression, rejectNonFoldableDefs: Boolean): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {

Review Comment:
   An inner definition referencing a foldable outer definition is rejected before the outer reference is substituted: With(Literal(1))(o => With(o + 1)(i => i + i)).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3813412638


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,20 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   */
+  def applyForExpression(expression: Expression): Expression = {

Review Comment:
   No tests are added for the five new applyForExpression entry points (in finishAnalysis.scala, RewriteWithExpression.scala). The plan-level behavior is unchanged and is covered by existing tests, but the expression-level API itself has zero direct test coverage. A minimal unit test (e.g. calling ComputeCurrentTime.applyForExpression on a CurrentDate() leaf expression and asserting it becomes a Literal) would provide a regression anchor before the single-pass analyzer adopts these entry points.



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -112,106 +112,128 @@ object EvalInlineTables extends Rule[LogicalPlan] with CastSupport {
  */
 object ComputeCurrentTime extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = {
-    val instant = Instant.now()
-    val currentTimestampMicros = instantToMicros(instant)
-    val currentTime = Literal.create(currentTimestampMicros, TimestampType)
-    val currentTimeOfDayNanos = instantToNanosOfDay(instant, conf.sessionLocalTimeZone)
-    val timezone = Literal.create(conf.sessionLocalTimeZone, StringType)
+    val snapshot = new TimeSnapshot(Instant.now())
+    plan.transformDownWithSubqueriesAndPruning(transformCondition) {
+      case subQuery =>
+        subQuery.transformAllExpressionsWithPruning(transformCondition)(
+          expressionTransform(snapshot))
+    }
+  }
+
+  /** Rewrites the current date/time functions in a single expression tree. */
+  def applyForExpression(expression: Expression): Expression =
+    applyForExpression(expression, Instant.now())
+
+  /**
+   * Rewrites the current date/time functions in a single expression tree using `instant`. Callers
+   * reducing several expressions that must observe the same wall clock pass one shared instant.
+   */
+  def applyForExpression(expression: Expression, instant: Instant): Expression = {
+    val snapshot = new TimeSnapshot(instant)
+    expression.transformWithPruning(transformCondition)(expressionTransform(snapshot))

Review Comment:
   ComputeCurrentTime.applyForExpression, ReplaceCurrentLike.applyForExpression, and SpecialDatetimeValues.applyForExpression all use `expression.transformWithPruning(...)`, which does not traverse into ScalarSubquery plan nodes embedded within expressions.
   
   The plan-level apply() counterparts use transformDownWithSubqueriesAndPruning / transformAllExpressionsWithPruning, both of which do cross subquery boundaries. A caller that passes an expression containing a ScalarSubquery (e.g. WHERE current_date() < (SELECT MIN(dt) FROM t)) would silently leave current_date() inside the subquery unrewritten.
   
   There are no callers in this PR so this is not an active bug, but the contract of applyForExpression should document this limitation explicitly so future callers (the single-pass analyzer) can account for it.



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,20 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   */
+  def applyForExpression(expression: Expression): Expression = {
+    expression.transformUpWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+      case With(child, defs) =>
+        val refToExpr = defs.map(commonExprDef => commonExprDef.id -> commonExprDef.child).toMap
+        child.transformWithPruning(_.containsPattern(COMMON_EXPR_REF)) {
+          case ref: CommonExpressionRef => refToExpr(ref.id)

Review Comment:
   ```suggestion
             case ref: CommonExpressionRef if refToExpr.contains(ref.id) => refToExpr(ref.id)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814855087


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -112,106 +112,128 @@ object EvalInlineTables extends Rule[LogicalPlan] with CastSupport {
  */
 object ComputeCurrentTime extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = {
-    val instant = Instant.now()
-    val currentTimestampMicros = instantToMicros(instant)
-    val currentTime = Literal.create(currentTimestampMicros, TimestampType)
-    val currentTimeOfDayNanos = instantToNanosOfDay(instant, conf.sessionLocalTimeZone)
-    val timezone = Literal.create(conf.sessionLocalTimeZone, StringType)
+    val snapshot = new TimeSnapshot(Instant.now())
+    plan.transformDownWithSubqueriesAndPruning(transformCondition) {
+      case subQuery =>
+        subQuery.transformAllExpressionsWithPruning(transformCondition)(
+          expressionTransform(snapshot))
+    }
+  }
+
+  /** Rewrites the current date/time functions in a single expression tree. */
+  def applyForExpression(expression: Expression): Expression =
+    applyForExpression(expression, Instant.now())
+
+  /**
+   * Rewrites the current date/time functions in a single expression tree using `instant`. Callers
+   * reducing several expressions that must observe the same wall clock pass one shared instant.
+   */
+  def applyForExpression(expression: Expression, instant: Instant): Expression = {
+    val snapshot = new TimeSnapshot(instant)
+    expression.transformWithPruning(transformCondition)(expressionTransform(snapshot))

Review Comment:
   Added comment for every `applyForExpression`:
   ```
   Does not descend into subquery plans (e.g. `ScalarSubquery`); a caller whose expression
   may contain a subquery must rewrite those plans separately.
   ```
   
   Since we want to use it only for procedure's arguments reducing, where ScalarSubquery is prohibited, it should work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "matvei-zamiatin-db (via GitHub)" <gi...@apache.org>.
matvei-zamiatin-db commented on code in PR #58119:
URL: https://github.com/apache/spark/pull/58119#discussion_r3814841672


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -66,6 +66,20 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
     }
   }
 
+  /**
+   * Rewrites the `With` expressions in a single expression tree by inlining their common
+   * expressions. Uses `transformUp` to handle nested `With`.
+   */
+  def applyForExpression(expression: Expression): Expression = {

Review Comment:
   Added tests for every `applyForExpression`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58119:
URL: https://github.com/apache/spark/pull/58119#issuecomment-5512314630

   I will go ahead and merge, thank you @matvei-zamiatin-db.
   @cloud-fan if you have any more concerns, we can follow up!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58866][SQL] Add expression-level entry points to optimizer rules [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #58119: [SPARK-58866][SQL] Add expression-level entry points to optimizer rules
URL: https://github.com/apache/spark/pull/58119


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org