You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "vladimirg-db (via GitHub)" <gi...@apache.org> on 2024/04/24 19:34:03 UTC

[PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

vladimirg-db opened a new pull request, #46209:
URL: https://github.com/apache/spark/pull/46209

   ### What changes were proposed in this pull request?
   Mark `DescribeQueryCommand` and `ExplainCommand` as `SupervisingCommand` (they don't expose their wrapped nodes, but supervise them internally). Introduce a new Analyzer rule `MoveParameterizedQueriesDown `, which moves `ParameterizedQuery` inside `SupervisingCommand` for parameters to be resolved correctly.
   
   
   ### Why are the changes needed?
   Parameterized `EXPLAIN` and `DESCRIBE` queries:
   - `spark.sql("describe select ?", Array(1)).show();`
   - `spark.sql("explain select ?", Array(1)).show();`
   fail with
   `org.apache.spark.sql.catalyst.ExtendedAnalysisException: [UNBOUND_SQL_PARAMETER] Found the unbound parameter: _16. Please, fix `args` and provide a mapping of the parameter to either a SQL literal or collection constructor functions such as `map()`, `array()`, `struct()`. SQLSTATE: 42P02; line 1 pos 16; 'Project [unresolvedalias(posparameter(16))] +- OneRowRelation`
   
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, parameterized `EXPLAIN` and `DESCRIBE` should start working for users
   
   
   ### How was this patch tested?
   - Run `sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala`
   - Run `sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala`
   - New tests for `SQLQuerySuite`
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on PR #46209:
URL: https://github.com/apache/spark/pull/46209#issuecomment-2077882647

   @milastdbx @stefankandic @cloud-fan hi! This PR is ready for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on code in PR #46209:
URL: https://github.com/apache/spark/pull/46209#discussion_r1579933358


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala:
##########
@@ -63,3 +63,14 @@ trait AnalysisOnlyCommand extends Command {
   // on the `AnalysisContext`
   def markAsAnalyzed(analysisContext: AnalysisContext): LogicalPlan
 }
+
+/**
+ * A logical node that does not expose its sub-nodes as children, but rather supervises them
+ * in an implementation-defined manner.
+ */
+trait SupervisingCommand extends LeafCommand {
+  /**
+   * Transform its supervised plan using `transformer` and returns a copy of `SupervisingCommand`
+   */
+  def withTransformedSupervisedPlan(transformer: (LogicalPlan) => LogicalPlan): LogicalPlan

Review Comment:
   This can be rewritten as two methods:
   ```scala
   def supervisedPlan: LogicalPlan
   def withNewSupervisedPlan(newPlan: LogicalPlan): LogicalPlan
   ```
   But IMO the current impl is more concise



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on code in PR #46209:
URL: https://github.com/apache/spark/pull/46209#discussion_r1579933358


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala:
##########
@@ -63,3 +63,14 @@ trait AnalysisOnlyCommand extends Command {
   // on the `AnalysisContext`
   def markAsAnalyzed(analysisContext: AnalysisContext): LogicalPlan
 }
+
+/**
+ * A logical node that does not expose its sub-nodes as children, but rather supervises them
+ * in an implementation-defined manner.
+ */
+trait SupervisingCommand extends LeafCommand {
+  /**
+   * Transform its supervised plan using `transformer` and returns a copy of `SupervisingCommand`
+   */
+  def withTransformedSupervisedPlan(transformer: (LogicalPlan) => LogicalPlan): LogicalPlan

Review Comment:
   This can be rewritten as two methods:
   ```scala
   def supervisedPlan: LogicalPlan
   def withNewSupervisedPlan(new: LogicalPlan): LogicalPlan
   ```
   But IMO the current impl is more concise



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #46209: [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand
URL: https://github.com/apache/spark/pull/46209


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #46209:
URL: https://github.com/apache/spark/pull/46209#issuecomment-2082201660

   thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46209:
URL: https://github.com/apache/spark/pull/46209#discussion_r1580908359


##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4715,6 +4715,145 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     val df6 = df3.join(df2, col("df3.zaak_id") === col("df2.customer_id"), "outer")
     df5.crossJoin(df6)
   }
+
+  test("SPARK-47939: Describe should work with parameterized queries") {
+    checkAnswer(
+      spark.sql("describe select ?", Array(1)),
+      Array(
+        Row("1", "int", null)
+      )
+    )
+    checkAnswer(
+      spark.sql("describe select :first", Map("first" -> 1)),
+      Array(
+        Row("1", "int", null)
+      )
+    )
+
+    checkAnswer(
+      spark.sql("describe select * from values (?, ?) t(x, y)", Array(1, "a")),
+      Array(
+        Row("x", "int", null),
+        Row("y", "string", null)
+      )
+    )
+    checkAnswer(
+      spark.sql(
+        "describe select * from values (:first, :second) t(x, y)",
+        Map("first" -> 1, "second" -> "a")
+      ),
+      Array(
+        Row("x", "int", null),
+        Row("y", "string", null)
+      )
+    )
+  }
+
+  test("SPARK-47939: Explain should work with parameterized queries") {
+    def checkQueryPlan(df: DataFrame, plan: String): Unit = assert(
+      df.collect()
+        .map(_.getString(0))
+        .map(_.replaceAll("#[0-9]+", "#N"))
+        .sameElements(Array(plan))
+    )
+
+    checkQueryPlan(
+      spark.sql("explain select ?", Array(1)),
+      """== Physical Plan ==

Review Comment:
   nit: use `stripMargin`
   ```
   """
     |line1
     |line2
   """.stripMargin
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org