You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "srielau (via GitHub)" <gi...@apache.org> on 2025/12/18 16:08:49 UTC
[PR] [SPARK-54759][SQL][WIP] Cursor support [spark]
srielau opened a new pull request, #53530:
URL: https://github.com/apache/spark/pull/53530
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690577969
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:
##########
@@ -224,7 +305,18 @@ class SqlScriptingExecution(
handler.reset()
handlerFrame.executionPlan.enterScope()
case None =>
- throw e.asInstanceOf[Throwable]
+ // SQL Standard: Unhandled completion conditions (SQLSTATE class '02' - no data)
+ // continue execution after the statement that caused the condition.
+ // Unhandled exception conditions (all other classes) are resignaled (thrown).
+ // Note: SQLSTATE class '01' (warnings) are not currently raised by Spark.
+ val sqlState = e.getSqlState
+ if (sqlState != null && sqlState.startsWith("02")) {
+ // Completion condition (no data) - continue execution, don't throw
+ // This allows statements like FETCH to return "no data" without causing script failure
+ } else {
+ // Exception condition - resignal (throw)
+ throw e.asInstanceOf[Throwable]
+ }
Review Comment:
I would prefer to not have empty if body:
```
case None =>
val sqlState = e.getSqlState
val isCompletionCondition = sqlState != null && sqlState.startsWith("02")
if (!isCompletionCondition) {
throw e.asInstanceOf[Throwable]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #53530:
URL: https://github.com/apache/spark/pull/53530#issuecomment-3751513269
> should we add an overload of `def getCursorState` that accepts a `CursorReference`? This code snippet repeats 3 times.
The debate between sql scripts with goldenfiles and scala unittests is very much philosophical.
Personbally I find a sql.out file much more readable than a scala file.
More importantly though I have very much hostile to unit-tests which only tests certain phases (e.g. parsing).
They break due to design changes, and they do not test outcome. And outcomes are what matters.
Golden files, of course have teh downside risk, that verytime they are re-generated teh user has to actually look at the diff.
If there are strong feelings to convert the SQL scripts to Scala, I can certainly do that. Curiously, in another PR I was asked to do the inverse....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699436436
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
It is important to note that TABLEREF is a "type" and the object is still a variable (same namespace).
Cursors are in a separate namespace. So I'm not clear how we can share code here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2707190088
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
Also now share the resolution logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648786677
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * @param cursorName Name of the cursor to open
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursorName: String,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContextManager = SqlScriptingContextManager.get()
+ .getOrElse(throw new AnalysisException(
+ errorClass = "CURSOR_OUTSIDE_SCRIPT",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ val scriptingContext = scriptingContextManager.getContext
+ .asInstanceOf[org.apache.spark.sql.scripting.SqlScriptingExecutionContext]
+
+ // Parse and normalize cursor name for case sensitivity
+ val cursorNameParts = parseCursorName(cursorName)
+
+ // Find cursor in scope hierarchy
+ val cursorDef = scriptingContext.currentFrame.findCursorByNameParts(cursorNameParts)
+ .getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ // Validate cursor is not already open
+ if (cursorDef.isOpen) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_OPEN",
+ messageParameters = Map("cursorName" -> cursorName))
+ }
+
+ // Execute the query and collect results
+ val resultData = if (args.nonEmpty) {
+ // Parameterized cursor: re-parse query text with bound parameters
+ val (data, analyzedPlan) = executeParameterizedQuery(cursorDef.queryText, args)
+ // Update the cursor's query with the analyzed plan so FETCH can access the schema
+ cursorDef.query = analyzedPlan
+ data
+ } else {
+ // Non-parameterized cursor: execute the analyzed plan directly
+ val queryExecution = session.sessionState.executePlan(cursorDef.query)
+ queryExecution.executedPlan.executeCollect()
Review Comment:
scrollable cursor would be a future feature. It's a pretty rare bird though, I don't expect demand for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #53530:
URL: https://github.com/apache/spark/pull/53530#issuecomment-3814306201
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646483969
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,70 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR command.
+ * Note: The query child should not be fully analyzed/optimized at declaration time
+ * because we need to re-analyze it each time the cursor is opened to see fresh
+ * data.
+ */
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * @param cursorName Name of the cursor
+ * @param query The parsed logical plan of the cursor query
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ query: LogicalPlan,
+ queryText: String,
+ asensitive: Boolean = true) extends UnaryCommand {
+ override def child: LogicalPlan = query
+ override protected def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
+ copy(query = newChild)
+
+ // Mark that the child query should not be eagerly analyzed
+ override lazy val resolved: Boolean = cursorName.nonEmpty
+}
+
+/**
+ * The logical plan of the OPEN cursor command.
+ *
+ * @param cursorName Name of the cursor to open
+ * @param args Parameter expressions from USING clause
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursor(
+ cursorName: String,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends Command {
+ override def children: Seq[LogicalPlan] = Nil
+ override protected def withNewChildrenInternal(
+ newChildren: IndexedSeq[LogicalPlan]): LogicalPlan = this
+
+ override lazy val resolved: Boolean = args.forall(_.resolved)
+}
+
+/**
+ * The logical plan of the FETCH cursor command.
+ */
+case class FetchCursor(
+ cursorName: String,
+ targetVariables: Seq[Expression]) extends LeafCommand {
+ override lazy val resolved: Boolean = targetVariables.forall(_.resolved)
+}
+
+/**
+ * The logical plan of the CLOSE cursor command.
+ */
+case class CloseCursor(
+ cursorName: String) extends Command {
Review Comment:
```suggestion
cursorName: String) extends LeafCommand {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2685353576
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.classic.Dataset
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorDeclared, CursorOpened}
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * Transitions cursor from Declared to Opened state by analyzing the query and binding parameters.
+ * Does not execute the query or create result iterator - that happens on first FETCH.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursor: Expression,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContextManager = SqlScriptingContextManager.get()
+ .getOrElse(throw new AnalysisException(
+ errorClass = "CURSOR_OUTSIDE_SCRIPT",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ val scriptingContext = scriptingContextManager.getContext
+ .asInstanceOf[org.apache.spark.sql.scripting.SqlScriptingExecutionContext]
+
+ // Find cursor using normalized name from CursorReference
+ val cursorDefOpt: Option[org.apache.spark.sql.scripting.CursorDefinition] =
+ if (cursorRef.scopePath.nonEmpty) {
+ scriptingContext.currentFrame.findCursorInScope(
+ cursorRef.scopePath.head, cursorRef.normalizedName)
+ } else {
+ scriptingContext.currentFrame.findCursorByName(cursorRef.normalizedName)
+ }.asInstanceOf[Option[org.apache.spark.sql.scripting.CursorDefinition]]
+ val cursorDef = cursorDefOpt.getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ // Get current cursor state and validate it's in Declared state
+ val currentState = scriptingContext.currentFrame.getCursorState(
+ cursorRef.normalizedName,
+ cursorRef.scopePath).getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ val query = currentState match {
+ case CursorDeclared(q) => q
+ case _ =>
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_OPEN",
+ messageParameters = Map("cursorName" -> cursorRef.sql))
+ }
+
+ // Execute the query and get the analyzed plan
+ val analyzedQuery = if (args.nonEmpty) {
+ // Parameterized cursor: re-parse query text with bound parameters
Review Comment:
then shall we delay the parsing to OPEN CURSOR stage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2681810046
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule normalizes cursor names based on case sensitivity configuration and
+ * separates qualified cursor names (label.cursor) into scope path and cursor name.
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope path and cursor name
+ val (scopePath, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (nameParts.dropRight(1), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (Seq.empty[String], nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
Review Comment:
shall we check the scripting context and fail earlier if the cursor does not exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2681820841
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule normalizes cursor names based on case sensitivity configuration and
+ * separates qualified cursor names (label.cursor) into scope path and cursor name.
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope path and cursor name
+ val (scopePath, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (nameParts.dropRight(1), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (Seq.empty[String], nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
Review Comment:
maybe we can even look up the cursor state and put it in `CursorReference`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691248515
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorDeclared, CursorOpened}
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * Transitions cursor from Declared to Opened state by:
+ * 1. Parsing the cursor's SQL query text to a LogicalPlan
+ * 2. Binding parameters (if USING clause is provided)
+ * 3. Analyzing the query (semantic analysis, resolution, type checking)
+ *
+ * Does not execute the query or create result iterator - that happens on first FETCH.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursor: Expression,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.sql)
+
+ // Get cursor definition from CursorReference (looked up during analysis)
+ val cursorDef = cursorRef.definition
+
+ // Get current cursor state and validate it's in Declared state
+ val currentState = scriptingContext.currentFrame.getCursorState(
Review Comment:
should we add an overload of `def getCursorState` that accepts a `CursorReference`? This code snippet repeats 3 times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646486155
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
Review Comment:
but empty name is not no name, how do we handle positional parameter?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(cursorName, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
Review Comment:
ditto, why qualified?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646496576
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
+
+ /**
+ * Checks for duplicate variable names and throws an exception if found.
+ * Names are normalized when the variables are created.
+ * No need for case insensitive comparison here.
+ */
+ private def checkForDuplicateVariables(variables: Seq[VariableReference]): Unit = {
+ val dups = variables.groupBy(_.identifier).filter(kv => kv._2.length > 1)
+ if (dups.nonEmpty) {
+ throw new AnalysisException(
+ errorClass = "DUPLICATE_ASSIGNMENTS",
+ messageParameters = Map("nameList" ->
+ dups.keys.map(key => toSQLId(key.name())).mkString(", ")))
+ }
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ _.containsPattern(COMMAND), ruleId) {
+ // Resolve FetchCursor wrapped in SingleStatement
+ case s @ SingleStatement(fetchCursor: FetchCursor)
+ if !fetchCursor.targetVariables.forall(_.resolved) =>
+ val resolvedVars = fetchCursor.targetVariables.map {
+ case u: UnresolvedAttribute =>
+ variableResolution.lookupVariable(
+ nameParts = u.nameParts
+ ) match {
+ case Some(variable) => variable.copy(canFold = false)
+ case _ => throw unresolvedVariableError(u.nameParts, Seq("SYSTEM", "SESSION"))
+ }
+
+ case other => throw SparkException.internalError(
+ "Unexpected target variable expression in FetchCursor: " + other)
+ }
+
+ s.copy(parsedPlan = fetchCursor.copy(targetVariables = resolvedVars))
+
+ // Check for duplicates after resolution
+ case s @ SingleStatement(fetchCursor: FetchCursor)
+ if fetchCursor.targetVariables.forall(_.resolved) =>
+ val targetVariables = fetchCursor.targetVariables.map(_.asInstanceOf[VariableReference])
+ checkForDuplicateVariables(targetVariables)
+ s
+
+ // Also resolve unwrapped FetchCursor (when extracted from SingleStatement for execution)
Review Comment:
if it's unwrapped for execution, it must be resolved already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646496107
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
+
+ /**
+ * Checks for duplicate variable names and throws an exception if found.
+ * Names are normalized when the variables are created.
+ * No need for case insensitive comparison here.
+ */
+ private def checkForDuplicateVariables(variables: Seq[VariableReference]): Unit = {
+ val dups = variables.groupBy(_.identifier).filter(kv => kv._2.length > 1)
+ if (dups.nonEmpty) {
+ throw new AnalysisException(
+ errorClass = "DUPLICATE_ASSIGNMENTS",
+ messageParameters = Map("nameList" ->
+ dups.keys.map(key => toSQLId(key.name())).mkString(", ")))
+ }
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ _.containsPattern(COMMAND), ruleId) {
+ // Resolve FetchCursor wrapped in SingleStatement
+ case s @ SingleStatement(fetchCursor: FetchCursor)
+ if !fetchCursor.targetVariables.forall(_.resolved) =>
+ val resolvedVars = fetchCursor.targetVariables.map {
+ case u: UnresolvedAttribute =>
+ variableResolution.lookupVariable(
+ nameParts = u.nameParts
+ ) match {
+ case Some(variable) => variable.copy(canFold = false)
+ case _ => throw unresolvedVariableError(u.nameParts, Seq("SYSTEM", "SESSION"))
+ }
+
+ case other => throw SparkException.internalError(
+ "Unexpected target variable expression in FetchCursor: " + other)
+ }
+
+ s.copy(parsedPlan = fetchCursor.copy(targetVariables = resolvedVars))
+
+ // Check for duplicates after resolution
+ case s @ SingleStatement(fetchCursor: FetchCursor)
+ if fetchCursor.targetVariables.forall(_.resolved) =>
+ val targetVariables = fetchCursor.targetVariables.map(_.asInstanceOf[VariableReference])
+ checkForDuplicateVariables(targetVariables)
Review Comment:
can we do it right after we resolve the variables? why adding another case match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699574309
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/DeclareCursorExec.scala:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorDefinition}
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorDeclared
+
+/**
+ * Physical plan node for declaring cursors.
+ *
+ * Creates a cursor definition and initializes it in the Declared state.
+ * The cursor query is stored as SQL text and is not parsed or analyzed until OPEN time.
+ * This allows parameter markers to be preserved and bound correctly at OPEN.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText Original SQL text of the cursor query (with parameter markers preserved)
+ * @param asensitive Whether the cursor is ASENSITIVE (sensitivity to underlying data changes,
+ * not case sensitivity). Currently all cursors are effectively INSENSITIVE.
+ */
+case class DeclareCursorExec(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean) extends LeafV2CommandExec with DataTypeErrorsBase {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorName)
+ val currentScope = scriptingContext.currentScope
+
+ // Normalize cursor name based on case sensitivity configuration
+ val normalizedName = if (session.sessionState.conf.caseSensitiveAnalysis) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(java.util.Locale.ROOT)
+ }
+
+ // Validate cursor doesn't already exist in current scope
+ if (currentScope.cursors.contains(normalizedName)) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_EXISTS",
+ messageParameters = Map("cursorName" -> toSQLId(cursorName)))
+ }
+
+ // Create immutable cursor definition with name and SQL text
+ // Query parsing and analysis is deferred until OPEN time
+ val cursorDef = CursorDefinition(
+ name = cursorName,
Review Comment:
Aligned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2706101341
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
Fix teh name resolution
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
Fixed the name resolution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on PR #53530:
URL: https://github.com/apache/spark/pull/53530#issuecomment-3806316199
Change that should unblock failing tests after removing `returnCurrentWithoutAdvancing` is implemented in this PR https://github.com/apache/spark/pull/53894. After it is merged, please rebase this changes onto it.
cc @srielau @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #53530:
URL: https://github.com/apache/spark/pull/53530#issuecomment-3807696397
> Thanks for incorporating the change from #53894, but let's please merge that PR independently from this one. They clearly do 2 different things, and the "fix" PR is self contained and has its purpose. It will be easier to debug and understand changes later if we merge it one by one.
Yes, I'm awaiting this PR's merge. At that point the next merge to master shoudl cancel out teh changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648790054
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
Review Comment:
No reason.... fixed, switched to LeafCommand as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648810878
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE strictIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
Review Comment:
That's an oversight... fixing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2681805105
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CursorReference.scala:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.UnresolvedException
+import org.apache.spark.sql.catalyst.trees.TreePattern.{CURSOR_REFERENCE, TreePattern => TP, UNRESOLVED_CURSOR}
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unresolved reference to a cursor. This is used during parsing and will be resolved
+ * to a [[CursorReference]] by the [[ResolveCursors]] analyzer rule.
+ *
+ * @param nameParts The parts of the cursor name (e.g., Seq("cursor") or Seq("label", "cursor"))
+ */
+case class UnresolvedCursor(nameParts: Seq[String]) extends LeafExpression with Unevaluable {
+ override def dataType: DataType = throw new UnresolvedException("dataType")
+ override def nullable: Boolean = throw new UnresolvedException("nullable")
+ override lazy val resolved: Boolean = false
+ override def sql: String = nameParts.mkString(".")
+ override def toString: String = s"UnresolvedCursor(${nameParts.mkString(".")})"
+
+ final override val nodePatterns: Seq[TP] = Seq(UNRESOLVED_CURSOR)
+}
+
+/**
+ * A resolved reference to a cursor. This is created by the [[ResolveCursors]] analyzer rule
+ * after normalizing the cursor name parts and checking case sensitivity.
+ *
+ * @param nameParts The original cursor name parts (unnormalized)
+ * @param normalizedName The normalized cursor name (for lookups considering case sensitivity)
+ * @param scopePath The scope path (e.g., Seq("label") for qualified cursors, empty for unqualified)
+ */
+case class CursorReference(
+ nameParts: Seq[String],
+ normalizedName: String,
+ scopePath: Seq[String]) extends LeafExpression with Unevaluable {
Review Comment:
should it be `Option[String]` or we pick `Seq[String]` for the future? For now the qualifier can only be label name which only has one name part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690808231
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorDefinition, CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
Review Comment:
```suggestion
case uc: UnresolvedCursor =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2685372945
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CursorReference.scala:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.UnresolvedException
+import org.apache.spark.sql.catalyst.trees.TreePattern.{CURSOR_REFERENCE, TreePattern => TP, UNRESOLVED_CURSOR}
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unresolved reference to a cursor. This is used during parsing and will be resolved
+ * to a [[CursorReference]] by the [[ResolveCursors]] analyzer rule.
+ *
+ * @param nameParts The parts of the cursor name (e.g., Seq("cursor") or Seq("label", "cursor"))
+ */
+case class UnresolvedCursor(nameParts: Seq[String]) extends LeafExpression with Unevaluable {
+ override def dataType: DataType = throw new UnresolvedException("dataType")
+ override def nullable: Boolean = throw new UnresolvedException("nullable")
+ override lazy val resolved: Boolean = false
+ override def sql: String = nameParts.mkString(".")
+ override def toString: String = s"UnresolvedCursor(${nameParts.mkString(".")})"
+
+ final override val nodePatterns: Seq[TP] = Seq(UNRESOLVED_CURSOR)
+}
+
+/**
+ * A resolved reference to a cursor. This is created by the [[ResolveCursors]] analyzer rule
+ * after normalizing the cursor name parts, checking case sensitivity, and looking up the
+ * cursor definition from the scripting context.
+ *
+ * @param nameParts The original cursor name parts (unnormalized)
+ * @param normalizedName The normalized cursor name (for lookups considering case sensitivity)
+ * @param scopeLabel Optional label qualifier for scoped cursors (e.g., Some("label") for
+ * "label.cursor", None for unqualified cursors)
+ * @param definition The cursor definition looked up during analysis. Type is Any to avoid
+ * circular dependency (catalyst cannot import sql/core classes). At runtime,
+ * this will be cast to CursorDefinition.
+ */
+case class CursorReference(
+ nameParts: Seq[String],
+ normalizedName: String,
+ scopeLabel: Option[String],
+ definition: Any) extends LeafExpression with Unevaluable {
Review Comment:
shall we put `CursorDefinition` in this file? It's just two string fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2695006389
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE name=errorCapturingIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
+ ;
+
+openCursorStatement
+ : OPEN multipartIdentifier (USING (LEFT_PAREN params=namedExpressionSeq RIGHT_PAREN | params=namedExpressionSeq))?
+ ;
+
+fetchCursorStatement
+ : FETCH ((NEXT FROM) | FROM | NEXT)? cursorName=multipartIdentifier INTO targets=multipartIdentifierList
Review Comment:
Good catch! The spoec is right, the gramar is wrong! FETCH NEXT cur1 is not allowed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2694921040
##########
common/utils/src/main/resources/error/error-conditions.json:
##########
@@ -1037,6 +1037,48 @@
},
"sqlState" : "21S01"
},
+ "CURSOR_ALREADY_EXISTS" : {
Review Comment:
I'm not a huge fan of subclasses. One of my major beefs is that it collapses the SQLSTATEs and we end up with inappropriate states for part of thr subclasses.
CURSOR_ALREADY_EXISTS collapses into "object already exists" (42XXX), an analysis failure. While cursor state errors fall under 24XXX which is a SQL Standard state specifically for those, and it's runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #53530:
URL: https://github.com/apache/spark/pull/53530#issuecomment-3761345919
> > should we add an overload of `def getCursorState` that accepts a `CursorReference`? This code snippet repeats 3 times.
> > The debate between sql scripts with goldenfiles and scala unittests is very much philosophical.
> > Personbally I find a sql.out file much more readable than a scala file.
> > More importantly though I have very much hostile to unit-tests which only tests certain phases (e.g. parsing).
> > They break due to design changes, and they do not test outcome. And outcomes are what matters.
>
> Golden files, of course have teh downside risk, that verytime they are re-generated teh user has to actually look at the diff.
>
> If there are strong feelings to convert the SQL scripts to Scala, I can certainly do that. Curiously, in another PR I was asked to do the inverse....
Test suite has been turned into Scala.
Some further reflection: Turns out Cursor is terribly bad (or my prompts) about preventing drift when regenerating golden files.
So this exercise did in fact flush out bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2681831358
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule normalizes cursor names based on case sensitivity configuration and
+ * separates qualified cursor names (label.cursor) into scope path and cursor name.
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope path and cursor name
+ val (scopePath, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (nameParts.dropRight(1), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (Seq.empty[String], nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
Review Comment:
I don't have a strong preference, but my point is we should do it in a central place. Either we fully resolve the cursor during analysis, or we fully resolve cursor at runtime in the physical plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690812929
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorDefinition, CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope label and cursor name
+ val (scopeLabel, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
Review Comment:
should fail if the qualifier has more than one name parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690841208
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6581,6 +6583,105 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = if (ctx.INSENSITIVE() != null) false else true
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Create UnresolvedCursor from the cursor name
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
Review Comment:
shall we fail earlier if there are more than two name parts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2692034652
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -329,6 +335,34 @@ class CompoundBodyExec(
@tailrec
override def next(): CompoundStatementExec = {
+ // If we need to return current without advancing (after CONTINUE handler),
Review Comment:
@miland-db Actually that code tunred out to be crucial. Without it I get: "No more elements" internal error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690596895
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:
##########
@@ -112,6 +112,82 @@ class SqlScriptingExecution(
}
}
+ /**
+ * Helper method to set the returnCurrentWithoutAdvancing flag on CompoundBodyExec.
+ * After a CONTINUE handler, the next statement to execute is already in curr,
+ * so we shouldn't advance the iterator.
+ * @param executionPlan Execution plan.
+ */
+ private def setReturnCurrentWithoutAdvancing(executionPlan: NonLeafStatementExec): Unit = {
+ // We need to find the innermost CompoundBodyExec that's currently executing
+ // Navigate through the execution plan following curr pointers
+ var currExecPlan: NonLeafStatementExec = executionPlan
+
+ // Keep going deeper while we have non-leaf statements
+ var foundTarget = false
+ while (!foundTarget && currExecPlan.curr.isDefined) {
+ currExecPlan.curr.get match {
+ case compound: CompoundBodyExec =>
+ // Found a CompoundBodyExec - set the flag and try to go deeper
+ compound.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ currExecPlan = compound
+
+ case loop: RepeatStatementExec =>
+ // For REPEAT, navigate to its body
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true // Stop if no body
+ }
+
+ case loop: WhileStatementExec =>
+ // For WHILE, navigate to its body
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case loop: ForStatementExec =>
+ // For FOR, the body is in the statements list
+ // Set flag on all CompoundBodyExec children
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case ifElse: IfElseStatementExec =>
+ // For IF, navigate to current branch being executed
+ ifElse.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case other: NonLeafStatementExec =>
+ // Other non-leaf, try to go deeper
+ currExecPlan = other
+
+ case _ =>
+ // Leaf statement, stop here
+ foundTarget = true
+ }
+ }
+
+ // Also set on the top-level if it's a CompoundBodyExec
+ executionPlan match {
+ case compound: CompoundBodyExec =>
+ compound.returnCurrentWithoutAdvancing = true
+ case _ => // pass
+ }
+ }
+
Review Comment:
I am not sure we need this after this fix: https://github.com/apache/spark/pull/53759
Please merge latest changes and test cursors without this flag `returnCurrentWithoutAdvancing`
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:
##########
@@ -112,6 +112,82 @@ class SqlScriptingExecution(
}
}
+ /**
+ * Helper method to set the returnCurrentWithoutAdvancing flag on CompoundBodyExec.
+ * After a CONTINUE handler, the next statement to execute is already in curr,
+ * so we shouldn't advance the iterator.
+ * @param executionPlan Execution plan.
+ */
+ private def setReturnCurrentWithoutAdvancing(executionPlan: NonLeafStatementExec): Unit = {
+ // We need to find the innermost CompoundBodyExec that's currently executing
+ // Navigate through the execution plan following curr pointers
+ var currExecPlan: NonLeafStatementExec = executionPlan
+
+ // Keep going deeper while we have non-leaf statements
+ var foundTarget = false
+ while (!foundTarget && currExecPlan.curr.isDefined) {
+ currExecPlan.curr.get match {
+ case compound: CompoundBodyExec =>
+ // Found a CompoundBodyExec - set the flag and try to go deeper
+ compound.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ currExecPlan = compound
+
+ case loop: RepeatStatementExec =>
+ // For REPEAT, navigate to its body
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true // Stop if no body
+ }
+
+ case loop: WhileStatementExec =>
+ // For WHILE, navigate to its body
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case loop: ForStatementExec =>
+ // For FOR, the body is in the statements list
+ // Set flag on all CompoundBodyExec children
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case ifElse: IfElseStatementExec =>
+ // For IF, navigate to current branch being executed
+ ifElse.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case other: NonLeafStatementExec =>
+ // Other non-leaf, try to go deeper
+ currExecPlan = other
+
+ case _ =>
+ // Leaf statement, stop here
+ foundTarget = true
+ }
+ }
+
+ // Also set on the top-level if it's a CompoundBodyExec
+ executionPlan match {
+ case compound: CompoundBodyExec =>
+ compound.returnCurrentWithoutAdvancing = true
+ case _ => // pass
+ }
+ }
+
Review Comment:
I am not sure we need this code after this fix: https://github.com/apache/spark/pull/53759
Please merge latest changes and test cursors without this flag `returnCurrentWithoutAdvancing`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2737580445
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingLocalVariableManager.scala:
##########
@@ -76,39 +76,13 @@ class SqlScriptingLocalVariableManager(context: SqlScriptingExecutionContext)
throw SparkException.internalError("ScriptingVariableManager expects 1 or 2 nameParts.")
}
- // First search for variable in entire current frame.
- val resCurrentFrame = context.currentFrame.scopes
- .findLast(scope => isScopeOfVar(nameParts, scope))
- if (resCurrentFrame.isDefined) {
- return resCurrentFrame
- }
-
- // When searching in previous frames, for each frame we have to check only scopes before and
- // including the scope where the previously checked exception handler frame is defined.
- // Exception handler frames should not have access to variables from scopes
- // which are nested below the scope where the handler is defined.
- var previousFrameDefinitionLabel = context.currentFrame.scopeLabel
-
- // dropRight(1) removes the current frame, which we already checked above.
- context.frames.dropRight(1).reverseIterator.foreach(frame => {
- // Drop scopes until we encounter the scope in which the previously checked
- // frame was defined. If it was not defined in this scope candidateScopes will be
- // empty.
- val candidateScopes = frame.scopes.reverse.dropWhile(
- scope => !previousFrameDefinitionLabel.contains(scope.label))
-
- val scope = candidateScopes.findLast(scope => isScopeOfVar(nameParts, scope))
- if (scope.isDefined) {
- return scope
- }
- // If candidateScopes is nonEmpty that means that we found the previous frame definition
- // in this frame. If we still have not found the variable, we now have to find the definition
- // of this new frame, so we reassign the frame definition label to search for.
- if (candidateScopes.nonEmpty) {
- previousFrameDefinitionLabel = frame.scopeLabel
- }
- })
- None
+ // Use the shared searchAcrossFrames helper to maintain consistent logic with cursors
+ context.searchAcrossFrames(
Review Comment:
@dusantism-db can you please take a look at this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648786677
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * @param cursorName Name of the cursor to open
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursorName: String,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContextManager = SqlScriptingContextManager.get()
+ .getOrElse(throw new AnalysisException(
+ errorClass = "CURSOR_OUTSIDE_SCRIPT",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ val scriptingContext = scriptingContextManager.getContext
+ .asInstanceOf[org.apache.spark.sql.scripting.SqlScriptingExecutionContext]
+
+ // Parse and normalize cursor name for case sensitivity
+ val cursorNameParts = parseCursorName(cursorName)
+
+ // Find cursor in scope hierarchy
+ val cursorDef = scriptingContext.currentFrame.findCursorByNameParts(cursorNameParts)
+ .getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ // Validate cursor is not already open
+ if (cursorDef.isOpen) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_OPEN",
+ messageParameters = Map("cursorName" -> cursorName))
+ }
+
+ // Execute the query and collect results
+ val resultData = if (args.nonEmpty) {
+ // Parameterized cursor: re-parse query text with bound parameters
+ val (data, analyzedPlan) = executeParameterizedQuery(cursorDef.queryText, args)
+ // Update the cursor's query with the analyzed plan so FETCH can access the schema
+ cursorDef.query = analyzedPlan
+ data
+ } else {
+ // Non-parameterized cursor: execute the analyzed plan directly
+ val queryExecution = session.sessionState.executePlan(cursorDef.query)
+ queryExecution.executedPlan.executeCollect()
Review Comment:
scrollable cursor would be a future feature. It's a pretty rare bird though, I don't expect demand for it. Let me change the iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648790131
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
Review Comment:
Labels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660323790
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ params.namedExpression().asScala.toSeq.map(visitNamedExpression).map {
+ case alias: Alias => (alias.child, alias.name)
Review Comment:
shall we infer the alias from attribute? what's the behavior for EXECUTE IMMEDIATE?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ params.namedExpression().asScala.toSeq.map(visitNamedExpression).map {
+ case alias: Alias => (alias.child, alias.name)
Review Comment:
shall we infer the parameter name from attribute? what's the behavior for EXECUTE IMMEDIATE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660342145
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
Review Comment:
does this include the scripting local variables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2692437434
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -57,26 +57,86 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
/**
* Find a cursor by its normalized name in the current scope and parent scopes.
* Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches across all frames (script frame first, then handler frames) to support
+ * cursor access from exception handlers.
*/
override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
if (frames.isEmpty) {
None
} else {
- currentFrame.findCursorByName(normalizedName)
+ // Search in script frame first (frames.head), then current frame
+ // This allows handlers to access cursors declared in the main script
+ frames.head.findCursorByName(normalizedName)
+ .orElse(if (frames.size > 1) currentFrame.findCursorByName(normalizedName) else None)
Review Comment:
Is `normalizedName` a qualified cursor name? Asking because search logic is to first try to resolve in main script and only then in handler, so I am worried what happens when cursor is shadowed by creating another one in handler with the same name as one in the main script. Which one would be found in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2704078815
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
A namespace is a domain of (qualified) names within which objects must be unique.
For example: VIEW and TABLE sharte the same namespace. They are both relations.
scalar FUNCTION, table FUNCTION and PROCEDURE share the same namespace. They are all routines.
Things get more interesting with leaf nodes of expressions. These can be variables, columns/aliases, or parameters. Hence more complicated occlusion rules (columns in from, the lateral, within the statement, then local variables, then parameters, then session varicbles).
Cursors are yet another namespace, and condition yet another.
Key is that distinct namespaces do not collide by definition.
So you can have a cursor with the same name as a local variable defined in the same compound statement.
And you could declare a local table (if we had such a thing) with the same name as a variable.
Specifically for cursors: They can only be referenced in OPEN, CLOSE, FETCH. You cannot SET a cursor, or use it in an expression.
That's why when you pass a table to a polymorphic table function you have to say foo(TABLE t). The table keyword makes it clear that t is not a variable, or column, but a relation.
One of teh unsolved issues around "TABLEREF" is whether they can be used directly or also need a keyword.
SELECT * FROM t;
How do we know t is a table ref from the variable namspace and not a relation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660317974
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,55 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * The queryText is stored to support both parameterized and non-parameterized cursors.
+ * The query is parsed and analyzed when the cursor is declared at execution time.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean = true) extends LeafCommand
+
+/**
+ * The logical plan of the OPEN cursor command.
+ *
+ * @param nameParts Cursor name parts (unqualified: Seq(name) or qualified: Seq(label, name))
+ * @param args Parameter expressions from USING clause
+ * @param paramNames Names for each parameter (empty string "" for positional parameters)
+ */
+case class OpenCursor(
+ nameParts: Seq[String],
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafCommand {
+ override lazy val resolved: Boolean = args.forall(_.resolved)
Review Comment:
shall we add an assert that `args.length` must be equal to `paramNames.length`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690825281
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorDefinition, CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope label and cursor name
+ val (scopeLabel, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (Some(nameParts.head), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (None, nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
+ val normalizedName = if (caseSensitive) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(Locale.ROOT)
+ }
+
+ // Look up cursor definition from scripting context using reflection
+ // Reflection is needed because SqlScriptingExecutionContext is in sql/core
+ // which would create a circular dependency if imported into catalyst
Review Comment:
now it seems better to simply keep `Seq[String]` in the logical plan and resolve cursors at the runtime. The existing API design does not welcome analyzer to do too much script related resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2692443427
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -59,7 +59,7 @@ case class OpenCursorExec(
messageParameters = Map("cursorName" -> cursorRef.sql)))
currentState match {
- case CursorDeclared => // Expected state
+ case CursorDeclared | CursorClosed => // Expected states - new or closed cursor
Review Comment:
It is possible to reopen closed cursor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "davidm-db (via GitHub)" <gi...@apache.org>.
davidm-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2698216913
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
I don't think example like this would work for cursors:
<img width="587" height="520" alt="image" src="https://github.com/user-attachments/assets/a3a38780-8ed8-4f87-8383-45f1125eb70a" />
Specifically if `x2` was a cursor, it wouldn't get resolved because we are searching only the last and the first frame.
I know this is a very artificial example, especially for cursors I cannot imagine why someone would do it this way, but it is something we allow. But, my comment is more on a level that we already have a resolution logic implemented in the aforementioned places in code and I don't think we should duplicate it - highlighting again that we will/might have similar types in future that should also adhere to the same resolution logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690596895
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:
##########
@@ -112,6 +112,82 @@ class SqlScriptingExecution(
}
}
+ /**
+ * Helper method to set the returnCurrentWithoutAdvancing flag on CompoundBodyExec.
+ * After a CONTINUE handler, the next statement to execute is already in curr,
+ * so we shouldn't advance the iterator.
+ * @param executionPlan Execution plan.
+ */
+ private def setReturnCurrentWithoutAdvancing(executionPlan: NonLeafStatementExec): Unit = {
+ // We need to find the innermost CompoundBodyExec that's currently executing
+ // Navigate through the execution plan following curr pointers
+ var currExecPlan: NonLeafStatementExec = executionPlan
+
+ // Keep going deeper while we have non-leaf statements
+ var foundTarget = false
+ while (!foundTarget && currExecPlan.curr.isDefined) {
+ currExecPlan.curr.get match {
+ case compound: CompoundBodyExec =>
+ // Found a CompoundBodyExec - set the flag and try to go deeper
+ compound.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ currExecPlan = compound
+
+ case loop: RepeatStatementExec =>
+ // For REPEAT, navigate to its body
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true // Stop if no body
+ }
+
+ case loop: WhileStatementExec =>
+ // For WHILE, navigate to its body
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case loop: ForStatementExec =>
+ // For FOR, the body is in the statements list
+ // Set flag on all CompoundBodyExec children
+ loop.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case ifElse: IfElseStatementExec =>
+ // For IF, navigate to current branch being executed
+ ifElse.curr match {
+ case Some(body: CompoundBodyExec) =>
+ body.returnCurrentWithoutAdvancing = true
+ foundTarget = true
+ case _ => foundTarget = true
+ }
+
+ case other: NonLeafStatementExec =>
+ // Other non-leaf, try to go deeper
+ currExecPlan = other
+
+ case _ =>
+ // Leaf statement, stop here
+ foundTarget = true
+ }
+ }
+
+ // Also set on the top-level if it's a CompoundBodyExec
+ executionPlan match {
+ case compound: CompoundBodyExec =>
+ compound.returnCurrentWithoutAdvancing = true
+ case _ => // pass
+ }
+ }
+
Review Comment:
If I am not mistaken, we don't need this code after this fix: https://github.com/apache/spark/pull/53759?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690836816
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorDefinition, CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope label and cursor name
+ val (scopeLabel, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (Some(nameParts.head), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (None, nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
+ val normalizedName = if (caseSensitive) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(Locale.ROOT)
+ }
+
+ // Look up cursor definition from scripting context using reflection
+ // Reflection is needed because SqlScriptingExecutionContext is in sql/core
+ // which would create a circular dependency if imported into catalyst
Review Comment:
or we should improve the `SqlScriptingExecutionContextExtension` to provide cursor lookup APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691234388
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6581,6 +6583,120 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = if (ctx.INSENSITIVE() != null) false else true
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Helper method to create an UnresolvedCursor with validation.
+ * Validates that cursor references have at most one qualifier (label.cursor).
+ *
+ * @param nameParts The cursor name parts from multipartIdentifier
+ * @return UnresolvedCursor instance
+ * @throws ParseException if more than two name parts are provided
+ */
+ private def createUnresolvedCursor(nameParts: Seq[String]): UnresolvedCursor = {
+ if (nameParts.length > 2) {
+ throw QueryParsingErrors.cursorInvalidQualifierError(nameParts.mkString("."))
Review Comment:
I think there is a util function to generate qualified identifier string, like `toSQLId`, which handles escaping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2680309339
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -244,3 +299,36 @@ class SqlScriptingExecutionScope(
errorHandler
}
}
+
+/**
+ * Definition of a cursor in SQL scripting.
+ *
+ * @param name
+ * Name of the cursor.
+ * @param query
+ * The query that defines the cursor (LogicalPlan). For parameterized cursors,
+ * this is updated with the analyzed plan when the cursor is opened.
+ * @param queryText
+ * The original SQL text of the query (preserves parameter markers).
+ * @param isOpen
+ * Whether the cursor is currently open.
+ * @param resultIterator
+ * The iterator over result rows when cursor is open. Uses toLocalIterator()
+ * to avoid loading all data into memory at once.
+ */
+case class CursorDefinition(
+ name: String,
+ var query: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,
+ queryText: String,
+ var isOpen: Boolean = false,
+ var resultIterator: Option[java.util.Iterator[org.apache.spark.sql.Row]] = None) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646480765
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,70 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR command.
+ * Note: The query child should not be fully analyzed/optimized at declaration time
+ * because we need to re-analyze it each time the cursor is opened to see fresh
+ * data.
+ */
+/**
+ * The logical plan of the DECLARE CURSOR statement.
Review Comment:
Let's merge it with the classdoc above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646500400
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -313,6 +319,34 @@ class CompoundBodyExec(
@tailrec
override def next(): CompoundStatementExec = {
+ // If we need to return current without advancing (after CONTINUE handler),
Review Comment:
We need someone more familiar with the scripting engine to review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2692868801
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -57,26 +57,86 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
/**
* Find a cursor by its normalized name in the current scope and parent scopes.
* Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches across all frames (script frame first, then handler frames) to support
+ * cursor access from exception handlers.
*/
override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
if (frames.isEmpty) {
None
} else {
- currentFrame.findCursorByName(normalizedName)
+ // Search in script frame first (frames.head), then current frame
+ // This allows handlers to access cursors declared in the main script
+ frames.head.findCursorByName(normalizedName)
+ .orElse(if (frames.size > 1) currentFrame.findCursorByName(normalizedName) else None)
Review Comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690743854
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -329,6 +335,34 @@ class CompoundBodyExec(
@tailrec
override def next(): CompoundStatementExec = {
+ // If we need to return current without advancing (after CONTINUE handler),
Review Comment:
Addressed by removing the dead code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646481926
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,70 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR command.
+ * Note: The query child should not be fully analyzed/optimized at declaration time
+ * because we need to re-analyze it each time the cursor is opened to see fresh
+ * data.
+ */
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * @param cursorName Name of the cursor
+ * @param query The parsed logical plan of the cursor query
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ query: LogicalPlan,
+ queryText: String,
+ asensitive: Boolean = true) extends UnaryCommand {
+ override def child: LogicalPlan = query
+ override protected def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
+ copy(query = newChild)
+
+ // Mark that the child query should not be eagerly analyzed
+ override lazy val resolved: Boolean = cursorName.nonEmpty
Review Comment:
what does this mean? how can cursor name be empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646485672
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
Review Comment:
nit:
```
namedExprs.map {
case alias: Alias => (alias.child, alias.name)
case expr => (expr, "")
}.unzip2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "davidm-db (via GitHub)" <gi...@apache.org>.
davidm-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2703996580
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
I might be misunderstanding something here... What does the term "namespace" mean here?
Per your code, you want to be able to resolve cursors using labels - i.e. have the scoping mechanism work for cursors. To me this looked like CURSOR is a type - a very specific type though, similar to how TABLEREF would be.
Can you explain what the namespace means and how it affects cursors and makes them different compared to other types that could be used for variables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2733585289
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -314,6 +314,12 @@ class CompoundBodyExec(
/** Used to stop the iteration in cases when LEAVE statement is encountered. */
private var stopIteration = false
+ /**
+ * Flag to prevent advancing curr after an exception was thrown and handled by a CONTINUE handler.
+ * When true, the next call to next() should return curr without advancing.
+ */
+ private[scripting] var returnCurrentWithoutAdvancing = false
Review Comment:
Replaced with "official fix"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2737736920
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -115,6 +389,8 @@ class SqlScriptingExecutionFrame(
def enterScope(
label: String,
triggerToExceptionHandlerMap: TriggerToExceptionHandlerMap): Unit = {
+ // Note: Labels are already lowercased in ParserUtils#enterLabeledScope during parsing,
Review Comment:
nit: No need for `Note:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #53530: [SPARK-54759][SQL] SQL scripting Cursor support
URL: https://github.com/apache/spark/pull/53530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648790457
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(cursorName, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
Review Comment:
Ditto labels
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(cursorName, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+ val targetVariables = ctx.identifierReference().asScala.map { varIdent =>
+ val varName = if (varIdent.expression() != null) {
+ // IDENTIFIER(expression) case - not supported for variables
+ throw new ParseException(
+ errorClass = "INVALID_SQL_SYNTAX.IDENTIFIER_CLAUSE_NOT_ALLOWED",
+ messageParameters = Map.empty,
+ varIdent)
+ } else {
+ visitMultipartIdentifier(varIdent.multipartIdentifier())
+ }
+ UnresolvedAttribute(varName)
+ }.toSeq
+ SingleStatement(FetchCursor(cursorName, targetVariables))
+ }
+
+ /**
+ * Create a [[CloseCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * CLOSE cursor_name;
+ * }}}
+ */
+ override def visitCloseCursorStatement(
+ ctx: CloseCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
Review Comment:
labels :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648789276
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,70 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR command.
+ * Note: The query child should not be fully analyzed/optimized at declaration time
+ * because we need to re-analyze it each time the cursor is opened to see fresh
+ * data.
+ */
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * @param cursorName Name of the cursor
+ * @param query The parsed logical plan of the cursor query
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ query: LogicalPlan,
+ queryText: String,
+ asensitive: Boolean = true) extends UnaryCommand {
+ override def child: LogicalPlan = query
+ override protected def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
+ copy(query = newChild)
+
+ // Mark that the child query should not be eagerly analyzed
+ override lazy val resolved: Boolean = cursorName.nonEmpty
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648784311
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE strictIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
+ ;
+
+openCursorStatement
+ : OPEN multipartIdentifier (USING (LEFT_PAREN params=namedExpressionSeq RIGHT_PAREN | params=namedExpressionSeq))?
Review Comment:
Correct. It's labels. Actually in the standard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660357464
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -244,3 +299,36 @@ class SqlScriptingExecutionScope(
errorHandler
}
}
+
+/**
+ * Definition of a cursor in SQL scripting.
+ *
+ * @param name
+ * Name of the cursor.
+ * @param query
+ * The query that defines the cursor (LogicalPlan). For parameterized cursors,
+ * this is updated with the analyzed plan when the cursor is opened.
+ * @param queryText
+ * The original SQL text of the query (preserves parameter markers).
+ * @param isOpen
+ * Whether the cursor is currently open.
+ * @param resultIterator
+ * The iterator over result rows when cursor is open. Uses toLocalIterator()
+ * to avoid loading all data into memory at once.
+ */
+case class CursorDefinition(
+ name: String,
+ var query: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,
+ queryText: String,
+ var isOpen: Boolean = false,
+ var resultIterator: Option[java.util.Iterator[org.apache.spark.sql.Row]] = None) {
Review Comment:
it's hacky to manage cursor life cycle with 3 vars in a case class...
I think cursor definition should only contains the name and SQL text. We should have an explicit cursor state to manage the life cycle: 1) opened: the query is parsed with parameter substitution. 2) fetched: result iterator is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660343453
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4451,10 +4451,17 @@ object SQLConf {
val SQL_SCRIPTING_CONTINUE_HANDLER_ENABLED =
buildConf("spark.sql.scripting.continueHandlerEnabled")
.internal()
- .doc("EXPERIMENTAL FEATURE/WORK IN PROGRESS: SQL Scripting CONTINUE HANDLER feature " +
- "is under development and still not working as intended. This feature switch is intended " +
- "to be used internally for development and testing, not by end users. " +
- "YOU ARE ADVISED AGAINST USING THIS FEATURE AS ITS NOT FINISHED.")
+ .doc("SQL Scripting CONTINUE HANDLER feature enables CONTINUE exception handlers " +
+ "which are essential for cursor iteration with NOT FOUND conditions.")
+ .version("4.1.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ val SQL_SCRIPTING_CURSOR_ENABLED =
+ buildConf("spark.sql.scripting.cursorEnabled")
+ .internal()
+ .doc("SQL Scripting CURSOR feature enables declarative cursors with " +
+ "DECLARE CURSOR, OPEN, FETCH, and CLOSE statements.")
.version("4.1.0")
Review Comment:
```suggestion
.version("4.2.0")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660319532
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,55 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * The queryText is stored to support both parameterized and non-parameterized cursors.
+ * The query is parsed and analyzed when the cursor is declared at execution time.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean = true) extends LeafCommand
+
+/**
+ * The logical plan of the OPEN cursor command.
+ *
+ * @param nameParts Cursor name parts (unqualified: Seq(name) or qualified: Seq(label, name))
+ * @param args Parameter expressions from USING clause
+ * @param paramNames Names for each parameter (empty string "" for positional parameters)
+ */
+case class OpenCursor(
+ nameParts: Seq[String],
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafCommand {
+ override lazy val resolved: Boolean = args.forall(_.resolved)
+}
+
+/**
+ * The logical plan of the FETCH cursor command.
+ *
+ * @param nameParts Cursor name parts (unqualified: Seq(name) or qualified: Seq(label, name))
+ * @param targetVariables Target variables to fetch into
+ */
+case class FetchCursor(
+ nameParts: Seq[String],
+ targetVariables: Seq[Expression]) extends LeafCommand {
+ override lazy val resolved: Boolean = targetVariables.forall(_.resolved)
Review Comment:
I think it's effectively the same as the default implementation, we don't need this override. Same to `OpenCursor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690845359
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/DeclareCursorExec.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorDefinition}
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorDeclared
+
+/**
+ * Physical plan node for declaring cursors.
+ *
+ * Creates a cursor definition and initializes it in the Declared state.
+ * The cursor query is stored as SQL text and is not parsed or analyzed until OPEN time.
+ * This allows parameter markers to be preserved and bound correctly at OPEN.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText Original SQL text of the cursor query (with parameter markers preserved)
+ * @param asensitive Whether the cursor is ASENSITIVE (sensitivity to underlying data changes,
+ * not case sensitivity). Currently all cursors are effectively INSENSITIVE.
+ */
+case class DeclareCursorExec(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorName)
+ val currentScope = scriptingContext.currentScope
+
+ // Normalize cursor name based on case sensitivity configuration
Review Comment:
isn't it already done by the analyzer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690990893
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorDefinition, CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope label and cursor name
+ val (scopeLabel, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (Some(nameParts.head), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (None, nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
+ val normalizedName = if (caseSensitive) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(Locale.ROOT)
+ }
+
+ // Look up cursor definition from scripting context using reflection
+ // Reflection is needed because SqlScriptingExecutionContext is in sql/core
+ // which would create a circular dependency if imported into catalyst
Review Comment:
Improving SqlScriptingExecutionContextExtension
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "davidm-db (via GitHub)" <gi...@apache.org>.
davidm-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2693848671
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE name=errorCapturingIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
Review Comment:
are we using `errorCapturingIdentifier` here and `multipartIdentifier` in other statements to forbid declarations with dots, or is there any additional meaning to it?
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE name=errorCapturingIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
+ ;
+
+openCursorStatement
+ : OPEN multipartIdentifier (USING (LEFT_PAREN params=namedExpressionSeq RIGHT_PAREN | params=namedExpressionSeq))?
+ ;
+
+fetchCursorStatement
+ : FETCH ((NEXT FROM) | FROM | NEXT)? cursorName=multipartIdentifier INTO targets=multipartIdentifierList
Review Comment:
per ref spec it's `FETCH [ [ NEXT ] FROM ]` which maps to either:
- `FETCH FROM` or
- `FETCH NEXT FROM`
here we have three options, so just double checking which one is expected?
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
Do we understand the limitations of the current cursor resolution logic? The proper variable resolution logic is implemented in:
- `VariableResolution#lookupVariable` and
- `SqlScriptingLocalVariableManaged#get`
and is lot more complex than this.
I don't have an issue with going like this initially, but I think we need to know how we will address the issue with resolution. I would expect cursors to behave as any other variable, which at the moment isn't the case.
This is especially important if/when we go down the way of adding similar constructs (like TableRef variables) - all of them would need to behave like variables, but they are not regular variables.
If we do special casing like this for each of them, that would be bad. I think we need to create a mechanism to consider them as regular variables - i.e. special cases/classes that extend `VariableDefinition` maybe? This way we would have a unified mechanism for adding new types of var defs and it would fit into the variable resolution logic that exists today (hopefully with a very little amount of case matching).
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.catalog.VariableDefinition
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorReference, Expression, Literal, VariableReference}
+import org.apache.spark.sql.classic.Dataset
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorFetching, CursorOpened}
+
+/**
+ * Physical plan node for fetching from cursors.
+ *
+ * Transitions cursor from Opened to Fetching state on first fetch (creating result iterator),
+ * then fetches rows from the iterator on subsequent calls. Assigns fetched values to target
+ * variables with ANSI store assignment rules.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param targetVariables Variables to fetch into
+ */
+case class FetchCursorExec(
+ cursor: Expression,
+ targetVariables: Seq[VariableReference]) extends LeafV2CommandExec with DataTypeErrorsBase {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.definition.name)
+ val variableManager = SqlScriptingContextManager.get().get.getVariableManager
+
+ // Get current cursor state
+ val currentState = scriptingContext.getCursorState(cursorRef).getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name))))
+
+ // Get or create iterator based on current state
+ val (iterator, analyzedQuery) = currentState match {
+ case CursorOpened(query) =>
+ // First fetch - create iterator and transition to Fetching state
+ val df = Dataset.ofRows(
+ session.asInstanceOf[org.apache.spark.sql.classic.SparkSession],
+ query)
+ val iter = df.toLocalIterator()
+ scriptingContext.updateCursorState(
+ cursorRef.normalizedName,
+ cursorRef.scopeLabel,
+ CursorFetching(query, iter))
+ (iter, query)
+
+ case CursorFetching(query, iter) =>
+ // Subsequent fetch - use existing iterator
+ (iter, query)
+
+ case _ =>
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_OPEN",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name)))
+ }
+
+ // Get next row from iterator
+ if (!iterator.hasNext) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_NO_MORE_ROWS",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name)))
+ }
+
+ val externalRow = iterator.next()
+
+ // Convert Row to InternalRow for processing
+ val schema = org.apache.spark.sql.catalyst.types.DataTypeUtils
+ .fromAttributes(analyzedQuery.output)
+ val converter = org.apache.spark.sql.catalyst.CatalystTypeConverters
+ .createToCatalystConverter(schema)
+ val currentRow = converter(externalRow).asInstanceOf[InternalRow]
Review Comment:
I spent quite some time to figure this out and I think we might be (not sure how much?) inefficient here (similar holds for the `ForStatementExec` but there it is exposed in a bit different manner). cc @cloud-fan to help here, I'm not sure if I'm correct.
TLDR: by calling `toLocalIterator` we are basically converting from `InternalRow` to `Row`, just to take it then and convert back to `InternalRow`:
<img width="989" height="292" alt="image" src="https://github.com/user-attachments/assets/5fdf8b52-3839-4697-a3b3-2578d95e2c80" />
And it looks like the `executeToIterator` is actually behind the implementation of `toLocalIterator`, so we do still preserve the partition-by-partition collection of results:
<img width="849" height="421" alt="image" src="https://github.com/user-attachments/assets/f78540c1-6d35-4aae-97a3-b57c5e144f7f" />
Similar thing happens in `ForStatementExec`, but there the we use the "public" values from `Row`, just to get them back to the `InternalRow` format through the `SetVariable` statement. There, we can avoid creating `SetVariable` statements in a similar manner we do here, by directly updating the variable values.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Parser already validates this, so we can assert
+ assert(nameParts.length <= 2,
+ s"Cursor reference has too many parts: ${nameParts.mkString(".")}")
+
+ // Split qualified name into scope label and cursor name
+ val (scopeLabel, cursorName) = if (nameParts.length == 2) {
+ // Qualified cursor: "label.cursor"
+ (Some(nameParts.head), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (None, nameParts.head)
+ }
+
+ // Normalize cursor name and scope label based on case sensitivity
+ val normalizedName = if (caseSensitive) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(Locale.ROOT)
+ }
+
+ val normalizedScopeLabel = scopeLabel.map { label =>
+ if (caseSensitive) {
+ label
+ } else {
+ label.toLowerCase(Locale.ROOT)
+ }
+ }
+
+ // Look up cursor definition from scripting context using the extension API
+ val contextOpt = SqlScriptingContextManager.get().map(_.getContext)
+
+ val cursorDefOpt = contextOpt.flatMap { context =>
+ // Use the SqlScriptingExecutionContextExtension API for cursor lookup
+ normalizedScopeLabel match {
+ case Some(label) =>
+ // Qualified cursor: look up in specific labeled scope
+ context.findCursorInScope(label, normalizedName)
+ case None =>
+ // Unqualified cursor: search current and parent scopes
+ context.findCursorByName(normalizedName)
+ }
+ }
+
+ // If cursor not found and we're in a scripting context, fail immediately
+ if (contextOpt.isDefined && cursorDefOpt.isEmpty) {
Review Comment:
nit: why do we need the `contextOpt.isDefined` when we allow cursors only in scripts? even if we allowed outside of scripts, the question would still be the same.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ * Variables can be either scripting local variables or session variables.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ // VariableResolution looks up both scripting local variables (via SqlScriptingContextManager)
+ // and session variables (via tempVariableManager), checking local variables first.
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
+
+ /**
+ * Checks for duplicate variable names and throws an exception if found.
+ * Names are normalized when the variables are created.
+ * No need for case insensitive comparison here.
+ */
+ private def checkForDuplicateVariables(variables: Seq[VariableReference]): Unit = {
+ val dups = variables.groupBy(_.identifier).filter(kv => kv._2.length > 1)
+ if (dups.nonEmpty) {
+ throw new AnalysisException(
+ errorClass = "DUPLICATE_ASSIGNMENTS",
+ messageParameters = Map("nameList" ->
+ dups.keys.map(key => toSQLId(key.name())).mkString(", ")))
+ }
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ _.containsPattern(COMMAND), ruleId) {
+ // Resolve FetchCursor wrapped in SingleStatement
+ case s @ SingleStatement(fetchCursor: FetchCursor)
+ if !fetchCursor.targetVariables.forall(_.resolved) =>
+ val resolvedVars = fetchCursor.targetVariables.map {
+ case u: UnresolvedAttribute =>
+ variableResolution.lookupVariable(
+ nameParts = u.nameParts
+ ) match {
+ case Some(variable) => variable.copy(canFold = false)
+ case _ => throw unresolvedVariableError(u.nameParts, Seq("SYSTEM", "SESSION"))
+ }
+
+ case other => throw SparkException.internalError(
+ "Unexpected target variable expression in FetchCursor: " + other)
+ }
+
+ // Check for duplicates immediately after resolution
+ checkForDuplicateVariables(resolvedVars)
Review Comment:
exactly the same as in the `case` below - let's at least create a helper function
##########
common/utils/src/main/resources/error/error-conditions.json:
##########
@@ -1037,6 +1037,48 @@
},
"sqlState" : "21S01"
},
+ "CURSOR_ALREADY_EXISTS" : {
Review Comment:
Does it make sense to create a main "CURSOR" class and subclasses for all exceptions?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorReference, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorClosed, CursorDeclared, CursorOpened}
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * Transitions cursor from Declared to Opened state by:
+ * 1. Parsing the cursor's SQL query text to a LogicalPlan
+ * 2. Binding parameters (if USING clause is provided)
+ * 3. Analyzing the query (semantic analysis, resolution, type checking)
+ *
+ * Does not execute the query or create result iterator - that happens on first FETCH.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
Review Comment:
Similar to my earlier comment about "SELECT INTO" semantics, `OpenCursorExec` does have a lot of similarities/overlapping with `ExecuteImmediate`:
- `USING` clause
- how nested queries are executed
I would like to see that unified into a trait or something similar. I think there's a huge value to it, because these specific parts of the code (related to variables, parameters, etc) have evolved a lot in the past period and it's easier to refactor now rather than later, especially considering it is obvious even now that it should happen.
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -314,6 +314,12 @@ class CompoundBodyExec(
/** Used to stop the iteration in cases when LEAVE statement is encountered. */
private var stopIteration = false
+ /**
+ * Flag to prevent advancing curr after an exception was thrown and handled by a CONTINUE handler.
+ * When true, the next call to next() should return curr without advancing.
+ */
+ private[scripting] var returnCurrentWithoutAdvancing = false
Review Comment:
I think I wouldn't be comfortable with the change about `returnCurrentWithoutAdvancing` going in and wouldn't like the PR to be merged before this is properly understood and resolved.
The comments are a bit contradictory - for example, bolded parts here: "If we need to return current **without advancing** (after CONTINUE handler), do so, **advance curr**, and clear the flag".
The semantic of the change, from what I'm getting (but I don't think the logic is commented out well), is that we don't want the `curr` to advance after we exit the `CONTINUE HANDLER`, but we do it ourselves. The existing logic should also behave the expected way.
If we have some bug, let's try to figure it out and address it properly. This seems like a LLM special casing something, but it's not clearly explained why and I don't understand it from looking at the code.
Do we have an example of the query that fails without this change? Can you share it so we can look into it and figure out if there is any bug? Honestly the comments about this change sound like the behavior that should already exist, so I wouldn't expect we need this much special casing.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/DeclareCursorExec.scala:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorDefinition}
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorDeclared
+
+/**
+ * Physical plan node for declaring cursors.
+ *
+ * Creates a cursor definition and initializes it in the Declared state.
+ * The cursor query is stored as SQL text and is not parsed or analyzed until OPEN time.
+ * This allows parameter markers to be preserved and bound correctly at OPEN.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText Original SQL text of the cursor query (with parameter markers preserved)
+ * @param asensitive Whether the cursor is ASENSITIVE (sensitivity to underlying data changes,
+ * not case sensitivity). Currently all cursors are effectively INSENSITIVE.
+ */
+case class DeclareCursorExec(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean) extends LeafV2CommandExec with DataTypeErrorsBase {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorName)
+ val currentScope = scriptingContext.currentScope
+
+ // Normalize cursor name based on case sensitivity configuration
+ val normalizedName = if (session.sessionState.conf.caseSensitiveAnalysis) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(java.util.Locale.ROOT)
+ }
+
+ // Validate cursor doesn't already exist in current scope
+ if (currentScope.cursors.contains(normalizedName)) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_EXISTS",
+ messageParameters = Map("cursorName" -> toSQLId(cursorName)))
+ }
+
+ // Create immutable cursor definition with name and SQL text
+ // Query parsing and analysis is deferred until OPEN time
+ val cursorDef = CursorDefinition(
+ name = cursorName,
Review Comment:
I don't quite get the reason why we are mixing usage of `normalizedName` and `cursorName`? For example, IIRC variable definitions always use `normalizedName`. Why do we do it differently here?
I see that @cloud-fan had some similar comments, but also don't understand how they were addressed.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6581,6 +6583,120 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = if (ctx.INSENSITIVE() != null) false else true
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
Review Comment:
why do we wrap `DeclareCursor` into `SingleStatement` here? It should already be handled by `visitCompoundBodyImp` - I think this way we might end up with
```
SingleStatement(SingleStatement(DeclareCursor, ...), ...)
```
aside from it being unnecessary, not sure if there might be any downsides to it from the top of my mind.
cc @miland-db to double check
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorReference, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorClosed, CursorDeclared, CursorOpened}
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * Transitions cursor from Declared to Opened state by:
+ * 1. Parsing the cursor's SQL query text to a LogicalPlan
+ * 2. Binding parameters (if USING clause is provided)
+ * 3. Analyzing the query (semantic analysis, resolution, type checking)
+ *
+ * Does not execute the query or create result iterator - that happens on first FETCH.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursor: Expression,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec with DataTypeErrorsBase {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.definition.name)
+
+ // Get cursor definition from CursorReference (looked up during analysis)
+ val cursorDef = cursorRef.definition
+
+ // Get current cursor state and validate it's in Declared state
+ val currentState = scriptingContext.getCursorState(cursorRef).getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name))))
+
+ currentState match {
+ case CursorDeclared | CursorClosed => // Expected states - new or closed cursor
+ case _ =>
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_OPEN",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name)))
+ }
+
+ // Parse and analyze the query from the stored SQL text
+ // For both parameterized and non-parameterized cursors, we parse at OPEN time
+ val analyzedQuery = if (args.nonEmpty) {
+ // Parameterized cursor: parse with bound parameters
+ executeParameterizedQuery(cursorDef.queryText, args)
+ } else {
+ // Non-parameterized cursor: parse without parameters
+ val df = session.asInstanceOf[org.apache.spark.sql.classic.SparkSession]
+ .sql(cursorDef.queryText)
+ df.queryExecution.analyzed
+ }
+
+ // Transition cursor state to Opened
+ scriptingContext.updateCursorState(
+ cursorRef.normalizedName,
+ cursorRef.scopeLabel,
+ CursorOpened(analyzedQuery))
+
+ Nil
+ }
+
+ /**
+ * Executes a parameterized query by parsing the SQL text with bound parameters.
+ * This uses the same parameter binding mechanism as EXECUTE IMMEDIATE.
+ *
+ * @param queryText The SQL query text with parameter markers (? or :name)
+ * @param args Parameter expressions to bind
+ * @return The analyzed logical plan with parameters bound
+ */
+ private def executeParameterizedQuery(
+ queryText: String,
+ args: Seq[Expression]): LogicalPlan = {
+ val (paramValues, paramNames) = buildUnifiedParameters(args)
+
+ // Use session.sql() with parameters (same as EXECUTE IMMEDIATE)
+ val df = session.asInstanceOf[org.apache.spark.sql.classic.SparkSession]
+ .sql(queryText, paramValues, paramNames)
+
+ df.queryExecution.analyzed
+ }
+
+ /**
+ * Builds parameter arrays for the session.sql() API.
+ * This mirrors the exact logic in EXECUTE IMMEDIATE to ensure identical behavior.
Review Comment:
I think this example can be used to reflect on my previous thoughts. We have exactly the same ref spec for `EXECUTE IMMEDIATE` and `OPEN CURSOR`, with respect to `USING` clause. But our code is different. For cursors, we do `Alias` logic in `AstBuilder` and for execute immediate we do it during analysis (in `ResolveExecuteImmediate`). While this is a simple example, it paints the picture how this code can easily diverge even though the logic is completely the same (if I'm not misunderstanding anything).
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -115,7 +295,13 @@ class SqlScriptingExecutionFrame(
def enterScope(
label: String,
triggerToExceptionHandlerMap: TriggerToExceptionHandlerMap): Unit = {
- scopes.append(new SqlScriptingExecutionScope(label, triggerToExceptionHandlerMap))
+ // Normalize label for case-insensitive lookups
+ val normalizedLabel = if (SQLConf.get.caseSensitiveAnalysis) {
+ label
+ } else {
+ label.toLowerCase(Locale.ROOT)
+ }
Review Comment:
this is not needed, labels are always lowercased in `ParserUtils#enterLabeledScope`. Not sure why we did it this way, maybe we need to fix it to account for `SQLConf.get.caseSensitiveAnalysis`.
Anyways, cc @miland-db to double check this, I think you added the labels logic so you might recall why exactly we did it this way and whether it makes sense to take `SQLConf.get.caseSensitiveAnalysis` into account.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6581,6 +6583,120 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = if (ctx.INSENSITIVE() != null) false else true
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
Review Comment:
same holds for other `visit*CursorStatement` functions
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ * Variables can be either scripting local variables or session variables.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ // VariableResolution looks up both scripting local variables (via SqlScriptingContextManager)
+ // and session variables (via tempVariableManager), checking local variables first.
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
+
+ /**
+ * Checks for duplicate variable names and throws an exception if found.
+ * Names are normalized when the variables are created.
+ * No need for case insensitive comparison here.
+ */
+ private def checkForDuplicateVariables(variables: Seq[VariableReference]): Unit = {
Review Comment:
we are starting to duplicate a lot of code related to variables, and omit it where it makes sense.
for example, this is the same as in `ResolveSetVariable` but we don't have the same check in `ExecuteImmediate` (or I might be missing it). anyways, all of those statements follow the "SELECT INTO" semantics, and I think they should behave consistently.
maybe we should create a common trait for things related to "SELECT INTO" semantics?
we don't have to do it in this PR though, but if we don't do it then one of the outputs needs to be at least a work item to track it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2686814184
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.classic.Dataset
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorDeclared, CursorOpened}
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * Transitions cursor from Declared to Opened state by analyzing the query and binding parameters.
+ * Does not execute the query or create result iterator - that happens on first FETCH.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursor: Expression,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContextManager = SqlScriptingContextManager.get()
+ .getOrElse(throw new AnalysisException(
+ errorClass = "CURSOR_OUTSIDE_SCRIPT",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ val scriptingContext = scriptingContextManager.getContext
+ .asInstanceOf[org.apache.spark.sql.scripting.SqlScriptingExecutionContext]
+
+ // Find cursor using normalized name from CursorReference
+ val cursorDefOpt: Option[org.apache.spark.sql.scripting.CursorDefinition] =
+ if (cursorRef.scopePath.nonEmpty) {
+ scriptingContext.currentFrame.findCursorInScope(
+ cursorRef.scopePath.head, cursorRef.normalizedName)
+ } else {
+ scriptingContext.currentFrame.findCursorByName(cursorRef.normalizedName)
+ }.asInstanceOf[Option[org.apache.spark.sql.scripting.CursorDefinition]]
+ val cursorDef = cursorDefOpt.getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ // Get current cursor state and validate it's in Declared state
+ val currentState = scriptingContext.currentFrame.getCursorState(
+ cursorRef.normalizedName,
+ cursorRef.scopePath).getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ val query = currentState match {
+ case CursorDeclared(q) => q
+ case _ =>
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_OPEN",
+ messageParameters = Map("cursorName" -> cursorRef.sql))
+ }
+
+ // Execute the query and get the analyzed plan
+ val analyzedQuery = if (args.nonEmpty) {
+ // Parameterized cursor: re-parse query text with bound parameters
Review Comment:
Well, not really parsing as such. as ANTLR need to drill past it. But we can avoid building the logical plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660337390
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ params.namedExpression().asScala.toSeq.map(visitNamedExpression).map {
+ case alias: Alias => (alias.child, alias.name)
+ case expr => (expr, "") // Empty string for positional parameter
+ }.unzip
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(nameParts, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ val targetVariables = ctx.identifierReference().asScala.map { varIdent =>
+ val varName = if (varIdent.expression() != null) {
+ // IDENTIFIER(expression) case - not supported for variables
Review Comment:
or we want to provide a better error message here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646477405
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE strictIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
Review Comment:
not a big deal but why reserved keywords are not allowed here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690632340
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -244,3 +336,51 @@ class SqlScriptingExecutionScope(
errorHandler
}
}
+
+/**
+ * Defines a cursor with its name and SQL query text.
+ * This is immutable - lifecycle state is managed separately via CursorState.
+ *
+ * @param name
+ * Name of the cursor.
+ * @param queryText
Review Comment:
All other classes in this file are general sql scripting execution concepts and are not related specific statements. This `CursorState` is there to track the cursor state and is not even used in this file. Can we move this to another file so we don't pollute the code too much, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690730059
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -244,3 +336,51 @@ class SqlScriptingExecutionScope(
errorHandler
}
}
+
+/**
+ * Defines a cursor with its name and SQL query text.
+ * This is immutable - lifecycle state is managed separately via CursorState.
+ *
+ * @param name
+ * Name of the cursor.
+ * @param queryText
Review Comment:
Moved to CursorState.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699605211
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorReference, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorClosed, CursorDeclared, CursorOpened}
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * Transitions cursor from Declared to Opened state by:
+ * 1. Parsing the cursor's SQL query text to a LogicalPlan
+ * 2. Binding parameters (if USING clause is provided)
+ * 3. Analyzing the query (semantic analysis, resolution, type checking)
+ *
+ * Does not execute the query or create result iterator - that happens on first FETCH.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
Review Comment:
Cleaned up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660347156
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+
+/**
+ * Physical plan node for closing cursors.
+ * Releases cursor resources and resets its state to closed.
+ *
+ * @param nameParts Cursor name parts (unqualified: Seq(name) or qualified: Seq(label, name))
+ */
+case class CloseCursorExec(nameParts: Seq[String]) extends LeafV2CommandExec {
Review Comment:
shall we add a new Expression for cursor? I think it's better to resolve it during analysis, instead of at execution time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646486865
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(cursorName, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+ val targetVariables = ctx.identifierReference().asScala.map { varIdent =>
+ val varName = if (varIdent.expression() != null) {
+ // IDENTIFIER(expression) case - not supported for variables
+ throw new ParseException(
+ errorClass = "INVALID_SQL_SYNTAX.IDENTIFIER_CLAUSE_NOT_ALLOWED",
+ messageParameters = Map.empty,
+ varIdent)
+ } else {
+ visitMultipartIdentifier(varIdent.multipartIdentifier())
+ }
+ UnresolvedAttribute(varName)
+ }.toSeq
+ SingleStatement(FetchCursor(cursorName, targetVariables))
+ }
+
+ /**
+ * Create a [[CloseCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * CLOSE cursor_name;
+ * }}}
+ */
+ override def visitCloseCursorStatement(
+ ctx: CloseCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648060761
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE strictIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
+ ;
+
+openCursorStatement
+ : OPEN multipartIdentifier (USING (LEFT_PAREN params=namedExpressionSeq RIGHT_PAREN | params=namedExpressionSeq))?
Review Comment:
It is probably because of SQL Script scoping and you can use `label.cursorName`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2692840041
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -59,7 +59,7 @@ case class OpenCursorExec(
messageParameters = Map("cursorName" -> cursorRef.sql)))
currentState match {
- case CursorDeclared => // Expected state
+ case CursorDeclared | CursorClosed => // Expected states - new or closed cursor
Review Comment:
eYes, if you re-open a closed cursor the query is re-executed and teh cursor set before the first row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699440817
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
PPS: We COULD decide that table ref is separate as well. We could also introduce cursor ref variables. The later being not cursors though any more than tableref are tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699550359
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.catalog.VariableDefinition
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorReference, Expression, Literal, VariableReference}
+import org.apache.spark.sql.classic.Dataset
+import org.apache.spark.sql.errors.DataTypeErrorsBase
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorFetching, CursorOpened}
+
+/**
+ * Physical plan node for fetching from cursors.
+ *
+ * Transitions cursor from Opened to Fetching state on first fetch (creating result iterator),
+ * then fetches rows from the iterator on subsequent calls. Assigns fetched values to target
+ * variables with ANSI store assignment rules.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param targetVariables Variables to fetch into
+ */
+case class FetchCursorExec(
+ cursor: Expression,
+ targetVariables: Seq[VariableReference]) extends LeafV2CommandExec with DataTypeErrorsBase {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.definition.name)
+ val variableManager = SqlScriptingContextManager.get().get.getVariableManager
+
+ // Get current cursor state
+ val currentState = scriptingContext.getCursorState(cursorRef).getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name))))
+
+ // Get or create iterator based on current state
+ val (iterator, analyzedQuery) = currentState match {
+ case CursorOpened(query) =>
+ // First fetch - create iterator and transition to Fetching state
+ val df = Dataset.ofRows(
+ session.asInstanceOf[org.apache.spark.sql.classic.SparkSession],
+ query)
+ val iter = df.toLocalIterator()
+ scriptingContext.updateCursorState(
+ cursorRef.normalizedName,
+ cursorRef.scopeLabel,
+ CursorFetching(query, iter))
+ (iter, query)
+
+ case CursorFetching(query, iter) =>
+ // Subsequent fetch - use existing iterator
+ (iter, query)
+
+ case _ =>
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_OPEN",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name)))
+ }
+
+ // Get next row from iterator
+ if (!iterator.hasNext) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_NO_MORE_ROWS",
+ messageParameters = Map("cursorName" -> toSQLId(cursorRef.definition.name)))
+ }
+
+ val externalRow = iterator.next()
+
+ // Convert Row to InternalRow for processing
+ val schema = org.apache.spark.sql.catalyst.types.DataTypeUtils
+ .fromAttributes(analyzedQuery.output)
+ val converter = org.apache.spark.sql.catalyst.CatalystTypeConverters
+ .createToCatalystConverter(schema)
+ val currentRow = converter(externalRow).asInstanceOf[InternalRow]
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699551509
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6581,6 +6583,120 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = if (ctx.INSENSITIVE() != null) false else true
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
Review Comment:
Fixed by rearranging the gramar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2680021294
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ params.namedExpression().asScala.toSeq.map(visitNamedExpression).map {
+ case alias: Alias => (alias.child, alias.name)
Review Comment:
I've aligned the code with EXECUTE IMMEDIATE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2680060867
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ params.namedExpression().asScala.toSeq.map(visitNamedExpression).map {
+ case alias: Alias => (alias.child, alias.name)
+ case expr => (expr, "") // Empty string for positional parameter
+ }.unzip
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(nameParts, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ val targetVariables = ctx.identifierReference().asScala.map { varIdent =>
+ val varName = if (varIdent.expression() != null) {
+ // IDENTIFIER(expression) case - not supported for variables
Review Comment:
I've added IDENTIFIER support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2680063418
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
Review Comment:
It does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2680270324
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+
+/**
+ * Physical plan node for closing cursors.
+ * Releases cursor resources and resets its state to closed.
+ *
+ * @param nameParts Cursor name parts (unqualified: Seq(name) or qualified: Seq(label, name))
+ */
+case class CloseCursorExec(nameParts: Seq[String]) extends LeafV2CommandExec {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690639092
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -329,6 +335,34 @@ class CompoundBodyExec(
@tailrec
override def next(): CompoundStatementExec = {
+ // If we need to return current without advancing (after CONTINUE handler),
Review Comment:
Let's please test code without this `returnCurrentWithoutAdvancing` because it would be much simpler to avoid implement another flag that needs to be carefully tracked during script execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690815817
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorDefinition, CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope label and cursor name
+ val (scopeLabel, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (Some(nameParts.head), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (None, nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
+ val normalizedName = if (caseSensitive) {
+ cursorName
+ } else {
+ cursorName.toLowerCase(Locale.ROOT)
Review Comment:
shall we normalize the `scopeLabel` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2695133681
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
@davidm-db How are cursors behving differently than variables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2738070202
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -115,6 +389,8 @@ class SqlScriptingExecutionFrame(
def enterScope(
label: String,
triggerToExceptionHandlerMap: TriggerToExceptionHandlerMap): Unit = {
+ // Note: Labels are already lowercased in ParserUtils#enterLabeledScope during parsing,
Review Comment:
```suggestion
// Labels are already lowercased in ParserUtils#enterLabeledScope during parsing,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2738068255
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:
##########
@@ -186,7 +186,7 @@ class SqlScriptingExecution(
}
}
// If there are still frames available, get the next statement.
- if (context.frames.nonEmpty) {
+ if (context.frames.nonEmpty && context.frames.last.hasNext) {
Review Comment:
```suggestion
if (context.frames.nonEmpty) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646498662
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/DeclareCursorExec.scala:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorDefinition
+
+/**
+ * Physical plan node for declaring cursors.
+ * Creates a cursor definition in the current scope and parses its query from SQL text.
+ * The query is parsed at declaration time but not analyzed to preserve parameter markers.
+ *
+ * @param cursorName Name of the cursor
+ * @param query Placeholder LogicalPlan (not used at execution)
+ * @param queryText Original SQL text of the cursor query (used for execution)
+ * @param asensitive Whether the cursor is ASENSITIVE (currently all cursors are INSENSITIVE)
+ */
+case class DeclareCursorExec(
+ cursorName: String,
+ query: LogicalPlan,
Review Comment:
again, why do we add this field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646484412
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
Review Comment:
why do we need a LogicalPlan field in `DeclareCursor`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646478656
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE strictIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
+ ;
+
+openCursorStatement
+ : OPEN multipartIdentifier (USING (LEFT_PAREN params=namedExpressionSeq RIGHT_PAREN | params=namedExpressionSeq))?
Review Comment:
how can cursor be qualified?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646481693
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,70 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR command.
+ * Note: The query child should not be fully analyzed/optimized at declaration time
+ * because we need to re-analyze it each time the cursor is opened to see fresh
Review Comment:
This looks not true, as we also record the SQL text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on PR #53530:
URL: https://github.com/apache/spark/pull/53530#issuecomment-3807228784
Thanks for incorporating the change from https://github.com/apache/spark/pull/53894, but let's please merge that PR independently from this one. They clearly do 2 different stuff, and the "fix" PR is self contained and has its purpose. It will be easier to debug and understand changes later if we merge it one by one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2695022643
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.VariableReference
+import org.apache.spark.sql.catalyst.plans.logical.{FetchCursor, LogicalPlan, SingleStatement}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.errors.QueryCompilationErrors.unresolvedVariableError
+
+/**
+ * Resolves the target SQL variables in FetchCursor command.
+ * Variables can be either scripting local variables or session variables.
+ */
+class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[LogicalPlan]
+ with ColumnResolutionHelper {
+ // VariableResolution looks up both scripting local variables (via SqlScriptingContextManager)
+ // and session variables (via tempVariableManager), checking local variables first.
+ private val variableResolution = new VariableResolution(catalogManager.tempVariableManager)
+
+ /**
+ * Checks for duplicate variable names and throws an exception if found.
+ * Names are normalized when the variables are created.
+ * No need for case insensitive comparison here.
+ */
+ private def checkForDuplicateVariables(variables: Seq[VariableReference]): Unit = {
Review Comment:
I have a SELECT INTO PR which will follow. We can unify there. I have already unified USING for EXECUTE and OPEN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699711370
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -314,6 +314,12 @@ class CompoundBodyExec(
/** Used to stop the iteration in cases when LEAVE statement is encountered. */
private var stopIteration = false
+ /**
+ * Flag to prevent advancing curr after an exception was thrown and handled by a CONTINUE handler.
+ * When true, the next call to next() should return curr without advancing.
+ */
+ private[scripting] var returnCurrentWithoutAdvancing = false
Review Comment:
The Key Insight
FETCH INTO raising CURSOR_NO_MORE_ROWS is indeed the FIRST statement in Spark to raise a SQLSTATE '02xxx' (NOT FOUND / NO DATA) condition.
What This Means
Before cursors: Spark had never encountered a completion condition (SQLSTATE '02xxx')
Existing exception handling: Assumes ALL exceptions abort execution without a handler
SQL Standard: SQLSTATE '02xxx' are completion conditions, not exceptions - they should allow continuation even WITHOUT a handler
The problem: Spark treats completion conditions as fatal exceptions
The Evidence
✅ Only ONE SQLSTATE '02xxx' in all of Spark: CURSOR_NO_MORE_ROWS (02000)
✅ All 23 non-cursor CONTINUE handler tests pass WITHOUT the patch
✅ 11 cursor tests fail WITHOUT the patch
✅ The issue reproduces even WITHOUT declaring any handlers
The Wrong Fix (What We Removed)
The returnCurrentWithoutAdvancing patch was trying to fix iterator advancement after exception handling, but this was addressing the symptom (wrong iterator state) not the cause (completion conditions treated as exceptions).
The Right Fix (What's Needed)
Simply check if the SQLSTATE starts with '02' and allow continuation:
case None if e.getSqlState != null && e.getSqlState.startsWith("02") => // Completion condition without handler - continue execution (SQL Standard) ()case None => // True exception without handler - abort throw e
This is the proper SQL Standard-compliant solution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2699428865
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -53,6 +55,108 @@ class SqlScriptingExecutionContext extends SqlScriptingExecutionContextExtension
def currentFrame: SqlScriptingExecutionFrame = frames.last
def currentScope: SqlScriptingExecutionScope = currentFrame.currentScope
+ /**
+ * Find a cursor by its normalized name in the current scope and parent scopes.
+ * Implementation of SqlScriptingExecutionContextExtension API.
+ *
+ * Searches current frame first (respects shadowing), then script frame (for cross-frame access).
+ * This ensures cursors declared in handlers shadow cursors with the same name in the script.
+ */
+ override def findCursorByName(normalizedName: String): Option[CursorDefinition] = {
Review Comment:
Actually it does work. We have tests where we close the cursor inside the exit handler.
The tests are only one level, instead of two, but that gives me confidence...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691442031
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleIdCollection.scala:
##########
@@ -97,6 +97,7 @@ object RuleIdCollection {
"org.apache.spark.sql.catalyst.analysis.ResolveLateralColumnAliasReference" ::
"org.apache.spark.sql.catalyst.analysis.ResolveOrderByAll" ::
"org.apache.spark.sql.catalyst.analysis.ResolveRowLevelCommandAssignments" ::
+ "org.apache.spark.sql.catalyst.analysis.ResolveFetchCursor" ::
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2737742321
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingLocalVariableManager.scala:
##########
@@ -76,39 +76,13 @@ class SqlScriptingLocalVariableManager(context: SqlScriptingExecutionContext)
throw SparkException.internalError("ScriptingVariableManager expects 1 or 2 nameParts.")
}
- // First search for variable in entire current frame.
- val resCurrentFrame = context.currentFrame.scopes
- .findLast(scope => isScopeOfVar(nameParts, scope))
- if (resCurrentFrame.isDefined) {
- return resCurrentFrame
- }
-
- // When searching in previous frames, for each frame we have to check only scopes before and
- // including the scope where the previously checked exception handler frame is defined.
- // Exception handler frames should not have access to variables from scopes
- // which are nested below the scope where the handler is defined.
- var previousFrameDefinitionLabel = context.currentFrame.scopeLabel
-
- // dropRight(1) removes the current frame, which we already checked above.
- context.frames.dropRight(1).reverseIterator.foreach(frame => {
- // Drop scopes until we encounter the scope in which the previously checked
- // frame was defined. If it was not defined in this scope candidateScopes will be
- // empty.
- val candidateScopes = frame.scopes.reverse.dropWhile(
- scope => !previousFrameDefinitionLabel.contains(scope.label))
-
- val scope = candidateScopes.findLast(scope => isScopeOfVar(nameParts, scope))
- if (scope.isDefined) {
- return scope
- }
- // If candidateScopes is nonEmpty that means that we found the previous frame definition
- // in this frame. If we still have not found the variable, we now have to find the definition
- // of this new frame, so we reassign the frame definition label to search for.
- if (candidateScopes.nonEmpty) {
- previousFrameDefinitionLabel = frame.scopeLabel
- }
- })
- None
+ // Use the shared searchAcrossFrames helper to maintain consistent logic with cursors
+ context.searchAcrossFrames(
Review Comment:
It looks good to me for variables, it just a refactor and should work as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2682630496
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule normalizes cursor names based on case sensitivity configuration and
+ * separates qualified cursor names (label.cursor) into scope path and cursor name.
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope path and cursor name
+ val (scopePath, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (nameParts.dropRight(1), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (Seq.empty[String], nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
Review Comment:
I think consolidating to Analysis is better. Will implement that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691227851
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.SqlScriptingContextManager
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule:
+ * 1. Normalizes cursor names based on case sensitivity configuration
+ * 2. Separates qualified cursor names (label.cursor) into scope label and cursor name
+ * 3. Looks up the cursor definition from the scripting context
+ * 4. Fails early if cursor is not found
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Validate name parts: only support unqualified (cursor) or
+ // single-label qualified (label.cursor)
+ if (nameParts.length > 2) {
Review Comment:
we can simply do `assert`, as the parser already validate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2690985543
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/DeclareCursorExec.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, CursorDefinition}
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorDeclared
+
+/**
+ * Physical plan node for declaring cursors.
+ *
+ * Creates a cursor definition and initializes it in the Declared state.
+ * The cursor query is stored as SQL text and is not parsed or analyzed until OPEN time.
+ * This allows parameter markers to be preserved and bound correctly at OPEN.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText Original SQL text of the cursor query (with parameter markers preserved)
+ * @param asensitive Whether the cursor is ASENSITIVE (sensitivity to underlying data changes,
+ * not case sensitivity). Currently all cursors are effectively INSENSITIVE.
+ */
+case class DeclareCursorExec(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorName)
+ val currentScope = scriptingContext.currentScope
+
+ // Normalize cursor name based on case sensitivity configuration
Review Comment:
Already addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691066901
##########
sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala:
##########
@@ -852,4 +852,21 @@ private[sql] object QueryParsingErrors extends DataTypeErrorsBase {
messageParameters = Map("parameterMarker" -> ctx.getText),
ctx = ctx)
}
+
+ /**
+ * Throws an exception when a cursor reference has more than one qualifier.
+ * Valid: cursor or label.cursor
+ * Invalid: a.b.cursor
+ *
+ * @param cursorName
+ * The fully qualified cursor name with multiple qualifiers
+ * @throws ParseException
+ * Always throws this exception
+ */
+ def cursorInvalidQualifierError(cursorName: String): Nothing = {
+ throw new ParseException(
+ errorClass = "CURSOR_REFERENCE_INVALID_QUALIFIER",
+ messageParameters = Map("cursorName" -> cursorName),
Review Comment:
Should we use `toSQLId(cursorName)` here?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleIdCollection.scala:
##########
@@ -97,6 +97,7 @@ object RuleIdCollection {
"org.apache.spark.sql.catalyst.analysis.ResolveLateralColumnAliasReference" ::
"org.apache.spark.sql.catalyst.analysis.ResolveOrderByAll" ::
"org.apache.spark.sql.catalyst.analysis.ResolveRowLevelCommandAssignments" ::
+ "org.apache.spark.sql.catalyst.analysis.ResolveFetchCursor" ::
Review Comment:
Why does `ResolveCursors` not need to be here?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,55 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * The queryText is stored to support both parameterized and non-parameterized cursors.
+ * The query is parsed and analyzed when the cursor is declared at execution time.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean = true) extends LeafCommand
+
+/**
+ * The logical plan of the OPEN cursor command.
+ *
+ * @param cursor Cursor reference (UnresolvedCursor during parsing, CursorReference after analysis)
+ * @param args Parameter expressions from USING clause
+ * @param paramNames Names for each parameter (empty string "" for positional parameters)
+ */
+case class OpenCursor(
+ cursor: Expression,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafCommand {
+
+ assert(args.length == paramNames.length,
+ s"args.length (${args.length}) must equal paramNames.length (${paramNames.length})")
+}
+
+/**
+ * The logical plan of the FETCH cursor command.
+ *
+ * @param cursor Cursor reference (UnresolvedCursor during parsing, CursorReference after analysis)
+ * @param targetVariables Target variables to fetch into
+ */
+case class FetchCursor(
+ cursor: Expression,
+ targetVariables: Seq[Expression]) extends LeafCommand
+
+/**
+ * The logical plan of the CLOSE cursor command.
+ *
+ * @param cursor Cursor reference (UnresolvedCursor during parsing, CursorReference after analysis)
+ */
+case class CloseCursor(
+ cursor: Expression) extends LeafCommand
Review Comment:
nit: This can be a single line.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.catalog.VariableDefinition
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, Literal, VariableReference}
+import org.apache.spark.sql.classic.Dataset
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorFetching, CursorOpened}
+
+/**
+ * Physical plan node for fetching from cursors.
+ *
+ * Transitions cursor from Opened to Fetching state on first fetch (creating result iterator),
+ * then fetches rows from the iterator on subsequent calls. Assigns fetched values to target
+ * variables with ANSI store assignment rules.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param targetVariables Variables to fetch into
+ */
+case class FetchCursorExec(
+ cursor: Expression,
+ targetVariables: Seq[VariableReference]) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.sql)
+ val variableManager = SqlScriptingContextManager.get().get.getVariableManager
+
+ // Get current cursor state
+ val currentState = scriptingContext.currentFrame.getCursorState(
Review Comment:
Can we open cursor in a script body, and then fetch in exception handler body?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorClosed
+
+/**
+ * Physical plan node for closing cursors.
+ *
+ * Transitions cursor from Opened or Fetching state to Closed state, releasing resources.
+ * Closing an already closed cursor or a declared-but-not-opened cursor raises an error.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ */
+case class CloseCursorExec(cursor: Expression) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
Review Comment:
nit: Can we import this properly and use only `CursorReference`?
Also there are multiple places in code where this can be improved.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorClosed
+
+/**
+ * Physical plan node for closing cursors.
+ *
+ * Transitions cursor from Opened or Fetching state to Closed state, releasing resources.
+ * Closing an already closed cursor or a declared-but-not-opened cursor raises an error.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ */
+case class CloseCursorExec(cursor: Expression) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.sql)
+
+ // Get current cursor state and validate it exists
+ val currentState = scriptingContext.currentFrame.getCursorState(
Review Comment:
Similar question to the one in Fetch Cursor, is there a way to open cursor in a script body, use it, and then close it in exception handler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691237637
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorClosed
+
+/**
+ * Physical plan node for closing cursors.
+ *
+ * Transitions cursor from Opened or Fetching state to Closed state, releasing resources.
+ * Closing an already closed cursor or a declared-but-not-opened cursor raises an error.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ */
+case class CloseCursorExec(cursor: Expression) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.sql)
+
+ // Get current cursor state and validate it exists
+ val currentState = scriptingContext.currentFrame.getCursorState(
+ cursorRef.normalizedName,
+ cursorRef.scopeLabel).getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorRef.sql)))
+
+ // Validate cursor is in an open state (Opened or Fetching)
+ currentState match {
+ case CursorClosed | org.apache.spark.sql.scripting.CursorDeclared =>
Review Comment:
this is weird, shall we import both state classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2681826617
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCursors.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import java.util.Locale
+
+import org.apache.spark.sql.catalyst.expressions.{CursorReference, UnresolvedCursor}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.UNRESOLVED_CURSOR
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Resolves [[UnresolvedCursor]] expressions to [[CursorReference]] expressions.
+ * This rule normalizes cursor names based on case sensitivity configuration and
+ * separates qualified cursor names (label.cursor) into scope path and cursor name.
+ */
+class ResolveCursors extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveExpressionsWithPruning(
+ _.containsPattern(UNRESOLVED_CURSOR)) {
+ case uc: UnresolvedCursor if !uc.resolved =>
+ resolveCursor(uc)
+ }
+
+ private def resolveCursor(uc: UnresolvedCursor): CursorReference = {
+ val nameParts = uc.nameParts
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+
+ // Split qualified name into scope path and cursor name
+ val (scopePath, cursorName) = if (nameParts.length > 1) {
+ // Qualified cursor: "label.cursor"
+ (nameParts.dropRight(1), nameParts.last)
+ } else {
+ // Unqualified cursor: "cursor"
+ (Seq.empty[String], nameParts.head)
+ }
+
+ // Normalize cursor name based on case sensitivity
Review Comment:
thinking about it again, if we don't treat cursor resolution as an analysis step, we can move everything to the physical plan, probably with a util function to save duplicated code between these physical plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2681836097
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/DeclareCursorExec.scala:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorDeclared, CursorDefinition}
+
+/**
+ * Physical plan node for declaring cursors.
+ *
+ * Creates a cursor definition and initializes it in the Declared state.
+ * The cursor query is parsed but not analyzed/executed until OPEN.
+ *
+ * @param cursorName Name of the cursor
+ * @param queryText Original SQL text of the cursor query
+ * @param asensitive Whether the cursor is ASENSITIVE (sensitivity to underlying data changes,
+ * not case sensitivity). Currently all cursors are effectively INSENSITIVE.
+ */
+case class DeclareCursorExec(
+ cursorName: String,
+ queryText: String,
+ asensitive: Boolean) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContextManager = SqlScriptingContextManager.get()
+ .getOrElse(throw new AnalysisException(
+ errorClass = "CURSOR_OUTSIDE_SCRIPT",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ val scriptingContext = scriptingContextManager.getContext
+ .asInstanceOf[org.apache.spark.sql.scripting.SqlScriptingExecutionContext]
+ val currentScope = scriptingContext.currentScope
Review Comment:
shall we have a util function to get current cursors from the current scripting context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2679982661
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
Review Comment:
This isn't about case-sensitive. It's about whether the cursor can see concurrent changes after the cursor is opened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691224558
##########
sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala:
##########
@@ -852,4 +852,21 @@ private[sql] object QueryParsingErrors extends DataTypeErrorsBase {
messageParameters = Map("parameterMarker" -> ctx.getText),
ctx = ctx)
}
+
+ /**
+ * Throws an exception when a cursor reference has more than one qualifier.
+ * Valid: cursor or label.cursor
+ * Invalid: a.b.cursor
+ *
+ * @param cursorName
+ * The fully qualified cursor name with multiple qualifiers
+ * @throws ParseException
+ * Always throws this exception
+ */
+ def cursorInvalidQualifierError(cursorName: String): Nothing = {
+ throw new ParseException(
Review Comment:
nit: this is anti-pattern. Looking at most methods in this file, the method should return `Throwable` and the caller side should do `throw cursorInvalidQualifierError(...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691455877
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import java.util.Locale
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.catalog.VariableDefinition
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, Literal, VariableReference}
+import org.apache.spark.sql.classic.Dataset
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.{CursorFetching, CursorOpened}
+
+/**
+ * Physical plan node for fetching from cursors.
+ *
+ * Transitions cursor from Opened to Fetching state on first fetch (creating result iterator),
+ * then fetches rows from the iterator on subsequent calls. Assigns fetched values to target
+ * variables with ANSI store assignment rules.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ * @param targetVariables Variables to fetch into
+ */
+case class FetchCursorExec(
+ cursor: Expression,
+ targetVariables: Seq[VariableReference]) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.sql)
+ val variableManager = SqlScriptingContextManager.get().get.getVariableManager
+
+ // Get current cursor state
+ val currentState = scriptingContext.currentFrame.getCursorState(
Review Comment:
Adding a test for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2691454672
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+import org.apache.spark.sql.scripting.CursorClosed
+
+/**
+ * Physical plan node for closing cursors.
+ *
+ * Transitions cursor from Opened or Fetching state to Closed state, releasing resources.
+ * Closing an already closed cursor or a declared-but-not-opened cursor raises an error.
+ *
+ * @param cursor CursorReference resolved during analysis phase
+ */
+case class CloseCursorExec(cursor: Expression) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ // Extract CursorReference from the resolved cursor expression
+ val cursorRef = cursor.asInstanceOf[org.apache.spark.sql.catalyst.expressions.CursorReference]
+
+ val scriptingContext = CursorCommandUtils.getScriptingContext(cursorRef.sql)
+
+ // Get current cursor state and validate it exists
+ val currentState = scriptingContext.currentFrame.getCursorState(
Review Comment:
We have this test already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2737592270
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:
##########
@@ -186,7 +186,7 @@ class SqlScriptingExecution(
}
}
// If there are still frames available, get the next statement.
- if (context.frames.nonEmpty) {
+ if (context.frames.nonEmpty && context.frames.last.hasNext) {
Review Comment:
Above while loop guarantees that if `context.frames.nonEmpty`, `context.frames.last.hasNext` will be true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646483766
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -1927,6 +1927,70 @@ case class SetVariable(
copy(sourceQuery = newChild)
}
+/**
+ * The logical plan of the DECLARE CURSOR command.
+ * Note: The query child should not be fully analyzed/optimized at declaration time
+ * because we need to re-analyze it each time the cursor is opened to see fresh
+ * data.
+ */
+/**
+ * The logical plan of the DECLARE CURSOR statement.
+ *
+ * @param cursorName Name of the cursor
+ * @param query The parsed logical plan of the cursor query
+ * @param queryText The original SQL text of the query (preserves parameter markers)
+ * @param asensitive Whether the cursor is ASENSITIVE or INSENSITIVE
+ */
+case class DeclareCursor(
+ cursorName: String,
+ query: LogicalPlan,
+ queryText: String,
+ asensitive: Boolean = true) extends UnaryCommand {
+ override def child: LogicalPlan = query
+ override protected def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
+ copy(query = newChild)
+
+ // Mark that the child query should not be eagerly analyzed
+ override lazy val resolved: Boolean = cursorName.nonEmpty
+}
+
+/**
+ * The logical plan of the OPEN cursor command.
+ *
+ * @param cursorName Name of the cursor to open
+ * @param args Parameter expressions from USING clause
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursor(
+ cursorName: String,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends Command {
Review Comment:
we can extend `LeafCommand`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646485036
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
Review Comment:
This is so weird, why do we allow qualified name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2646502266
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/OpenCursorExec.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+
+/**
+ * Physical plan node for opening cursors.
+ *
+ * @param cursorName Name of the cursor to open
+ * @param args Parameter expressions from USING clause (for parameterized cursors)
+ * @param paramNames Names for each parameter (empty string for positional parameters)
+ */
+case class OpenCursorExec(
+ cursorName: String,
+ args: Seq[Expression] = Seq.empty,
+ paramNames: Seq[String] = Seq.empty) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val scriptingContextManager = SqlScriptingContextManager.get()
+ .getOrElse(throw new AnalysisException(
+ errorClass = "CURSOR_OUTSIDE_SCRIPT",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ val scriptingContext = scriptingContextManager.getContext
+ .asInstanceOf[org.apache.spark.sql.scripting.SqlScriptingExecutionContext]
+
+ // Parse and normalize cursor name for case sensitivity
+ val cursorNameParts = parseCursorName(cursorName)
+
+ // Find cursor in scope hierarchy
+ val cursorDef = scriptingContext.currentFrame.findCursorByNameParts(cursorNameParts)
+ .getOrElse(
+ throw new AnalysisException(
+ errorClass = "CURSOR_NOT_FOUND",
+ messageParameters = Map("cursorName" -> cursorName)))
+
+ // Validate cursor is not already open
+ if (cursorDef.isOpen) {
+ throw new AnalysisException(
+ errorClass = "CURSOR_ALREADY_OPEN",
+ messageParameters = Map("cursorName" -> cursorName))
+ }
+
+ // Execute the query and collect results
+ val resultData = if (args.nonEmpty) {
+ // Parameterized cursor: re-parse query text with bound parameters
+ val (data, analyzedPlan) = executeParameterizedQuery(cursorDef.queryText, args)
+ // Update the cursor's query with the analyzed plan so FETCH can access the schema
+ cursorDef.query = analyzedPlan
+ data
+ } else {
+ // Non-parameterized cursor: execute the analyzed plan directly
+ val queryExecution = session.sessionState.executePlan(cursorDef.query)
+ queryExecution.executedPlan.executeCollect()
Review Comment:
this is very memory intensive. Can we use `df.toLocalIterator`? This is also how we implement FOR loop in scripting.
Using iterator means we can only read the record one by one, not going back. Does the CURSOR feature need to go back and read previous rows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2648790995
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,100 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.strictIdentifier().getText
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ // Use a placeholder plan (LocalRelation) - the actual query will be parsed at execution time
+ // This prevents parameter markers from being resolved during script analysis
+ val query: LogicalPlan = LocalRelation()
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, query, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ val cursorName = ctx.multipartIdentifier().parts.asScala.map(_.getText).mkString(".")
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ val namedExprs = params.namedExpression().asScala.toSeq.map(visitNamedExpression)
+ val exprs = namedExprs.map {
+ case alias: Alias => alias.child // Extract child expression
+ case expr => expr
+ }
+ val names = namedExprs.map {
+ case alias: Alias => alias.name // Extract alias name
+ case _ => "" // Positional parameter (no name)
+ }
+ (exprs, names)
Review Comment:
Empty string "" in paramNames is treated as a positional parameter
It gets converted to "_pos_$idx" internally
So empty string IS the correct way to represent positional parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2694989892
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -144,6 +148,22 @@ declareHandlerStatement
: DECLARE (CONTINUE | EXIT) HANDLER FOR conditionValues (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
;
+declareCursorStatement
+ : DECLARE name=errorCapturingIdentifier (ASENSITIVE | INSENSITIVE)? CURSOR FOR query (FOR READ ONLY)?
Review Comment:
Yes. Declare cannot be qualified, references can be qualified with labels.
errorCapturingIdentiifer specifically also tries to give a better / semantic error message for dashes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2695112139
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala:
##########
@@ -115,7 +295,13 @@ class SqlScriptingExecutionFrame(
def enterScope(
label: String,
triggerToExceptionHandlerMap: TriggerToExceptionHandlerMap): Unit = {
- scopes.append(new SqlScriptingExecutionScope(label, triggerToExceptionHandlerMap))
+ // Normalize label for case-insensitive lookups
+ val normalizedLabel = if (SQLConf.get.caseSensitiveAnalysis) {
+ label
+ } else {
+ label.toLowerCase(Locale.ROOT)
+ }
Review Comment:
IIRC, labels are case insensitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660321942
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
Review Comment:
hmm, so the default behavior is case sensitive? This is different from Spark...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660337078
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -6573,6 +6575,110 @@ class AstBuilder extends DataTypeAstBuilder
)
}
+ /**
+ * Create a [[DeclareCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * DECLARE cursor_name CURSOR FOR SELECT * FROM table;
+ * }}}
+ */
+ override def visitDeclareCursorStatement(
+ ctx: DeclareCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ val cursorName = getIdentifierText(ctx.name)
+ // Extract original SQL text to preserve parameter markers
+ val queryText = getOriginalText(ctx.query())
+
+ val asensitive = ctx.ASENSITIVE() != null || ctx.INSENSITIVE() != null
+ SingleStatement(DeclareCursor(cursorName, queryText, asensitive))
+ }
+
+ /**
+ * Create an [[OpenCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * OPEN cursor_name;
+ * OPEN cursor_name USING expr1, expr2;
+ * OPEN cursor_name USING (expr1 AS param1, expr2 AS param2);
+ * }}}
+ */
+ override def visitOpenCursorStatement(
+ ctx: OpenCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ // Parse optional USING clause parameters
+ // Extract both expressions and their names (if aliased)
+ val (args, paramNames) = Option(ctx.params).map { params =>
+ params.namedExpression().asScala.toSeq.map(visitNamedExpression).map {
+ case alias: Alias => (alias.child, alias.name)
+ case expr => (expr, "") // Empty string for positional parameter
+ }.unzip
+ }.getOrElse((Seq.empty, Seq.empty))
+
+ SingleStatement(OpenCursor(nameParts, args, paramNames))
+ }
+
+ /**
+ * Create a [[FetchCursor]] command wrapped in SingleStatement.
+ *
+ * For example:
+ * {{{
+ * FETCH cursor_name INTO var1, var2;
+ * }}}
+ */
+ override def visitFetchCursorStatement(
+ ctx: FetchCursorStatementContext): LogicalPlan = withOrigin(ctx) {
+ if (!conf.getConf(SQLConf.SQL_SCRIPTING_CURSOR_ENABLED)) {
+ throw SqlScriptingErrors.cursorNotSupported(CurrentOrigin.get)
+ }
+
+ // Use visitMultipartIdentifier to properly handle IDENTIFIER('name')
+ val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier())
+
+ val targetVariables = ctx.identifierReference().asScala.map { varIdent =>
+ val varName = if (varIdent.expression() != null) {
+ // IDENTIFIER(expression) case - not supported for variables
Review Comment:
shall we enforce it at the parser rule level? e.g. use `multipartIdentifier` instead of `identifierReference`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2660350322
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.{InternalRow, SqlScriptingContextManager}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.execution.datasources.v2.LeafV2CommandExec
+
+/**
+ * Physical plan node for closing cursors.
+ * Releases cursor resources and resets its state to closed.
+ *
+ * @param nameParts Cursor name parts (unqualified: Seq(name) or qualified: Seq(label, name))
+ */
+case class CloseCursorExec(nameParts: Seq[String]) extends LeafV2CommandExec {
Review Comment:
it can also unify the cursor lookup logic and save duplicated code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54759][SQL] SQL scripting Cursor support [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #53530:
URL: https://github.com/apache/spark/pull/53530#discussion_r2661188274
##########
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##########
@@ -313,6 +319,34 @@ class CompoundBodyExec(
@tailrec
override def next(): CompoundStatementExec = {
+ // If we need to return current without advancing (after CONTINUE handler),
Review Comment:
`CONTINUE HANDLER` has a known bug and a fix is currently in progress. Specifically, it does not behave correctly when the handled statement is executed inside a loop.
The following example fails with the current implementation:
```
BEGIN
-- Multi-variable declaration (New in 4.1)
DECLARE x, y = 1;
-- Define error handling logic
DECLARE CONTINUE HANDLER FOR DIVIDE_BY_ZERO
BEGIN
SET y = -1;
END;
-- Loop logic
WHILE x < 8 DO
SET x = x + 1;
-- This might throw a divide-by-zero error, but the handler will catch it
SET y = y / (x - 3);
END WHILE;
SELECT x, y;
END;
```
I am working on fixing this issue and plan to send a dedicated PR for review in the coming days.
Given that this area of the code is particularly touchy, I’d strongly prefer that we review the other parts of this PR first. The current approach here feels somewhat hacky, and building additional logic on top of `CONTINUE HANDLER` is risky at this stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org