You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "srielau (via GitHub)" <gi...@apache.org> on 2025/09/13 20:42:52 UTC
[PR] [WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
srielau opened a new pull request, #52334:
URL: https://github.com/apache/spark/pull/52334
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375109387
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2194,7 +2194,8 @@ class AstBuilder extends DataTypeAstBuilder
Limit(expression(ctx.expression), query)
case ctx: SampleByPercentileContext =>
- val fraction = ctx.percentage.getText.toDouble
Review Comment:
why do we change this? I think the parameter markers should have been replaced already when we reach here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429735625
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
Review Comment:
To simplify the pre-parser, shall we use a fake plan to resolve and constant-fold the parameter values first? Then we don't need `LiteralToSqlConverter` as the parameter value is always `Literal` and we can simply use `Literal#sql`.
We can use a fake plan `Project(parameters, OneRowRelation)`, analyze and optimize, and get the project list. They must be all literals otherwise we should report error for parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433107220
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -320,25 +320,41 @@ class SparkConnectPlanner(
}
private def transformSql(sql: proto.SQL): LogicalPlan = {
+ val queryText = substituteParametersInQuery(sql)
+ // Parse the substituted query
+ parser.parsePlan(queryText)
Review Comment:
We can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433576054
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -957,13 +963,13 @@ joinCriteria
;
sample
- : TABLESAMPLE LEFT_PAREN sampleMethod? RIGHT_PAREN (REPEATABLE LEFT_PAREN seed=INTEGER_VALUE RIGHT_PAREN)?
+ : TABLESAMPLE LEFT_PAREN sampleMethod? RIGHT_PAREN (REPEATABLE LEFT_PAREN seed=integerValue RIGHT_PAREN)?
;
-sampleMethod
- : negativeSign=MINUS? percentage=(INTEGER_VALUE | DECIMAL_VALUE) PERCENTLIT #sampleByPercentile
+ sampleMethod
Review Comment:
```suggestion
sampleMethod
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2388937868
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
+ */
+class ParameterHandler {
+
+
+ // Thread-local storage for position mapper to ensure thread safety
+ private val positionMapperStorage = new ThreadLocal[Option[PositionMapper]]() {
+ override def initialValue(): Option[PositionMapper] = None
+ }
+
+ /**
+ * Get the current position mapper for translating error positions.
+ * This mapper can translate positions from substituted text back to original text.
+ *
+ * @return The current position mapper, if available
+ */
+ def getPositionMapper: Option[PositionMapper] = positionMapperStorage.get()
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param rule The substitution rule to use
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ rule: SubstitutionRule,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ positionMapperStorage.set(Some(PositionMapper.identity(sqlText)))
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText, rule,
+ namedParams = namedParams, positionalParams = positionalParams)
+ positionMapperStorage.set(Some(positionMapper))
+ substituted
+ }
+
+ /**
+ * Translate error context from substituted text positions to original text positions.
+ *
+ * @param context The error context with positions in substituted text
+ * @return The error context with positions mapped to original text
+ */
+ def translateErrorContext(context: SQLQueryContext): SQLQueryContext = {
+ positionMapperStorage.get() match {
+ case Some(mapper) =>
+ val translatedStartIndex = context.originStartIndex.map(mapper.mapToOriginal)
+ val translatedStopIndex = context.originStopIndex.map(mapper.mapToOriginal)
+
+ context.copy(
+ originStartIndex = translatedStartIndex,
+ originStopIndex = translatedStopIndex,
+ sqlText = Some(mapper.originalText) // Use original text instead of substituted
+ )
+ case None =>
+ context // No position mapper available, return as-is
+ }
+ }
+
+ /**
+ * Substitute parameters in SQL text based on the parameter context.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param context The parameter context (named or positional)
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteParameters(
+ sqlText: String,
+ context: ParameterContext,
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ context match {
+ case NamedParameterContext(params) =>
+ performSubstitution(sqlText, rule, namedParams = params.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
+ })
+
+ case PositionalParameterContext(params) =>
+ performSubstitution(sqlText, rule,
+ positionalParams = params.map(expr =>
+ LiteralToSqlConverter.convert(expr)).toList)
+
+ case HybridParameterContext(args, paramNames) =>
+ // Detect which parameter types are actually used in the query (single call)
+ val (hasPositional, hasNamed) = detectParameters(sqlText, rule)
+
+ // Validate that the query doesn't mix parameter types
+ if (hasPositional && hasNamed) {
+ throw org.apache.spark.sql.errors.QueryCompilationErrors
+ .invalidQueryMixedQueryParameters()
+ }
+
+ // Validate ALL_PARAMETERS_MUST_BE_NAMED: if query uses named parameters,
+ // all USING expressions must have names
+ if (hasNamed && !hasPositional) {
+ val unnamedExpressions = paramNames.zipWithIndex.collect {
+ case ("", index) => index // empty strings are unnamed
+ }
+ if (unnamedExpressions.nonEmpty) {
+ val unnamedExprs = unnamedExpressions.map(args(_)).map {
+ case literal: org.apache.spark.sql.catalyst.expressions.Literal =>
+ // Already a Literal from ResolveExecuteImmediate - use it directly
+ literal.asInstanceOf[Expression]
+ case expr: org.apache.spark.sql.catalyst.expressions.Expression =>
+ // Expression from Column object - use it directly
+ expr
+ case value =>
+ // Raw value - create new Literal
+ Literal(value).asInstanceOf[Expression]
+ }.toSeq
+ throw org.apache.spark.sql.errors.QueryCompilationErrors
+ .invalidQueryAllParametersMustBeNamed(unnamedExprs)
+ }
+ }
+
+ if (hasPositional && !hasNamed) {
+ // Query uses only positional parameters
+ // All arguments are already expressions at this point
+ val positionalParams = args.map(_.asInstanceOf[Expression]).toSeq
+ performSubstitution(sqlText, rule,
+ positionalParams = positionalParams.map(expr =>
+ LiteralToSqlConverter.convert(expr)).toList)
+ } else if (hasNamed && !hasPositional) {
+ // Query uses only named parameters
+ val namedParams = paramNames.zip(args).collect {
+ case (name, literal: org.apache.spark.sql.catalyst.expressions.Literal)
+ if name.nonEmpty =>
+ // Already a Literal from ResolveExecuteImmediate - use it directly
+ name -> literal.asInstanceOf[Expression]
+ case (name, expr: org.apache.spark.sql.catalyst.expressions.Expression)
+ if name.nonEmpty =>
+ // Expression from Column object - use it directly
+ name -> expr
+ case (name, value) if name.nonEmpty =>
+ // Raw value - create new Literal
+ name -> Literal(value).asInstanceOf[Expression]
+ }.toMap
+ performSubstitution(sqlText, rule,
+ namedParams = namedParams.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
+ })
+ } else {
+ // No parameters in query - return as-is
+ performSubstitution(sqlText, rule)
+ }
+ }
+ }
+
+ /**
+ * Substitute parameters in SQL text with optional context.
+ * If no context is provided, returns the original SQL text.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param contextOpt Optional parameter context
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteParametersIfNeeded(
+ sqlText: String,
+ contextOpt: Option[ParameterContext],
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ contextOpt match {
+ case Some(context) => substituteParameters(sqlText, context, rule)
+ case None => sqlText
+ }
+ }
+
+ /**
+ * Substitute parameters using named parameter map.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param paramMap Map of parameter names to expressions
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteNamedParameters(
+ sqlText: String,
+ paramMap: Map[String, Expression],
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ if (paramMap.isEmpty) return sqlText
+
+ val paramValues = paramMap.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
+ }
+ performSubstitution(sqlText, rule, namedParams = paramValues)
+ }
+
+ /**
+ * Substitute parameters using positional parameter list.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param paramList Sequence of parameter expressions
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substitutePositionalParameters(
+ sqlText: String,
+ paramList: Seq[Expression],
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ if (paramList.isEmpty) return sqlText
+
+ val paramValues = paramList.map(expr =>
+ LiteralToSqlConverter.convert(expr)).toList
+ performSubstitution(sqlText, rule, positionalParams = paramValues)
+ }
+
+ /**
+ * Determine the appropriate substitution rule based on SQL content.
+ *
+ * @param sqlText The SQL text to analyze
+ * @return The appropriate substitution rule
+ */
+ def determineSubstitutionRule(sqlText: String): SubstitutionRule = {
+ val trimmed = sqlText.trim
+ if (trimmed.length >= 5 && trimmed.regionMatches(true, 0, "BEGIN", 0, 5)) {
Review Comment:
This entire rule determination stuff is obsolete. It originated from the time when I tried to parse expressions etc selectively...
I'm removing this API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2388780630
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Create a named parameter which represents a literal with a non-bound value and unknown type.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): String =
Review Comment:
It does not. FIxed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416631359
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -45,16 +45,43 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
withOrigin(ctx)(StructType(visitColTypeList(ctx.colTypeList)))
}
- override def visitStringLit(ctx: StringLitContext): Token = {
- if (ctx != null) {
- if (ctx.STRING_LITERAL != null) {
- ctx.STRING_LITERAL.getSymbol
- } else {
- ctx.DOUBLEQUOTED_STRING.getSymbol
- }
- } else {
- null
- }
+ override def visitStringLiteralValue(ctx: StringLiteralValueContext): Token =
Review Comment:
can we add some comments to explain that, this parser assumes all parameter markers to be replaced beforehand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2418070200
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
+ val ctx = try {
+ try {
+ // First attempt: SLL mode with bail error strategy
+ parser.setErrorHandler(new SparkParserBailErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ } catch {
+ case e: ParseCancellationException =>
+ // Second attempt: LL mode with full error strategy
+ tokenStream.seek(0) // rewind input stream
+ parser.reset()
+ parser.setErrorHandler(new SparkParserErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ parser.compoundOrSingleStatement()
+ }
+ } catch {
+ case e: Throwable => throw e
+ }
+ val parameterLocations = astBuilder.extractParameterLocations(ctx)
+
+ // Substitute parameters in the original text
+ val (substitutedSql, appliedSubstitutions) = substituteAtLocations(sqlText, parameterLocations,
+ namedParams, positionalParams)
+ val consumedPositionalParams = parameterLocations.positionalParameterLocations.length
+
+ // Create position mapper for error context translation
+ val positionMapper = PositionMapper(sqlText, substitutedSql, appliedSubstitutions)
+
+ (substitutedSql, consumedPositionalParams, positionMapper)
+ }
+
+ /**
+ * Detects parameter markers in SQL text without performing substitution.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text to analyze
+ * @return A tuple of (hasPositionalParameters, hasNamedParameters)
+ */
+ def detectParameters(sqlText: String): (Boolean, Boolean) = {
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (false, false)
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
+ val ctx = try {
+ try {
+ // First attempt: SLL mode with bail error strategy
+ parser.setErrorHandler(new SparkParserBailErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ } catch {
+ case e: ParseCancellationException =>
+ // Second attempt: LL mode with full error strategy
+ tokenStream.seek(0) // rewind input stream
+ parser.reset()
+ parser.setErrorHandler(new SparkParserErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ parser.compoundOrSingleStatement()
+ }
+ } catch {
+ case e: Throwable => throw e
Review Comment:
Outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426397902
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.trees.ParameterSubstitutionInfo
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ val identityMapper = PositionMapper.identity(sqlText)
+ setupSubstitutionContext(sqlText, sqlText, identityMapper, isIdentity = true)
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ setupSubstitutionContext(sqlText, substituted, positionMapper, isIdentity = false)
+ substituted
+ }
+
+ /**
+ * Set up parameter substitution context in CurrentOrigin for error reporting.
+ * This creates a ParameterSubstitutionInfo object with position mapping data.
+ *
+ * @param originalSql The original SQL text before substitution
+ * @param substitutedSql The SQL text after substitution
+ * @param positionMapper The mapper for translating positions
+ * @param isIdentity Whether this is an identity mapping (no substitution occurred)
+ */
+ private[sql] def setupSubstitutionContext(
+ originalSql: String,
+ substitutedSql: String,
+ positionMapper: PositionMapper,
+ isIdentity: Boolean): Unit = {
+
+ // Create parameter substitution info with the position mapper.
+ val substitutionInfo = ParameterSubstitutionInfo(
+ originalSql = originalSql,
+ isIdentity = isIdentity,
+ positionMapper = if (isIdentity) None else Some(positionMapper)
+ )
+
+ // Store the info in CurrentOrigin for position mapping during error reporting.
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val updatedOrigin = currentOrigin.copy(parameterSubstitutionInfo = Some(substitutionInfo))
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.set(updatedOrigin)
Review Comment:
@dtenedor Encouraged me strongly to get rid of all the thread locals... Y'all let me know what you agree on...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426979807
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2200,7 +2200,8 @@ class AstBuilder extends DataTypeAstBuilder
Limit(expression(ctx.expression), query)
case ctx: SampleByPercentileContext =>
- val fraction = ctx.percentage.getText.toDouble
+ val fraction = if (ctx.DECIMAL_VALUE() != null) { ctx.DECIMAL_VALUE().getText.toDouble }
+ else { ctx.integerValue().getText.toDouble }
Review Comment:
This and similar places can use some asserts, then teh else can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2435982265
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This object consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = ParameterHandler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+object ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Performs parameter substitution and returns both the substituted SQL and the position mapper.
+ * The position mapper enables accurate error reporting by mapping positions in the
+ * substituted SQL back to the original SQL text.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return A tuple of (substituted SQL, optional position mapper)
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Option[PositionMapper]) = {
Review Comment:
why `Option[PositionMapper]`? we always return a `Some`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375083365
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Create a named parameter which represents a literal with a non-bound value and unknown type.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): String =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) +=
+ ParameterLocation(startIndex, stopIndex)
Review Comment:
```suggestion
scala.collection.mutable.ListBuffer[ParameterLocation]()
) += ParameterLocation(startIndex, stopIndex)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375112151
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterContext.scala:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * Context for passing parameter values to the SQL parser.
+ * This allows parameter substitution to happen during the parsing phase
+ * rather than during analysis.
+ */
+sealed trait ParameterContext
+
+/**
+ * Context for named parameters (e.g., :paramName).
+ *
+ * @param params Map of parameter names to their expression values
+ */
+case class NamedParameterContext(params: Map[String, Expression]) extends ParameterContext
+
+/**
+ * Context for positional parameters (e.g., ?).
+ *
+ * @param params Sequence of expression values in order
+ */
+case class PositionalParameterContext(params: Seq[Expression]) extends ParameterContext
+
+/**
+ * Context that supports both named and positional parameters.
+ * This is used by EXECUTE IMMEDIATE where the parameter type is determined by the inner query.
+ *
+ * @param args Raw argument values from USING clause (already evaluated)
+ * @param paramNames Parameter names from USING clause (empty strings for positional)
+ */
+case class HybridParameterContext(
+ args: Array[_],
+ paramNames: Array[String]) extends ParameterContext
+
+/**
+ * Thread-local storage for parameter context to pass parameters
+ * from SparkSession to the parser without changing method signatures.
+ */
+object ThreadLocalParameterContext {
+ private val context = new ThreadLocal[Option[ParameterContext]]()
Review Comment:
can we use `LexicalThreadLocal`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375090722
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Create a named parameter which represents a literal with a non-bound value and unknown type.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): String =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) +=
+ ParameterLocation(startIndex, stopIndex)
+
+ paramName
+ }
+
+ /**
+ * Create a positional parameter which represents a literal with a non-bound value and unknown
+ * type.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): String =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ "?"
+ }
+
+ /**
+ * Handle namedParameterValue context for parameter markers in string literal contexts. This
+ * handles the namedParameterMarker case added to the stringLit grammar rule.
+ */
+ override def visitNamedParameterValue(ctx: NamedParameterValueContext): String =
Review Comment:
This is the same as `visitNamedParameterLiteral`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2374374284
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -138,7 +165,7 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
}
} else {
val badType = typeCtx.unsupportedType.getText
- val params = typeCtx.INTEGER_VALUE().asScala.toList
+ val params = typeCtx.integerValue().asScala.map(_.getText).toList
Review Comment:
So the assumption here is the parameter markers should have been replaces and we can safely get the original INTEGER_VALUE text?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -138,7 +165,7 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
}
} else {
val badType = typeCtx.unsupportedType.getText
- val params = typeCtx.INTEGER_VALUE().asScala.toList
+ val params = typeCtx.integerValue().asScala.map(_.getText).toList
Review Comment:
So the assumption here is the parameter markers should have been replaced and we can safely get the original INTEGER_VALUE text?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375139431
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
+ */
+class ParameterHandler {
+
+
+ // Thread-local storage for position mapper to ensure thread safety
+ private val positionMapperStorage = new ThreadLocal[Option[PositionMapper]]() {
+ override def initialValue(): Option[PositionMapper] = None
+ }
+
+ /**
+ * Get the current position mapper for translating error positions.
+ * This mapper can translate positions from substituted text back to original text.
+ *
+ * @return The current position mapper, if available
+ */
+ def getPositionMapper: Option[PositionMapper] = positionMapperStorage.get()
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param rule The substitution rule to use
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ rule: SubstitutionRule,
Review Comment:
are we going to extend it in the future, or a boolean `isScript` flag is sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425625657
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/parsers.scala:
##########
@@ -65,35 +65,14 @@ abstract class AbstractParser extends DataTypeParserInterface with Logging {
val tokenStream = new CommonTokenStream(lexer)
val parser = new SqlBaseParser(tokenStream)
- if (conf.manageParserCaches) AbstractParser.installCaches(parser)
- parser.addParseListener(PostProcessor)
- parser.addParseListener(UnclosedCommentProcessor(command, tokenStream))
- parser.removeErrorListeners()
- parser.addErrorListener(ParseErrorListener)
- parser.legacy_setops_precedence_enabled = conf.setOpsPrecedenceEnforced
- parser.legacy_exponent_literal_as_decimal_enabled = conf.exponentLiteralAsDecimalEnabled
- parser.SQL_standard_keyword_behavior = conf.enforceReservedKeywords
- parser.double_quoted_identifiers = conf.doubleQuotedIdentifiers
+
+ // Use shared parser configuration to ensure consistency.
+ AbstractParser.configureParser(parser, command, tokenStream, conf)
// https://github.com/antlr/antlr4/issues/192#issuecomment-15238595
// Save a great deal of time on correct inputs by using a two-stage parsing strategy.
Review Comment:
shall we move this comment to the method doc of `executeWithTwoStageStrategy`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433598667
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.SubstituteParamsParser
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This object consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = ParameterHandler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+object ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Performs parameter substitution and stores the position mapper in CurrentOrigin.
+ * The position mapper enables accurate error reporting by mapping positions in the
+ * substituted SQL back to the original SQL text.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Optimize: Skip parsing if no parameter markers are present in the SQL text.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ // Store the position mapper to enable accurate error position reporting in the original SQL.
+ val currentOrigin = CurrentOrigin.get
+ CurrentOrigin.set(currentOrigin.copy(positionMapper = Some(positionMapper)))
Review Comment:
This is a side effect that updates a global variable. I'm a bit worried about it as we never restore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429670820
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala:
##########
@@ -84,23 +86,44 @@ case class ResolveExecuteImmediate(sparkSession: SparkSession, catalogManager: C
// Extract the query string from the queryParam expression
val sqlString = extractQueryString(sqlStmtStr)
- // Parse and validate the query
- val parsedPlan = sparkSession.sessionState.sqlParser.parsePlan(sqlString)
- validateQuery(sqlString, parsedPlan)
+ // Create the origin for EXECUTE IMMEDIATE context - this will be used by expressions
+ // during parsing to set their queryContext, similar to how views work
+ val executeImmediateOrigin = Origin(
+ objectType = Some("EXECUTE IMMEDIATE"),
+ objectName = None, // No named object for EXECUTE IMMEDIATE, unlike views
+ sqlText = Some(sqlString),
+ startIndex = Some(0),
+ stopIndex = Some(sqlString.length - 1)
+ )
+
+ // Execute the query recursively with isolated local variable context and EXECUTE IMMEDIATE
+ // origin. The isolation must cover parsing, analysis, and execution phases.
+ // CurrentOrigin.withOrigin ensures expressions created during parsing get the proper context
+ val result = withIsolatedLocalVariableContext {
+ CurrentOrigin.withOrigin(executeImmediateOrigin) {
+ // Parse and validate the query inside the isolation context to ensure
+ // local variables are not accessible during parsing
+ if (args.isEmpty) {
+ val parsedPlan = sparkSession.sessionState.sqlParser.parsePlan(sqlString)
Review Comment:
we never use this `parsedPlan`? just validation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433601047
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.SubstituteParamsParser
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This object consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = ParameterHandler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+object ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Performs parameter substitution and stores the position mapper in CurrentOrigin.
+ * The position mapper enables accurate error reporting by mapping positions in the
+ * substituted SQL back to the original SQL text.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Optimize: Skip parsing if no parameter markers are present in the SQL text.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ // Store the position mapper to enable accurate error position reporting in the original SQL.
+ val currentOrigin = CurrentOrigin.get
+ CurrentOrigin.set(currentOrigin.copy(positionMapper = Some(positionMapper)))
Review Comment:
I think it's safer to return the position mapper to the caller, and let caller to set up the `CurrentOrigin` before invoking the main parser, and restore it afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416818249
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,26 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
- throw SqlScriptingErrors.positionalParametersAreNotSupportedWithSqlScripting()
+ val parsedPlan = if (args.nonEmpty) {
+ // Set parameter context for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ ThreadLocalParameterContext.withContext(paramContext) {
+ val parsed = sessionState.sqlParser.parsePlan(sqlText)
Review Comment:
Techinically we can avoid using ThreadLocal:
- when creating the pre-parser, pass in the parameter context
- invoke the pre-parser explicitly here, and then invoke the actual parser.
We can't change the signature of the actual parser but the new pre-parser is a different story.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416652700
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
Review Comment:
is it always `startIndex + 2`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2418067359
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +53,132 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+ // Thread-local flag to track whether we're in a top-level parse operation
+ // This is used to prevent parameter substitution during identifier/data type parsing
+ private val isTopLevelParse = new ThreadLocal[Boolean] {
Review Comment:
Yes, we can!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425248128
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
+ *
+ * @param originalText
+ * The original SQL text with parameter markers
+ * @param substitutedText
+ * The SQL text after parameter substitution
+ * @param substitutions
+ * List of substitutions that were applied
+ */
+class PositionMapper(
+ val originalText: String,
+ val substitutedText: String,
+ val substitutions: List[Substitution]) {
Review Comment:
shall we add an assert to make sure the substitutions have no overlap?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
+ *
+ * @param originalText
+ * The original SQL text with parameter markers
+ * @param substitutedText
+ * The SQL text after parameter substitution
+ * @param substitutions
+ * List of substitutions that were applied
+ */
+class PositionMapper(
+ val originalText: String,
+ val substitutedText: String,
+ val substitutions: List[Substitution]) {
+
+ // Build sparse position ranges for efficient lookup
+ private val positionRanges = buildPositionRanges()
+
+ /**
+ * Map a position in the substituted text back to the original text. Uses binary search for
+ * O(log k) lookup time.
+ *
+ * @param substitutedPos
+ * Position in the substituted text
+ * @return
+ * Position in the original text, or the same position if no mapping exists
+ */
+ def mapToOriginal(substitutedPos: Int): Int = {
+ // Binary search for the range containing this position
+ positionRanges.find(range =>
+ substitutedPos >= range.substitutedStart && substitutedPos < range.substitutedEnd) match {
+ case Some(range) =>
+ // Position is within a substitution range
+ range.originalStart
+ case None =>
+ // Position is in an unmapped region - apply cumulative offset
+ val cumulativeOffset = positionRanges
+ .takeWhile(_.substitutedStart <= substitutedPos)
+ .map(_.offsetDelta)
+ .sum
+ substitutedPos + cumulativeOffset
+ }
+ }
+
+ /**
+ * Build sparse position ranges using functional approach. O(k) space complexity where k =
+ * number of substitutions.
+ *
+ * @example
+ * For original "SELECT :name, :age" -> substituted "SELECT 'John', 25":
+ * - Substitution(7, 12, "'John'") replaces ":name" with "'John'"
+ * - Substitution(14, 18, "25") replaces ":age" with "25"
+ *
+ * Creates PositionRanges:
+ * - Range for "'John'": substituted[7,13) -> original[7,12), offset=-1
+ * - Range for "25": substituted[15,17) -> original[14,18), offset=-3
+ *
+ * This allows mapping any position in "SELECT 'John', 25" back to "SELECT :name, :age".
+ */
+ private def buildPositionRanges(): List[PositionRange] = {
+ if (substitutions.isEmpty) {
+ return List.empty
+ }
+
+ val sortedSubstitutions = substitutions.sortBy(_.start)
Review Comment:
shall we add an assert to make sure the substitutions have no overlap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425614439
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Question mark is single character, so stopIndex = startIndex + 1
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers. This handles the namedParameterMarker case
+ * in the shared parameterMarker grammar rule. Note: The return value is not used; this method
+ * operates via side effects.
+ */
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers. This handles the QUESTION case in the
+ * shared parameterMarker grammar rule. Note: The return value is not used; this method operates
+ * via side effects.
+ */
+ override def visitPositionalParameterMarkerRule(
+ ctx: PositionalParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Parameter marker is single character, so stopIndex = startIndex + 1
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: ParameterStringValueContext =>
Review Comment:
do we need this case match? doesn't the default case `visitChildren(ruleNode)` work for it?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Question mark is single character, so stopIndex = startIndex + 1
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers. This handles the namedParameterMarker case
+ * in the shared parameterMarker grammar rule. Note: The return value is not used; this method
+ * operates via side effects.
+ */
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers. This handles the QUESTION case in the
+ * shared parameterMarker grammar rule. Note: The return value is not used; this method operates
+ * via side effects.
+ */
+ override def visitPositionalParameterMarkerRule(
+ ctx: PositionalParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Parameter marker is single character, so stopIndex = startIndex + 1
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: ParameterStringValueContext =>
+ // Handle parameter markers in string contexts
+ visit(ctx.parameterMarker())
+ case ctx: ParameterIntegerValueContext =>
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425615049
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Question mark is single character, so stopIndex = startIndex + 1
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers. This handles the namedParameterMarker case
+ * in the shared parameterMarker grammar rule. Note: The return value is not used; this method
+ * operates via side effects.
+ */
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers. This handles the QUESTION case in the
+ * shared parameterMarker grammar rule. Note: The return value is not used; this method operates
+ * via side effects.
+ */
+ override def visitPositionalParameterMarkerRule(
+ ctx: PositionalParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Parameter marker is single character, so stopIndex = startIndex + 1
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: ParameterStringValueContext =>
+ // Handle parameter markers in string contexts
+ visit(ctx.parameterMarker())
+ case ctx: ParameterIntegerValueContext =>
+ // Handle parameter markers in integer contexts
+ visit(ctx.parameterMarker())
+ case ctx: NamedParameterMarkerRuleContext =>
+ visitNamedParameterMarkerRule(ctx)
+ case ctx: PositionalParameterMarkerRuleContext =>
+ visitPositionalParameterMarkerRule(ctx)
+ case ctx: StringLiteralInContextContext =>
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426986967
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
Review Comment:
@srielau @cloud-fan Are we re-parsing the injected literal string or just using it for error message generation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429621958
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -57,6 +59,44 @@ case class Origin(
*/
trait WithOrigin {
def origin: Origin
+
+ /**
+ * Update query contexts in this object with translated positions. Uses reflection to
+ * generically update any object with query context.
+ */
+ def updateQueryContext(translator: Array[QueryContext] => Array[QueryContext]): WithOrigin = {
Review Comment:
where do we use it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2417840262
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
Review Comment:
It shoudl be:
val stopIndex = startIndex + 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2418188892
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
Review Comment:
We do, parameter markers are case sesnitive. Adding some tests to drive this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2374359814
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -40,6 +40,12 @@ options { tokenVocab = SqlBaseLexer; }
* When true, double quoted literals are identifiers rather than STRINGs.
*/
public boolean double_quoted_identifiers = false;
+
+ /**
+ * When false, parameter markers (? and :param) are only allowed in constant contexts.
+ * When true, parameter markers are allowed everywhere a literal is supported.
+ */
+ public boolean parameter_substitution_enabled = true;
}
Review Comment:
This PR does not touch the CREATE TABLE parser rule, how do we support parameter markers there? Or it's already supported before this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407224038
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +52,110 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+
+ // Thread-local flag to track whether we're in a top-level parse operation
+ // This is used to prevent parameter substitution during identifier/data type parsing
+ private val isTopLevelParse = new ThreadLocal[Boolean] {
+ override def initialValue(): Boolean = true
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ val wasTopLevel = isTopLevelParse.get()
+
+ // Step 1: Check if parameter substitution should occur
+ val (paramSubstituted, substitutionOccurred) =
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution
+ (command, false)
+ } else {
+ // Modern mode: check if we have a parameterized query context
+ org.apache.spark.sql.catalyst.parser.ThreadLocalParameterContext.get() match {
+ case Some(context) =>
+ val substituted = substituteParametersIfNeeded(command, context)
+ // Position mapper is stored by the parameter handler for later retrieval
+ (substituted, substituted != command) // Track if substitution actually occurred
+ case None =>
+ (command, false) // No parameters to substitute
+ }
+ }
+
+ // Step 2: Apply existing variable substitution
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Only set special origin if parameter substitution actually occurred
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val originToUse = if (substitutionOccurred && wasTopLevel) {
+ // Parameter substitution occurred - set original SQL text for position mapping
+ currentOrigin.copy(
+ sqlText = Some(command), // Use original SQL text, not substituted
+ startIndex = Some(0),
+ stopIndex = Some(command.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged
+ currentOrigin
+ }
+
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.withOrigin(originToUse) {
+ try {
+ super.parse(variableSubstituted)(toResult)
+ } finally {
+ // Clear parameter substitution callback after parsing is complete
+ if (substitutionOccurred && wasTopLevel) {
+ org.apache.spark.sql.catalyst.util.SparkParserUtils.clearParameterSubstitutionCallback()
+ }
+ }
+ }
}
+
+ private def substituteParametersIfNeeded(
+ command: String,
+ context: org.apache.spark.sql.catalyst.parser.ParameterContext): String = {
Review Comment:
Maybe we can import this so we don't have to use fully qualified name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407208939
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -55,6 +55,9 @@ object SQLExecution extends Logging {
private val testing = sys.props.contains(IS_TESTING.key)
+
+
+
Review Comment:
nit: empty lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407510281
##########
sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala:
##########
@@ -780,9 +767,9 @@ class ParametersSuite extends QueryTest with SharedSparkSession {
}
checkAnswer(spark.sql(query(":cte"), args = Map("cte" -> "t1")),
- Seq(Row(1), Row(2), Row(3), Row(4), Row(5)))
+ Array(Row(1), Row(2), Row(3), Row(4), Row(5)))
Review Comment:
Probably not.. Reverting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416659893
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers in string literal contexts. This handles
+ * the namedParameterMarker case added to the stringLit grammar rule. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterValue(ctx: NamedParameterValueContext): AnyRef =
Review Comment:
This looks almost the same with `visitNamedParameterLiteral`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407254114
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +52,110 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+
+ // Thread-local flag to track whether we're in a top-level parse operation
+ // This is used to prevent parameter substitution during identifier/data type parsing
+ private val isTopLevelParse = new ThreadLocal[Boolean] {
+ override def initialValue(): Boolean = true
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ val wasTopLevel = isTopLevelParse.get()
+
+ // Step 1: Check if parameter substitution should occur
+ val (paramSubstituted, substitutionOccurred) =
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution
+ (command, false)
+ } else {
+ // Modern mode: check if we have a parameterized query context
+ org.apache.spark.sql.catalyst.parser.ThreadLocalParameterContext.get() match {
+ case Some(context) =>
+ val substituted = substituteParametersIfNeeded(command, context)
+ // Position mapper is stored by the parameter handler for later retrieval
+ (substituted, substituted != command) // Track if substitution actually occurred
+ case None =>
+ (command, false) // No parameters to substitute
+ }
+ }
+
+ // Step 2: Apply existing variable substitution
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Only set special origin if parameter substitution actually occurred
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val originToUse = if (substitutionOccurred && wasTopLevel) {
+ // Parameter substitution occurred - set original SQL text for position mapping
+ currentOrigin.copy(
+ sqlText = Some(command), // Use original SQL text, not substituted
+ startIndex = Some(0),
+ stopIndex = Some(command.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged
+ currentOrigin
+ }
+
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.withOrigin(originToUse) {
Review Comment:
There are multiple places in code where fully qualified name is used instead of importing it. Maybe we can improve that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407216008
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +52,110 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+
Review Comment:
nit: empty line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416651266
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
Review Comment:
it's mutable ListBuffer, so we can do
```
val locations = namedParamLocations.getOrElseUpdate(
paramName,
scala.collection.mutable.ListBuffer[ParameterLocation]())
locations += ParameterLocation(...)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2417840262
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
Review Comment:
Well, it's a question mark....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436013087
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -430,6 +431,77 @@ class SparkSession private(
| Everything else |
* ----------------- */
+ /**
+ * Resolves and validates parameter expressions using a fake Project plan.
+ * This ensures parameters are properly resolved and constant-folded before being
+ * passed to the pre-parser.
+ *
+ * @param args Map of parameter names to expressions
+ * @return Map of parameter names to resolved literal expressions
+ */
+ private[sql] def resolveAndValidateParameters(
+ args: Map[String, Expression]): Map[String, Literal] = {
+ if (args.isEmpty) {
+ return Map.empty
+ }
+
+ // Create a fake Project plan: Project(parameters, OneRowRelation)
+ val aliases = args.map { case (name, expr) =>
+ Alias(expr, name)()
+ }.toSeq
+ val fakePlan = Project(aliases, OneRowRelation())
+
+ // Analyze and optimize to resolve and constant-fold
+ val analyzed = sessionState.analyzer.execute(fakePlan)
+ val optimized = sessionState.optimizer.execute(analyzed)
+
+ // Extract the resolved expressions from the project list
+ val resolvedParams = optimized.asInstanceOf[Project].projectList.map { alias =>
+ val resolvedExpr = alias.asInstanceOf[Alias].child
+
+ // Validate: the expression tree must only contain allowed expression types.
+ // This mirrors the validation in BindParameters.checkArgs.
+ // The validation happens on the analyzed (but not yet optimized) plan to catch
+ // unsupported functions like str_to_map before they get folded.
+ val analyzedExpr = analyzed.asInstanceOf[Project].projectList
+ .find(_.asInstanceOf[Alias].name == alias.name)
+ .get.asInstanceOf[Alias].child
+
+ def isNotAllowed(expr: Expression): Boolean = expr.exists {
+ case _: Literal | _: CreateArray | _: CreateNamedStruct |
+ _: CreateMap | _: MapFromArrays | _: MapFromEntries | _: VariableReference => false
+ case a: Alias => isNotAllowed(a.child)
+ case _ => true
+ }
+
+ if (isNotAllowed(analyzedExpr)) {
+ // Both modern and legacy modes use INVALID_SQL_ARG for sql() API argument validation.
+ // UNSUPPORTED_EXPR_FOR_PARAMETER is reserved for EXECUTE IMMEDIATE.
+ throw new AnalysisException(
+ errorClass = "INVALID_SQL_ARG",
+ messageParameters = Map("name" -> alias.name),
+ origin = analyzedExpr.origin)
+ }
+
+ // For non-literal allowed expressions, evaluate them to literals
+ val literal = resolvedExpr match {
+ case lit: Literal => lit
+ case foldable if foldable.foldable =>
Review Comment:
I don't think this is needed, optimizer should have done it already.
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -430,6 +431,77 @@ class SparkSession private(
| Everything else |
* ----------------- */
+ /**
+ * Resolves and validates parameter expressions using a fake Project plan.
+ * This ensures parameters are properly resolved and constant-folded before being
+ * passed to the pre-parser.
+ *
+ * @param args Map of parameter names to expressions
+ * @return Map of parameter names to resolved literal expressions
+ */
+ private[sql] def resolveAndValidateParameters(
+ args: Map[String, Expression]): Map[String, Literal] = {
+ if (args.isEmpty) {
+ return Map.empty
+ }
+
+ // Create a fake Project plan: Project(parameters, OneRowRelation)
+ val aliases = args.map { case (name, expr) =>
+ Alias(expr, name)()
+ }.toSeq
+ val fakePlan = Project(aliases, OneRowRelation())
+
+ // Analyze and optimize to resolve and constant-fold
+ val analyzed = sessionState.analyzer.execute(fakePlan)
+ val optimized = sessionState.optimizer.execute(analyzed)
+
+ // Extract the resolved expressions from the project list
+ val resolvedParams = optimized.asInstanceOf[Project].projectList.map { alias =>
+ val resolvedExpr = alias.asInstanceOf[Alias].child
Review Comment:
```suggestion
val optimizedExpr = alias.asInstanceOf[Alias].child
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2427704255
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
Review Comment:
We can whitelist the functions. I.e. Map and Array constructors.
I don't think it's that complicated it's scalar types and the transitive closure of struct, map, and array
One reason it got more complicated was because some tests allow for soem non standard constructores. But we could still whitelist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433588444
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -217,14 +225,36 @@ trait SparkParserUtils {
objectName: Option[String]): Origin = {
val startOpt = Option(startToken)
val stopOpt = Option(stopToken)
+
+ // Get the current origin to check for position mapper
+ val currentOrigin = CurrentOrigin.get
+
+ // Don't map positions yet - store them as-is along with the mapper
+ // Position mapping will be applied when creating query context for errors
+ val (startIndex, stopIndex, text, mapper) =
Review Comment:
`startIndex` and `stopIndex` are the same between these two branches, we don't need to put them here, but just `val (text, mapper) =`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429626986
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2200,7 +2205,15 @@ class AstBuilder extends DataTypeAstBuilder
Limit(expression(ctx.expression), query)
case ctx: SampleByPercentileContext =>
- val fraction = ctx.percentage.getText.toDouble
+ val fraction = if (ctx.DECIMAL_VALUE() != null) { ctx.DECIMAL_VALUE().getText.toDouble }
+ else {
+ // Assert that parameter markers have been substituted before reaching AstBuilder
+ assert(ctx.integerValue().getChild(0).getClass.getSimpleName !=
Review Comment:
we probably should add a method to get integer value after parameter replacement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429664562
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -320,25 +320,41 @@ class SparkConnectPlanner(
}
private def transformSql(sql: proto.SQL): LogicalPlan = {
+ val queryText = substituteParametersInQuery(sql)
+ // Parse the substituted query
+ parser.parsePlan(queryText)
Review Comment:
what's the exact reason we can't call `parsePlanWithParamContext` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429678719
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ val parsed = sessionState.sqlParser.parsePlanWithParameters(sqlText, paramContext)
Review Comment:
shall we call `parsePlan` if the control flag is off?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429675118
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala:
##########
@@ -187,7 +210,10 @@ case class ResolveExecuteImmediate(sparkSession: SparkSession, catalogManager: C
// Variable references should be evaluated to their values
varRef.eval(EmptyRow)
Review Comment:
can we use wrap it with `Literal` and use `varRef.dataType`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425606639
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
Review Comment:
A high level question: for parameter markers that already work today (in the expression parser rule), do we need to handle it in the pre-parser? Or we leave it and still let it be bound during analysis time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436958078
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala:
##########
@@ -37,7 +37,11 @@ class QueryPlanSuite extends SparkFunSuite {
CurrentOrigin.reset()
val mappedQuery = query mapExpressions {
- case _: Expression => Literal(1)
+ case _: Expression =>
+ // Reset CurrentOrigin to prevent contamination from previous operations
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375124530
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
+ */
+class ParameterHandler {
+
+
+ // Thread-local storage for position mapper to ensure thread safety
+ private val positionMapperStorage = new ThreadLocal[Option[PositionMapper]]() {
+ override def initialValue(): Option[PositionMapper] = None
+ }
+
+ /**
+ * Get the current position mapper for translating error positions.
+ * This mapper can translate positions from substituted text back to original text.
+ *
+ * @return The current position mapper, if available
+ */
+ def getPositionMapper: Option[PositionMapper] = positionMapperStorage.get()
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param rule The substitution rule to use
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ rule: SubstitutionRule,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ positionMapperStorage.set(Some(PositionMapper.identity(sqlText)))
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText, rule,
+ namedParams = namedParams, positionalParams = positionalParams)
+ positionMapperStorage.set(Some(positionMapper))
+ substituted
+ }
+
+ /**
+ * Translate error context from substituted text positions to original text positions.
+ *
+ * @param context The error context with positions in substituted text
+ * @return The error context with positions mapped to original text
+ */
+ def translateErrorContext(context: SQLQueryContext): SQLQueryContext = {
+ positionMapperStorage.get() match {
+ case Some(mapper) =>
+ val translatedStartIndex = context.originStartIndex.map(mapper.mapToOriginal)
+ val translatedStopIndex = context.originStopIndex.map(mapper.mapToOriginal)
+
+ context.copy(
+ originStartIndex = translatedStartIndex,
+ originStopIndex = translatedStopIndex,
+ sqlText = Some(mapper.originalText) // Use original text instead of substituted
+ )
+ case None =>
+ context // No position mapper available, return as-is
+ }
+ }
+
+ /**
+ * Substitute parameters in SQL text based on the parameter context.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param context The parameter context (named or positional)
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteParameters(
+ sqlText: String,
+ context: ParameterContext,
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ context match {
+ case NamedParameterContext(params) =>
+ performSubstitution(sqlText, rule, namedParams = params.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
Review Comment:
so we only allow literal expression now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2376040743
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -138,7 +165,7 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
}
} else {
val badType = typeCtx.unsupportedType.getText
- val params = typeCtx.INTEGER_VALUE().asScala.toList
+ val params = typeCtx.integerValue().asScala.map(_.getText).toList
Review Comment:
Yes, and there are visits in the main parser which assert that is teh case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2376034470
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -40,6 +40,12 @@ options { tokenVocab = SqlBaseLexer; }
* When true, double quoted literals are identifiers rather than STRINGs.
*/
public boolean double_quoted_identifiers = false;
+
+ /**
+ * When false, parameter markers (? and :param) are only allowed in constant contexts.
+ * When true, parameter markers are allowed everywhere a literal is supported.
+ */
+ public boolean parameter_substitution_enabled = true;
}
Review Comment:
Why would it need to? It covers the literals within.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436211687
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala:
##########
@@ -37,7 +37,11 @@ class QueryPlanSuite extends SparkFunSuite {
CurrentOrigin.reset()
val mappedQuery = query mapExpressions {
- case _: Expression => Literal(1)
+ case _: Expression =>
+ // Reset CurrentOrigin to prevent contamination from previous operations
Review Comment:
I have been battling this one for a week. In fact the contamination came in from ANOTHER SUITE!
And I was NOT able to repro locally.
Let me try again what happens when I remove this. Maybe the recent rework has fixed it.
I think part of teh problem is tyhat these APIs are too low level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2417993779
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ val identityMapper = PositionMapper.identity(sqlText)
+ setupSubstitutionContext(sqlText, sqlText, identityMapper, isIdentity = true)
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ setupSubstitutionContext(sqlText, substituted, positionMapper, isIdentity = false)
+ substituted
+ }
+
+ /**
+ * Set up position mapping context for error reporting.
+ * Creates a callback function and stores it in CurrentOrigin for position translation
+ * between original and substituted SQL text.
+ *
+ * @param originalSql The original SQL text before parameter substitution
+ * @param substitutedSql The SQL text after parameter substitution
+ * @param positionMapper The position mapper for translating error positions
+ * @param isIdentity Whether this is an identity mapping (no substitution occurred)
+ */
+ private[sql] def setupSubstitutionContext(
+ originalSql: String,
+ substitutedSql: String,
+ positionMapper: PositionMapper,
+ isIdentity: Boolean): Unit = {
+
+ // Create callback function for position mapping.
+ // The positionMapper is captured in the closure for efficient position translation.
+ val callback: (org.antlr.v4.runtime.Token, org.antlr.v4.runtime.Token, String,
Review Comment:
It turns out the solution is to move it all to sql-api. I hope that's OK.
@dtenedor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52334:
URL: https://github.com/apache/spark/pull/52334#issuecomment-3385934205
where do we perform the basic type checking? e.g. we can't replace a parameter marker of string litetal with an array value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426050190
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.trees.ParameterSubstitutionInfo
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ val identityMapper = PositionMapper.identity(sqlText)
+ setupSubstitutionContext(sqlText, sqlText, identityMapper, isIdentity = true)
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ setupSubstitutionContext(sqlText, substituted, positionMapper, isIdentity = false)
+ substituted
+ }
+
+ /**
+ * Set up parameter substitution context in CurrentOrigin for error reporting.
+ * This creates a ParameterSubstitutionInfo object with position mapping data.
+ *
+ * @param originalSql The original SQL text before substitution
+ * @param substitutedSql The SQL text after substitution
+ * @param positionMapper The mapper for translating positions
+ * @param isIdentity Whether this is an identity mapping (no substitution occurred)
+ */
+ private[sql] def setupSubstitutionContext(
+ originalSql: String,
+ substitutedSql: String,
+ positionMapper: PositionMapper,
+ isIdentity: Boolean): Unit = {
+
+ // Create parameter substitution info with the position mapper.
+ val substitutionInfo = ParameterSubstitutionInfo(
+ originalSql = originalSql,
+ isIdentity = isIdentity,
+ positionMapper = if (isIdentity) None else Some(positionMapper)
+ )
+
+ // Store the info in CurrentOrigin for position mapping during error reporting.
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val updatedOrigin = currentOrigin.copy(parameterSubstitutionInfo = Some(substitutionInfo))
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.set(updatedOrigin)
Review Comment:
can we use a new thread local? We need a thread local to keep the substitution info, so that the main parser is aware of it and can construct the `Origin` prorperly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426097137
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ val parsed = sessionState.sqlParser.parsePlanWithParameters(sqlText, paramContext)
Review Comment:
when shall we call `parsePlanWithParameters` and when shall we create `ParameterHandler`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426465415
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Question mark is single character, so stopIndex = startIndex + 1
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers. This handles the namedParameterMarker case
+ * in the shared parameterMarker grammar rule. Note: The return value is not used; this method
+ * operates via side effects.
+ */
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers. This handles the QUESTION case in the
+ * shared parameterMarker grammar rule. Note: The return value is not used; this method operates
+ * via side effects.
+ */
+ override def visitPositionalParameterMarkerRule(
+ ctx: PositionalParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Parameter marker is single character, so stopIndex = startIndex + 1
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: ParameterStringValueContext =>
+ // Handle parameter markers in string contexts
+ visit(ctx.parameterMarker())
+ case ctx: ParameterIntegerValueContext =>
+ // Handle parameter markers in integer contexts
+ visit(ctx.parameterMarker())
+ case ctx: NamedParameterMarkerRuleContext =>
+ visitNamedParameterMarkerRule(ctx)
+ case ctx: PositionalParameterMarkerRuleContext =>
+ visitPositionalParameterMarkerRule(ctx)
+ case ctx: StringLiteralInContextContext =>
+ // For string literals in context, continue traversing to find any nested parameters
+ visitChildren(ctx)
+ case ruleNode: RuleNode =>
+ // Continue traversing children for rule nodes
+ visitChildren(ruleNode)
+ case _ =>
+ // For other types (like terminal nodes), don't traverse children
+ null
+ }
+ }
+
+ /**
+ * Visit all children of a node to find parameters.
+ */
+ override def visitChildren(node: RuleNode): AnyRef = {
+ if (node == null) {
+ return null
+ }
+
+ var result: AnyRef = null
+ for (i <- 0 until node.getChildCount) {
+ val child = node.getChild(i)
+ if (child != null) {
+ val childResult = visit(child)
+ if (result == null) {
+ result = childResult
+ }
+ }
+ }
+ result
+ }
+
+ /**
+ * Visit terminal nodes (leaf nodes in the parse tree).
+ */
+ override def visitTerminal(node: TerminalNode): AnyRef = {
+ // Terminal nodes don't contain parameters themselves,
+ // but we need to handle them in the traversal
+ null
+ }
+}
+
+/**
+ * Data class to hold information about extracted parameters. This is defined here to make it
+ * available to the AST builder.
+ */
+case class ParameterInfo(namedParameters: Set[String], positionalParameters: List[Int]) {
Review Comment:
Nowhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426678273
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -106,6 +106,7 @@ class SparkConnectPlanner(
import sessionHolder.session.toRichColumn
private[connect] def parser = session.sessionState.sqlParser
+ private val parameterHandler = new ParameterHandler()
Review Comment:
The handler is stateless. What would we gain from creating new instances?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429719081
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
Review Comment:
shall we do it first before the new pre-parser?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416627129
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -45,16 +45,43 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
withOrigin(ctx)(StructType(visitColTypeList(ctx.colTypeList)))
}
- override def visitStringLit(ctx: StringLitContext): Token = {
- if (ctx != null) {
- if (ctx.STRING_LITERAL != null) {
- ctx.STRING_LITERAL.getSymbol
- } else {
- ctx.DOUBLEQUOTED_STRING.getSymbol
- }
- } else {
- null
- }
+ override def visitStringLiteralValue(ctx: StringLiteralValueContext): Token =
Review Comment:
So this data type parser ignores all parameter markers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2434342130
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
Review Comment:
poof
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433178006
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala:
##########
@@ -80,10 +81,8 @@ class QueryExecution(
try {
analyzed
} catch {
- case e: AnalysisException =>
- // Because we do eager analysis for Dataframe, there will be no execution created after
- // AnalysisException occurs. So we need to explicitly create a new execution to post
- // start/end events to notify the listener and UI components.
+ case e: Throwable with WithOrigin =>
Review Comment:
No idea, removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433188195
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Set up origin with original SQL text for parameter-aware error reporting.
+ val currentOrigin = CurrentOrigin.get
+ val originToUse = if ((substitutionOccurred || hasParameters) && isTopLevel) {
+ // Set up origin with the substituted SQL text for proper error reporting
+ currentOrigin.copy(
+ sqlText = Some(variableSubstituted), // Use substituted SQL text
+ startIndex = Some(0),
+ stopIndex = Some(variableSubstituted.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged.
+ currentOrigin
+ }
+
+ CurrentOrigin.withOrigin(originToUse) {
+ super.parse(variableSubstituted)(toResult)
+ }
+ }
+
+ private def substituteParametersOrSetupCallback(
Review Comment:
Outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436025457
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -488,15 +573,36 @@ class SparkSession private(
args: Map[String, Any],
tracker: QueryPlanningTracker): DataFrame =
withActive {
- val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- NameParameterizedQuery(parsedPlan, args.transform((_, v) => lit(v).expr))
- } else {
- parsedPlan
+ // Always set parameter context if we have actual parameters
+ if (args.nonEmpty) {
+ // Resolve and validate parameters first
+ val resolvedParams = resolveAndValidateParameters(args.transform((_, v) => lit(v).expr))
+ val paramContext = NamedParameterContext(resolvedParams)
+ val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
+ val parsedPlan = sessionState.sqlParser.parsePlanWithParameters(sqlText, paramContext)
+ // In legacy mode, wrap the parsed plan with NameParameterizedQuery
+ // so that the BindParameters analyzer rule can bind the parameters
+ if (sessionState.conf.legacyParameterSubstitutionConstantsOnly) {
+ NameParameterizedQuery(parsedPlan, paramContext.params)
+ } else {
+ parsedPlan
+ }
+ }
+
+ Dataset.ofRows(self, plan, tracker)
+ } else {
+ // Reset position mapper to prevent contamination from previous parameterized sql() calls
Review Comment:
shall we do this reset in all the `def sql` overloads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375139431
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
+ */
+class ParameterHandler {
+
+
+ // Thread-local storage for position mapper to ensure thread safety
+ private val positionMapperStorage = new ThreadLocal[Option[PositionMapper]]() {
+ override def initialValue(): Option[PositionMapper] = None
+ }
+
+ /**
+ * Get the current position mapper for translating error positions.
+ * This mapper can translate positions from substituted text back to original text.
+ *
+ * @return The current position mapper, if available
+ */
+ def getPositionMapper: Option[PositionMapper] = positionMapperStorage.get()
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param rule The substitution rule to use
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ rule: SubstitutionRule,
Review Comment:
are we going to extend it in the future, or a boolean `isScript` flag is sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375188515
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,26 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
- throw SqlScriptingErrors.positionalParametersAreNotSupportedWithSqlScripting()
+ val parsedPlan = if (args.nonEmpty) {
+ // Set parameter context for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ ThreadLocalParameterContext.withContext(paramContext) {
+ val parsed = sessionState.sqlParser.parsePlan(sqlText)
Review Comment:
how do we invoke the pre-parser?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375084336
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Create a named parameter which represents a literal with a non-bound value and unknown type.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): String =
Review Comment:
does the return value matter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375185512
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ * @since 4.0.0
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ // Note: array, map, struct are naturally foldable but reach here as UnresolvedFunction
+ // map_from_arrays and map_from_entries are not naturally foldable
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ // Modern parameter substitution supports: legacy whitelist + foldable expressions
+ // Functions like str_to_map are intentionally not supported
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: Exception =>
+ // Fallback to manual conversion for cases where Literal.sql doesn't work
+ // (e.g., complex types that need constructor syntax)
+ lit match {
+ case Literal(null, _) => "NULL"
+ case Literal(value, dataType) => dataType match {
+ case ArrayType(elementType, _) => convertArrayLiteral(value, elementType)
+ case MapType(keyType, valueType, _) => convertMapLiteral(value, keyType, valueType)
+ case _: StructType =>
+ // Struct literals (row values) - convert to ROW constructor
+ value match {
+ case row: org.apache.spark.sql.catalyst.InternalRow =>
+ val structType = dataType.asInstanceOf[StructType]
+ val fieldValues = (0 until row.numFields).map { i =>
+ if (row.isNullAt(i)) {
+ "NULL"
+ } else {
+ val fieldType = structType.fields(i).dataType
+ val fieldValue = row.get(i, fieldType)
+ val fieldLiteral = Literal.create(fieldValue, fieldType)
+ convert(fieldLiteral)
+ }
+ }
+ s"ROW(${fieldValues.mkString(", ")})"
+ case _ => s"ROW(${value.toString})"
+ }
+ case _ =>
+ // For any other unsupported type, fall back to string representation
+ s"'${value.toString.replace("'", "''")}'"
+ }
+ }
+ }
+ }
+
+ private def convertArrayLiteral(value: Any, elementType: DataType): String = {
Review Comment:
how is this different from `Literal#sql`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2388793291
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterContext.scala:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * Context for passing parameter values to the SQL parser.
+ * This allows parameter substitution to happen during the parsing phase
+ * rather than during analysis.
+ */
+sealed trait ParameterContext
+
+/**
+ * Context for named parameters (e.g., :paramName).
+ *
+ * @param params Map of parameter names to their expression values
+ */
+case class NamedParameterContext(params: Map[String, Expression]) extends ParameterContext
+
+/**
+ * Context for positional parameters (e.g., ?).
+ *
+ * @param params Sequence of expression values in order
+ */
+case class PositionalParameterContext(params: Seq[Expression]) extends ParameterContext
+
+/**
+ * Context that supports both named and positional parameters.
+ * This is used by EXECUTE IMMEDIATE where the parameter type is determined by the inner query.
+ *
+ * @param args Raw argument values from USING clause (already evaluated)
+ * @param paramNames Parameter names from USING clause (empty strings for positional)
+ */
+case class HybridParameterContext(
+ args: Array[_],
+ paramNames: Array[String]) extends ParameterContext
+
+/**
+ * Thread-local storage for parameter context to pass parameters
+ * from SparkSession to the parser without changing method signatures.
+ */
+object ThreadLocalParameterContext {
+ private val context = new ThreadLocal[Option[ParameterContext]]()
Review Comment:
Cursor sez: yes... :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor closed pull request #52334: [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling
URL: https://github.com/apache/spark/pull/52334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429590259
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -45,18 +45,39 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
withOrigin(ctx)(StructType(visitColTypeList(ctx.colTypeList)))
}
- override def visitStringLit(ctx: StringLitContext): Token = {
- if (ctx != null) {
- if (ctx.STRING_LITERAL != null) {
- ctx.STRING_LITERAL.getSymbol
- } else {
- ctx.DOUBLEQUOTED_STRING.getSymbol
- }
- } else {
- null
- }
+ override def visitStringLiteralValue(ctx: StringLiteralValueContext): Token =
+ Option(ctx).map(_.STRING_LITERAL.getSymbol).orNull
+
+ override def visitDoubleQuotedStringLiteralValue(
+ ctx: DoubleQuotedStringLiteralValueContext): Token =
+ Option(ctx).map(_.DOUBLEQUOTED_STRING.getSymbol).orNull
+
+ override def visitIntegerVal(ctx: IntegerValContext): Token =
+ Option(ctx).map(_.INTEGER_VALUE.getSymbol).orNull
+
+ override def visitStringLiteralInContext(ctx: StringLiteralInContextContext): Token = {
+ visit(ctx.stringLitWithoutMarker).asInstanceOf[Token]
+ }
+
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): Token = {
+ // This should be unreachable due to grammar-level blocking of parameter markers
+ // when legacy parameter substitution is enabled
+ QueryParsingErrors.unexpectedUseOfParameterMarker(ctx)
+ }
+
+ override def visitPositionalParameterMarkerRule(
Review Comment:
shall we override `visitNamedParameterLiteral`/`visitPosParameterLiteral` here too and fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425751788
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala:
##########
@@ -90,15 +90,17 @@ abstract class AbstractSqlParser extends AbstractParser with ParserInterface {
}
/** Creates LogicalPlan for a given SQL string. */
- override def parsePlan(sqlText: String): LogicalPlan = parse(sqlText) { parser =>
- val ctx = parser.compoundOrSingleStatement()
- withErrorHandling(ctx, Some(sqlText)) {
- astBuilder.visitCompoundOrSingleStatement(ctx) match {
- case compoundBody: CompoundPlanStatement => compoundBody
- case plan: LogicalPlan => plan
- case _ =>
- val position = Origin(None, None)
- throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ override def parsePlan(sqlText: String): LogicalPlan = {
+ parse(sqlText) { parser =>
Review Comment:
let's revert the code style change. `parse(sqlText)` should be in the same line of `def parsePlan`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433347194
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
Review Comment:
outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2430142817
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
Review Comment:
We also need to resolve the map from array functions etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416646833
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
Review Comment:
do we need to consider case sensitivity for named parameters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416657879
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -1593,6 +1602,12 @@ number
| MINUS? BIGDECIMAL_LITERAL #bigDecimalLiteral
;
+integerValue
+ : INTEGER_VALUE #integerVal
+ | {parameter_substitution_enabled}? namedParameterMarker #namedParameterIntegerValue
+ | {parameter_substitution_enabled}? QUESTION #positionalParameterIntegerValue
Review Comment:
can we add a shared parser rule for parameter markers? then we don't need two similar visit methods for parameter markers?
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -1593,6 +1602,12 @@ number
| MINUS? BIGDECIMAL_LITERAL #bigDecimalLiteral
;
+integerValue
+ : INTEGER_VALUE #integerVal
+ | {parameter_substitution_enabled}? namedParameterMarker #namedParameterIntegerValue
+ | {parameter_substitution_enabled}? QUESTION #positionalParameterIntegerValue
Review Comment:
can we add a shared parser rule for parameter markers? then we don't need two similar visit methods for parameter markers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416627129
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -45,16 +45,43 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
withOrigin(ctx)(StructType(visitColTypeList(ctx.colTypeList)))
}
- override def visitStringLit(ctx: StringLitContext): Token = {
- if (ctx != null) {
- if (ctx.STRING_LITERAL != null) {
- ctx.STRING_LITERAL.getSymbol
- } else {
- ctx.DOUBLEQUOTED_STRING.getSymbol
- }
- } else {
- null
- }
+ override def visitStringLiteralValue(ctx: StringLiteralValueContext): Token =
Review Comment:
So this data type parser ignores all parameter markers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #52334:
URL: https://github.com/apache/spark/pull/52334#issuecomment-3387259575
> where do we perform the basic type checking? e.g. we can't replace a parameter marker of string litetal with an array value.
Type checking happens in the main parser. All we do in the pre-parser is to ensure we have "coverage". That is hit all the places "any" literals go. Then leave it to teh main parser to check legality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2418049361
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -187,25 +187,93 @@ trait SparkParserUtils {
* Register the origin of the context. Any TreeNode created in the closure will be assigned the
* registered origin. This method restores the previously set origin after completion of the
* closure.
+ *
+ * This method is parameter substitution-aware. If parameter substitution occurred before
+ * parsing, it will automatically adjust the positions and SQL text to refer to the original SQL
+ * (before substitution) instead of the substituted SQL.
*/
def withOrigin[T](ctx: ParserRuleContext, sqlText: Option[String] = None)(f: => T): T = {
val current = CurrentOrigin.get
val text = sqlText.orElse(current.sqlText)
+
if (text.isEmpty) {
CurrentOrigin.set(position(ctx.getStart))
} else {
- CurrentOrigin.set(
- positionAndText(
- ctx.getStart,
- ctx.getStop,
- text.get,
- current.objectType,
- current.objectName))
+ // Check if parameter substitution occurred and adjust origin accordingly.
+ val adjustedOrigin = adjustOriginForParameterSubstitution(
+ ctx.getStart,
+ ctx.getStop,
+ text.get,
+ current.objectType,
+ current.objectName)
+
+ // Preserve any existing callback when setting the new origin.
+ val finalOrigin = if (current.parameterSubstitutionCallback.isDefined) {
+ adjustedOrigin.copy(parameterSubstitutionCallback = current.parameterSubstitutionCallback)
+ } else {
+ adjustedOrigin
+ }
+
+ CurrentOrigin.set(finalOrigin)
}
try {
f
} finally {
- CurrentOrigin.set(current)
+ // When restoring origin, preserve any callback that was added during parsing.
+ val currentAfterParsing = CurrentOrigin.get
+ val originToRestore =
+ if (currentAfterParsing.parameterSubstitutionCallback.isDefined ||
+ current.parameterSubstitutionCallback.isDefined) {
+ // Either the current or the original has a callback - preserve it.
+ val callbackToPreserve = currentAfterParsing.parameterSubstitutionCallback
+ .orElse(current.parameterSubstitutionCallback)
+ current.copy(parameterSubstitutionCallback = callbackToPreserve)
+ } else {
+ // Neither has a callback - restore as normal.
+ current
+ }
+
+ CurrentOrigin.set(originToRestore)
+ }
+ }
+
+ /**
+ * Adjust origin information to account for parameter substitution.
+ *
+ * If parameter substitution occurred, this method maps positions from the substituted SQL back
+ * to the original SQL and uses the original SQL text in the origin.
+ *
+ * @param startToken
+ * The start token from the substituted SQL
+ * @param stopToken
+ * The stop token from the substituted SQL
+ * @param substitutedSql
+ * The SQL text after substitution
+ * @param objectType
+ * The object type for the origin
+ * @param objectName
+ * The object name for the origin
+ * @return
+ * Origin with positions and text adjusted for parameter substitution
+ */
+ private def adjustOriginForParameterSubstitution(
+ startToken: Token,
+ stopToken: Token,
+ substitutedSql: String,
+ objectType: Option[String],
+ objectName: Option[String]): Origin = {
+
+ // Try to get parameter substitution callback from CurrentOrigin.
+ CurrentOrigin.get.parameterSubstitutionCallback match {
+ case Some(callback) =>
+ // Cast the callback from Any to the proper type.
+ val typedCallback = callback.asInstanceOf[SparkParserUtils.ParameterSubstitutionCallback]
Review Comment:
Outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407283738
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +52,110 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+
+ // Thread-local flag to track whether we're in a top-level parse operation
+ // This is used to prevent parameter substitution during identifier/data type parsing
+ private val isTopLevelParse = new ThreadLocal[Boolean] {
+ override def initialValue(): Boolean = true
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ val wasTopLevel = isTopLevelParse.get()
+
+ // Step 1: Check if parameter substitution should occur
+ val (paramSubstituted, substitutionOccurred) =
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution
+ (command, false)
+ } else {
+ // Modern mode: check if we have a parameterized query context
+ org.apache.spark.sql.catalyst.parser.ThreadLocalParameterContext.get() match {
+ case Some(context) =>
+ val substituted = substituteParametersIfNeeded(command, context)
+ // Position mapper is stored by the parameter handler for later retrieval
+ (substituted, substituted != command) // Track if substitution actually occurred
+ case None =>
+ (command, false) // No parameters to substitute
+ }
+ }
+
+ // Step 2: Apply existing variable substitution
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Only set special origin if parameter substitution actually occurred
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val originToUse = if (substitutionOccurred && wasTopLevel) {
+ // Parameter substitution occurred - set original SQL text for position mapping
+ currentOrigin.copy(
+ sqlText = Some(command), // Use original SQL text, not substituted
+ startIndex = Some(0),
+ stopIndex = Some(command.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged
+ currentOrigin
+ }
+
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.withOrigin(originToUse) {
+ try {
+ super.parse(variableSubstituted)(toResult)
+ } finally {
+ // Clear parameter substitution callback after parsing is complete
+ if (substitutionOccurred && wasTopLevel) {
+ org.apache.spark.sql.catalyst.util.SparkParserUtils.clearParameterSubstitutionCallback()
+ }
+ }
+ }
}
+
+ private def substituteParametersIfNeeded(
+ command: String,
+ context: org.apache.spark.sql.catalyst.parser.ParameterContext): String = {
+
+ // Check legacy config - if parameter substitution limited to constants only, return original
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ command
+ } else {
+ parameterHandler.substituteParameters(command, context)
+ }
+ }
+
+ // Override parsing methods that should NOT use parameter substitution
+ // These methods parse identifiers and data types where parameters don't make sense
Review Comment:
Most comments in Spark follow the convention of using complete sentences and ending them with a punctuation. We should try to maintain that consistency here. Relevant discussion: https://github.com/apache/spark/pull/48950#discussion_r1857421840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2388762663
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Create a named parameter which represents a literal with a non-bound value and unknown type.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): String =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) +=
+ ParameterLocation(startIndex, stopIndex)
+
+ paramName
+ }
+
+ /**
+ * Create a positional parameter which represents a literal with a non-bound value and unknown
+ * type.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): String =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ "?"
+ }
+
+ /**
+ * Handle namedParameterValue context for parameter markers in string literal contexts. This
+ * handles the namedParameterMarker case added to the stringLit grammar rule.
+ */
+ override def visitNamedParameterValue(ctx: NamedParameterValueContext): String =
Review Comment:
Yes they are, but we can't share the visits due to the way antlr works. Given how short they are, I'd prefer to keep the duplication and later we can remove the old one with teh old way of doping parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2421338947
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
Review Comment:
you can reverse the logic and throw the exception, skipping the 'else' branch
```
if (!expr.foldable) {
throw SparkException.internalError(
s"LiteralToSqlConverter cannot convert non-foldable expression: " +
s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
s"literals before conversion.")
}
val value = expr.eval()
val dataType = expr.dataType
convertLiteral(Literal.create(value, dataType))
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
Review Comment:
There are too many levels of indentation below, we need to simplify it. I'd do the following:
* Do the `lit.sql` attempt outside of this function, catch the MatchError, and then put L129-L157 into the helper function instead like `convertLiteralManualConversion`
* Move the data type matches starting on L134 into the parent match, like
```
case Literal(value, ArrayType(elementType, _)) =>
convertArrayLiteral(value, elementType)
```
* Flatten out the match on L152 in the same way, like
```
case Literal(row: InternalRow, structType @ StructType(elementType, _)) =>
val fieldValues = (0 until row.numFields).map { i =>
...
case Literal(value, StructType(elementType, _)) =>
s"ROW(${value.toString})"
```
At the end it should just be one list of patterns against `lit` with no nested pattern matches.
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
+ *
+ * @param originalText
+ * The original SQL text with parameter markers
+ * @param substitutedText
+ * The SQL text after parameter substitution
+ * @param substitutions
+ * List of substitutions that were applied
+ */
+class PositionMapper(
+ val originalText: String,
+ val substitutedText: String,
+ val substitutions: List[Substitution]) {
+
+ // Build sparse position ranges for efficient lookup
+ private val positionRanges = buildPositionRanges()
+
+ /**
+ * Map a position in the substituted text back to the original text. Uses binary search for
+ * O(log k) lookup time.
+ *
+ * @param substitutedPos
+ * Position in the substituted text
+ * @return
+ * Position in the original text, or the same position if no mapping exists
+ */
+ def mapToOriginal(substitutedPos: Int): Int = {
+ // Binary search for the range containing this position
+ positionRanges.find(range =>
+ substitutedPos >= range.substitutedStart && substitutedPos < range.substitutedEnd) match {
+ case Some(range) =>
+ // Position is within a substitution range
+ range.originalStart
+ case None =>
+ // Position is in an unmapped region - apply cumulative offset
+ val cumulativeOffset = positionRanges
+ .takeWhile(_.substitutedStart <= substitutedPos)
+ .map(_.offsetDelta)
+ .sum
+ substitutedPos + cumulativeOffset
+ }
+ }
+
+ /**
+ * Build sparse position ranges using functional approach. O(k) space complexity where k =
+ * number of substitutions.
Review Comment:
can we possibly give a brief example here, it could help the reader understand :)
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: MatchError =>
+ // Fallback to manual conversion for cases where Literal.sql doesn't have
+ // a pattern match for the specific (value, dataType) combination
+ lit match {
+ case Literal(null, _) => "NULL"
+ case Literal(value, dataType) => dataType match {
+ case ArrayType(elementType, _) => convertArrayLiteral(value, elementType)
+ case MapType(keyType, valueType, _) => convertMapLiteral(value, keyType, valueType)
+ case _: StructType =>
+ // Struct literals (row values) - convert to ROW constructor
+ value match {
+ case row: InternalRow =>
+ val structType = dataType.asInstanceOf[StructType]
+ val fieldValues = (0 until row.numFields).map { i =>
+ if (row.isNullAt(i)) {
+ "NULL"
+ } else {
+ val fieldType = structType.fields(i).dataType
+ val fieldValue = row.get(i, fieldType)
+ val fieldLiteral = Literal.create(fieldValue, fieldType)
+ convert(fieldLiteral)
+ }
+ }
+ s"ROW(${fieldValues.mkString(", ")})"
+ case _ => s"ROW(${value.toString})"
+ }
+ case _ =>
+ // For any other unsupported type, fall back to string representation
+ s"'${value.toString.replace("'", "''")}'"
+ }
+ }
+ }
+ }
+
+ private def convertArrayLiteral(value: Any, elementType: DataType): String = {
+ if (value == null) "NULL"
+ else {
Review Comment:
```suggestion
} else {
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
+ val ctx = try {
+ // First attempt: SLL mode with bail error strategy
+ parser.setErrorHandler(new SparkParserBailErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ } catch {
+ case e: ParseCancellationException =>
+ // Second attempt: LL mode with full error strategy
+ tokenStream.seek(0) // rewind input stream
+ parser.reset()
+ parser.setErrorHandler(new SparkParserErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ parser.compoundOrSingleStatement()
+ }
+ val parameterLocations = astBuilder.extractParameterLocations(ctx)
+
+ // Substitute parameters in the original text
+ val (substitutedSql, appliedSubstitutions) = substituteAtLocations(sqlText, parameterLocations,
+ namedParams, positionalParams)
+ val consumedPositionalParams = parameterLocations.positionalParameterLocations.length
+
+ // Create position mapper for error context translation
+ val positionMapper = PositionMapper(sqlText, substitutedSql, appliedSubstitutions)
+
+ (substitutedSql, consumedPositionalParams, positionMapper)
+ }
+
+ /**
+ * Detects parameter markers in SQL text without performing substitution.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text to analyze
+ * @return A tuple of (hasPositionalParameters, hasNamedParameters)
+ */
+ def detectParameters(sqlText: String): (Boolean, Boolean) = {
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (false, false)
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
Review Comment:
same here, can we dedup it
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
Review Comment:
fix indentation to start at the same column as the params above?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: MatchError =>
+ // Fallback to manual conversion for cases where Literal.sql doesn't have
+ // a pattern match for the specific (value, dataType) combination
+ lit match {
+ case Literal(null, _) => "NULL"
+ case Literal(value, dataType) => dataType match {
+ case ArrayType(elementType, _) => convertArrayLiteral(value, elementType)
+ case MapType(keyType, valueType, _) => convertMapLiteral(value, keyType, valueType)
+ case _: StructType =>
+ // Struct literals (row values) - convert to ROW constructor
+ value match {
+ case row: InternalRow =>
+ val structType = dataType.asInstanceOf[StructType]
+ val fieldValues = (0 until row.numFields).map { i =>
+ if (row.isNullAt(i)) {
+ "NULL"
+ } else {
+ val fieldType = structType.fields(i).dataType
+ val fieldValue = row.get(i, fieldType)
+ val fieldLiteral = Literal.create(fieldValue, fieldType)
+ convert(fieldLiteral)
+ }
+ }
+ s"ROW(${fieldValues.mkString(", ")})"
+ case _ => s"ROW(${value.toString})"
+ }
+ case _ =>
+ // For any other unsupported type, fall back to string representation
+ s"'${value.toString.replace("'", "''")}'"
+ }
+ }
+ }
+ }
+
+ private def convertArrayLiteral(value: Any, elementType: DataType): String = {
+ if (value == null) "NULL"
Review Comment:
```suggestion
if (value == null) {
"NULL"
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: MatchError =>
+ // Fallback to manual conversion for cases where Literal.sql doesn't have
+ // a pattern match for the specific (value, dataType) combination
+ lit match {
+ case Literal(null, _) => "NULL"
+ case Literal(value, dataType) => dataType match {
+ case ArrayType(elementType, _) => convertArrayLiteral(value, elementType)
+ case MapType(keyType, valueType, _) => convertMapLiteral(value, keyType, valueType)
+ case _: StructType =>
+ // Struct literals (row values) - convert to ROW constructor
+ value match {
+ case row: InternalRow =>
+ val structType = dataType.asInstanceOf[StructType]
+ val fieldValues = (0 until row.numFields).map { i =>
+ if (row.isNullAt(i)) {
+ "NULL"
+ } else {
+ val fieldType = structType.fields(i).dataType
+ val fieldValue = row.get(i, fieldType)
+ val fieldLiteral = Literal.create(fieldValue, fieldType)
+ convert(fieldLiteral)
+ }
+ }
+ s"ROW(${fieldValues.mkString(", ")})"
+ case _ => s"ROW(${value.toString})"
+ }
+ case _ =>
+ // For any other unsupported type, fall back to string representation
+ s"'${value.toString.replace("'", "''")}'"
+ }
+ }
+ }
+ }
+
+ private def convertArrayLiteral(value: Any, elementType: DataType): String = {
+ if (value == null) "NULL"
+ else {
+ // Handle both Scala collections and Spark's GenericArrayData
+ val arraySeq = value match {
+ case gad: GenericArrayData =>
+ gad.array.toSeq
+ case seq: scala.collection.Seq[Any] =>
+ seq
+ case arr: Array[Any] =>
+ arr.toSeq
+ case other =>
+ // Fallback: try to convert to array and then to sequence
+ Array(other).toSeq
+ }
+ val elementStrs = arraySeq.map { elem =>
+ convert(Literal.create(elem, elementType))
+ }
+ s"ARRAY(${elementStrs.mkString(", ")})"
+ }
+ }
+
+ private def convertMapLiteral(value: Any, keyType: DataType, valueType: DataType): String = {
+ if (value == null) "NULL"
+ else {
Review Comment:
```suggestion
} else {
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
Review Comment:
Is this code copy/pasted from [1]? If so, can we dedup it into a refactored shared helper from all locations?
[1] https://github.com/apache/spark/blob/6eb4d3c9d38f6849b0acfcffdbadce03c8f49ac6/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/parsers.scala#L70
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -19,9 +19,20 @@ package org.apache.spark.sql.catalyst.trees
import java.util.regex.Pattern
import org.apache.spark.QueryContext
+import org.apache.spark.sql.catalyst.parser.PositionMapper
import org.apache.spark.sql.internal.SqlApiConf
import org.apache.spark.util.ArrayImplicits._
+/**
+ * Information needed for parameter substitution position mapping. Stored directly in Origin to
+ * avoid callback complexity.
Review Comment:
👍 👍 👍 👍 👍 👍 👍 👍 👍
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
Review Comment:
super super nit: can we add punctuation at the end of all comments in this PR to match elsewhere in the codebase. 🙏 🙏
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
+ *
+ * @param originalText
+ * The original SQL text with parameter markers
+ * @param substitutedText
+ * The SQL text after parameter substitution
+ * @param substitutions
+ * List of substitutions that were applied
+ */
+class PositionMapper(
+ val originalText: String,
+ val substitutedText: String,
+ val substitutions: List[Substitution]) {
+
+ // Build sparse position ranges for efficient lookup
+ private val positionRanges = buildPositionRanges()
+
+ /**
+ * Map a position in the substituted text back to the original text. Uses binary search for
+ * O(log k) lookup time.
+ *
+ * @param substitutedPos
+ * Position in the substituted text
+ * @return
+ * Position in the original text, or the same position if no mapping exists
+ */
+ def mapToOriginal(substitutedPos: Int): Int = {
+ // Binary search for the range containing this position
+ positionRanges.find(range =>
+ substitutedPos >= range.substitutedStart && substitutedPos < range.substitutedEnd) match {
+ case Some(range) =>
+ // Position is within a substitution range
+ range.originalStart
+ case None =>
+ // Position is in an unmapped region - apply cumulative offset
+ val cumulativeOffset = positionRanges
+ .takeWhile(_.substitutedStart <= substitutedPos)
+ .map(_.offsetDelta)
+ .sum
+ substitutedPos + cumulativeOffset
+ }
+ }
+
+ /**
+ * Build sparse position ranges using functional approach. O(k) space complexity where k =
+ * number of substitutions.
+ */
+ private def buildPositionRanges(): List[PositionRange] = {
+ if (substitutions.isEmpty) return List.empty
Review Comment:
```suggestion
if (substitutions.isEmpty) {
return List.empty
}
```
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
Review Comment:
let's just `import scala.collection.mutable` up top so you can just write `mutable.Set`, `mutable.ListBuffer`, etc. Then these can all be one-liners
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: MatchError =>
+ // Fallback to manual conversion for cases where Literal.sql doesn't have
+ // a pattern match for the specific (value, dataType) combination
+ lit match {
+ case Literal(null, _) => "NULL"
+ case Literal(value, dataType) => dataType match {
+ case ArrayType(elementType, _) => convertArrayLiteral(value, elementType)
+ case MapType(keyType, valueType, _) => convertMapLiteral(value, keyType, valueType)
+ case _: StructType =>
+ // Struct literals (row values) - convert to ROW constructor
+ value match {
+ case row: InternalRow =>
+ val structType = dataType.asInstanceOf[StructType]
+ val fieldValues = (0 until row.numFields).map { i =>
+ if (row.isNullAt(i)) {
+ "NULL"
+ } else {
+ val fieldType = structType.fields(i).dataType
+ val fieldValue = row.get(i, fieldType)
+ val fieldLiteral = Literal.create(fieldValue, fieldType)
+ convert(fieldLiteral)
+ }
+ }
+ s"ROW(${fieldValues.mkString(", ")})"
+ case _ => s"ROW(${value.toString})"
+ }
+ case _ =>
+ // For any other unsupported type, fall back to string representation
+ s"'${value.toString.replace("'", "''")}'"
+ }
+ }
+ }
+ }
+
+ private def convertArrayLiteral(value: Any, elementType: DataType): String = {
+ if (value == null) "NULL"
+ else {
+ // Handle both Scala collections and Spark's GenericArrayData
+ val arraySeq = value match {
+ case gad: GenericArrayData =>
+ gad.array.toSeq
+ case seq: scala.collection.Seq[Any] =>
+ seq
+ case arr: Array[Any] =>
+ arr.toSeq
+ case other =>
+ // Fallback: try to convert to array and then to sequence
+ Array(other).toSeq
+ }
+ val elementStrs = arraySeq.map { elem =>
+ convert(Literal.create(elem, elementType))
+ }
+ s"ARRAY(${elementStrs.mkString(", ")})"
+ }
+ }
+
+ private def convertMapLiteral(value: Any, keyType: DataType, valueType: DataType): String = {
+ if (value == null) "NULL"
Review Comment:
```suggestion
if (value == null) {
"NULL"
```
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -187,25 +187,108 @@ trait SparkParserUtils {
* Register the origin of the context. Any TreeNode created in the closure will be assigned the
* registered origin. This method restores the previously set origin after completion of the
* closure.
+ *
+ * This method is parameter substitution-aware. If parameter substitution occurred before
+ * parsing, it will automatically adjust the positions and SQL text to refer to the original SQL
+ * (before substitution) instead of the substituted SQL.
*/
def withOrigin[T](ctx: ParserRuleContext, sqlText: Option[String] = None)(f: => T): T = {
val current = CurrentOrigin.get
val text = sqlText.orElse(current.sqlText)
+
if (text.isEmpty) {
CurrentOrigin.set(position(ctx.getStart))
} else {
- CurrentOrigin.set(
- positionAndText(
- ctx.getStart,
- ctx.getStop,
- text.get,
- current.objectType,
- current.objectName))
+ // Check if parameter substitution occurred and adjust origin accordingly.
+ val adjustedOrigin = adjustOriginForParameterSubstitution(
+ ctx.getStart,
+ ctx.getStop,
+ text.get,
+ current.objectType,
+ current.objectName)
+
+ // Preserve any existing substitution info when setting the new origin.
+ val finalOrigin = if (current.parameterSubstitutionInfo.isDefined) {
+ adjustedOrigin.copy(parameterSubstitutionInfo = current.parameterSubstitutionInfo)
+ } else {
+ adjustedOrigin
+ }
+
+ CurrentOrigin.set(finalOrigin)
}
try {
f
} finally {
- CurrentOrigin.set(current)
+ // When restoring origin, only preserve substitution info if it was already present
+ // in the original context. This prevents contamination across unrelated parsing operations.
+ val originToRestore = if (current.parameterSubstitutionInfo.isDefined) {
+ // Original context had substitution info - preserve it.
+ current
+ } else {
+ // Original context had no substitution info - don't add any.
+ current
Review Comment:
this is the same as the other branch of the if statement :) we can just delete L222-230 and return `CurrentOrigin.set(current)`. You can keep the comments if desired.
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations = namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Question mark is single character, so stopIndex = startIndex + 1
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers. This handles the namedParameterMarker case
+ * in the shared parameterMarker grammar rule. Note: The return value is not used; this method
+ * operates via side effects.
+ */
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations = namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers. This handles the QUESTION case in the
+ * shared parameterMarker grammar rule. Note: The return value is not used; this method operates
+ * via side effects.
+ */
+ override def visitPositionalParameterMarkerRule(
+ ctx: PositionalParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Parameter marker is single character, so stopIndex = startIndex + 1
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: ParameterStringValueContext =>
+ // Handle parameter markers in string contexts
+ visit(ctx.parameterMarker())
+ case ctx: ParameterIntegerValueContext =>
+ // Handle parameter markers in integer contexts
+ visit(ctx.parameterMarker())
+ case ctx: NamedParameterMarkerRuleContext =>
+ visitNamedParameterMarkerRule(ctx)
+ case ctx: PositionalParameterMarkerRuleContext =>
+ visitPositionalParameterMarkerRule(ctx)
+ case ctx: StringLiteralInContextContext =>
+ // For string literals in context, continue traversing to find any nested parameters
+ visitChildren(ctx)
+ case ruleNode: RuleNode =>
+ // Continue traversing children for rule nodes
+ visitChildren(ruleNode)
+ case _ =>
+ // For other types (like terminal nodes), don't traverse children
+ null
+ }
+ }
+
+ /**
+ * Visit all children of a node to find parameters.
+ */
+ override def visitChildren(node: RuleNode): AnyRef = {
+ if (node == null) return null
Review Comment:
```suggestion
if (node == null) {
return null
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429625367
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2200,7 +2205,15 @@ class AstBuilder extends DataTypeAstBuilder
Limit(expression(ctx.expression), query)
case ctx: SampleByPercentileContext =>
- val fraction = ctx.percentage.getText.toDouble
+ val fraction = if (ctx.DECIMAL_VALUE() != null) { ctx.DECIMAL_VALUE().getText.toDouble }
+ else {
+ // Assert that parameter markers have been substituted before reaching AstBuilder
+ assert(ctx.integerValue().getChild(0).getClass.getSimpleName !=
Review Comment:
ditto: https://github.com/apache/spark/pull/52334/files#r2429594567
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -716,12 +716,17 @@ class AstBuilder extends DataTypeAstBuilder
private def withCTE(ctx: CtesContext, plan: LogicalPlan): LogicalPlan = {
val ctes = ctx.namedQuery.asScala.map { nCtx =>
val namedQuery = visitNamedQuery(nCtx)
- val rowLevelLimit: Option[Int] = if (nCtx.INTEGER_VALUE() != null) {
+ val rowLevelLimit: Option[Int] = if (nCtx.integerValue() != null) {
if (ctx.RECURSIVE() == null) {
operationNotAllowed("Cannot specify MAX RECURSION LEVEL when the CTE is not marked as " +
"RECURSIVE", ctx)
}
- Some(nCtx.INTEGER_VALUE().getText().toInt)
+ // Assert that parameter markers have been substituted before reaching AstBuilder
+ assert(nCtx.integerValue().getChild(0).getClass.getSimpleName !=
Review Comment:
ditto: https://github.com/apache/spark/pull/52334/files#r2429594567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429713670
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
Review Comment:
when it will be false? I can't find the recursive call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2395324460
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -234,4 +276,71 @@ trait SparkParserUtils {
}
}
-object SparkParserUtils extends SparkParserUtils
+object SparkParserUtils extends SparkParserUtils {
+
+ /**
+ * Callback type for parameter substitution integration.
+ *
+ * This callback allows the parameter substitution system to provide origin adjustment without
+ * creating circular dependencies between modules.
+ *
+ * @param startToken
+ * The start token from substituted SQL
+ * @param stopToken
+ * The stop token from substituted SQL
+ * @param substitutedSql
+ * The substituted SQL text
+ * @param objectType
+ * The object type
+ * @param objectName
+ * The object name
+ * @return
+ * Some(origin) if parameter substitution should be applied, None otherwise
+ */
+ type ParameterSubstitutionCallback =
+ (Token, Token, String, Option[String], Option[String]) => Option[Origin]
+
+ /**
+ * Thread-local callback for parameter substitution integration. This is set by the parameter
+ * handler when parameter substitution occurs.
+ */
+ private val parameterSubstitutionCallbackStorage =
+ new ThreadLocal[Option[ParameterSubstitutionCallback]]() {
Review Comment:
I wonder if there is any way we can set the callback using the parsing API rather than a thread local like this. This could make it more complex to support parallel parsing later, for example. If the parsing entry point is the `parsePlan` method of AbstractSqlParser, that introduces a call site with stack frames creating state, currently starting with the SQL string to parse. Is there any way we could pass this callback as part of that state into the stack frames instead of using a thread local?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -234,4 +276,71 @@ trait SparkParserUtils {
}
}
-object SparkParserUtils extends SparkParserUtils
+object SparkParserUtils extends SparkParserUtils {
+
+ /**
+ * Callback type for parameter substitution integration.
+ *
+ * This callback allows the parameter substitution system to provide origin adjustment without
+ * creating circular dependencies between modules.
+ *
+ * @param startToken
+ * The start token from substituted SQL
+ * @param stopToken
+ * The stop token from substituted SQL
+ * @param substitutedSql
+ * The substituted SQL text
+ * @param objectType
+ * The object type
+ * @param objectName
+ * The object name
+ * @return
+ * Some(origin) if parameter substitution should be applied, None otherwise
+ */
+ type ParameterSubstitutionCallback =
+ (Token, Token, String, Option[String], Option[String]) => Option[Origin]
+
+ /**
+ * Thread-local callback for parameter substitution integration. This is set by the parameter
+ * handler when parameter substitution occurs.
+ */
+ private val parameterSubstitutionCallbackStorage =
+ new ThreadLocal[Option[ParameterSubstitutionCallback]]() {
+ override def initialValue(): Option[ParameterSubstitutionCallback] = None
+ }
+
+ /**
+ * Get the current parameter substitution callback.
+ */
+ def parameterSubstitutionCallback: Option[ParameterSubstitutionCallback] = {
+ parameterSubstitutionCallbackStorage.get()
+ }
+
+ /**
+ * Set the parameter substitution callback for the current thread. This should be called by
+ * parameter handlers before parsing.
+ */
+ def setParameterSubstitutionCallback(callback: ParameterSubstitutionCallback): Unit = {
+ parameterSubstitutionCallbackStorage.set(Some(callback))
+ }
+
+ /**
+ * Clear the parameter substitution callback for the current thread.
+ */
+ def clearParameterSubstitutionCallback(): Unit = {
+ parameterSubstitutionCallbackStorage.remove()
+ }
+
+ /**
+ * Execute a block with a parameter substitution callback set.
+ */
+ def withParameterSubstitutionCallback[T](
Review Comment:
This is not used anywhere in the PR as of now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425285205
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
+ *
+ * @param originalText
+ * The original SQL text with parameter markers
+ * @param substitutedText
+ * The SQL text after parameter substitution
+ * @param substitutions
+ * List of substitutions that were applied
+ */
+class PositionMapper(
+ val originalText: String,
+ val substitutedText: String,
+ val substitutions: List[Substitution]) {
+
+ // Build sparse position ranges for efficient lookup
+ private val positionRanges = buildPositionRanges()
+
+ /**
+ * Map a position in the substituted text back to the original text. Uses binary search for
+ * O(log k) lookup time.
+ *
+ * @param substitutedPos
+ * Position in the substituted text
+ * @return
+ * Position in the original text, or the same position if no mapping exists
+ */
+ def mapToOriginal(substitutedPos: Int): Int = {
+ // Binary search for the range containing this position
+ positionRanges.find(range =>
+ substitutedPos >= range.substitutedStart && substitutedPos < range.substitutedEnd) match {
+ case Some(range) =>
+ // Position is within a substitution range
+ range.originalStart
+ case None =>
+ // Position is in an unmapped region - apply cumulative offset
+ val cumulativeOffset = positionRanges
+ .takeWhile(_.substitutedStart <= substitutedPos)
+ .map(_.offsetDelta)
+ .sum
+ substitutedPos + cumulativeOffset
+ }
+ }
+
+ /**
+ * Build sparse position ranges using functional approach. O(k) space complexity where k =
+ * number of substitutions.
+ *
+ * @example
+ * For original "SELECT :name, :age" -> substituted "SELECT 'John', 25":
+ * - Substitution(7, 12, "'John'") replaces ":name" with "'John'"
+ * - Substitution(14, 18, "25") replaces ":age" with "25"
+ *
+ * Creates PositionRanges:
+ * - Range for "'John'": substituted[7,13) -> original[7,12), offset=-1
+ * - Range for "25": substituted[15,17) -> original[14,18), offset=-3
+ *
+ * This allows mapping any position in "SELECT 'John', 25" back to "SELECT :name, :age".
+ */
+ private def buildPositionRanges(): List[PositionRange] = {
+ if (substitutions.isEmpty) {
+ return List.empty
+ }
+
+ val sortedSubstitutions = substitutions.sortBy(_.start)
+
+ // Use scanLeft for functional accumulation of position state
+ case class PositionState(originalPos: Int, substitutedPos: Int, ranges: List[PositionRange])
+
+ val finalState = sortedSubstitutions.foldLeft(PositionState(0, 0, List.empty)) {
+ case (state, substitution) =>
+ val PositionState(originalPos, substitutedPos, ranges) = state
+
+ // Calculate positions after accounting for text before this substitution
+ val textBeforeLength = substitution.start - originalPos
+ val newSubstitutedPos = substitutedPos + textBeforeLength
+
+ // Create range for the substitution
+ val substitutedLength = substitution.replacement.length
+ val originalLength = substitution.end - substitution.start
+ val offsetDelta = originalLength - substitutedLength
+
+ val newRange = PositionRange(
+ substitutedStart = newSubstitutedPos,
+ substitutedEnd = newSubstitutedPos + substitutedLength,
+ originalStart = substitution.start,
+ offsetDelta = offsetDelta)
+
+ PositionState(
+ originalPos = substitution.end,
+ substitutedPos = newSubstitutedPos + substitutedLength,
+ ranges = ranges :+ newRange)
+ }
+
+ finalState.ranges
+ }
+}
+
+/**
+ * Companion object for PositionMapper.
+ */
+object PositionMapper {
+
+ /**
+ * Create a PositionMapper from substitution information.
+ *
+ * @param originalText
+ * The original SQL text
+ * @param substitutedText
+ * The substituted SQL text
+ * @param substitutions
+ * The substitutions that were applied
+ * @return
+ * A PositionMapper instance
+ */
+ def apply(
Review Comment:
why not make `PositionMapper` a case class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426366496
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
Review Comment:
Ah, claude loves to comment on changes compared to previpous commits in the same PR (or previous state). I keep telling it to cut it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426360293
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -1666,9 +1680,19 @@ alterColumnAction
| dropDefault=DROP DEFAULT
;
+stringLitWithoutMarker
+ : STRING_LITERAL #stringLiteralValue
+ | {!double_quoted_identifiers}? DOUBLEQUOTED_STRING #doubleQuotedStringLiteralValue
+;
+
+parameterMarker
+ : {parameter_substitution_enabled}? namedParameterMarker #namedParameterMarkerRule
+ | {parameter_substitution_enabled}? QUESTION #positionalParameterMarkerRule
+ ;
+
stringLit
- : STRING_LITERAL
- | {!double_quoted_identifiers}? DOUBLEQUOTED_STRING
+ : stringLitWithoutMarker #stringLiteralInContext
+ | parameterMarker #parameterStringValue
Review Comment:
You are wrong :-) These grammar changes provide coverage for all (hopefully) places where literals are allowed.
For example: Wherever we allow an DOUBLE or DECIMAL we also allow INTEGER and thus have coverage for parameter markers. Wherever we allow a DATE/TIMESTAMP we also allow a STRING and thus allow parameter markers. And of course expressions include literals, so any generic expression allows parameter markers already in the grammar.
Note that I have specifically added testcases for "interesting" places. We can add more if you feel there is a gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425622259
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
Review Comment:
shall we use the case class `ParameterLocation` instead of the start and end fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425754439
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2200,7 +2200,8 @@ class AstBuilder extends DataTypeAstBuilder
Limit(expression(ctx.expression), query)
case ctx: SampleByPercentileContext =>
- val fraction = ctx.percentage.getText.toDouble
+ val fraction = if (ctx.DECIMAL_VALUE() != null) { ctx.DECIMAL_VALUE().getText.toDouble }
+ else { ctx.integerValue().getText.toDouble }
Review Comment:
I think the parameter marker should have been replaced here? Will we really hit the else branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426997990
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -36,7 +46,8 @@ case class Origin(
objectType: Option[String] = None,
objectName: Option[String] = None,
stackTrace: Option[Array[StackTraceElement]] = None,
- pysparkErrorContext: Option[(String, String)] = None) {
+ pysparkErrorContext: Option[(String, String)] = None,
+ parameterSubstitutionInfo: Option[ParameterSubstitutionInfo] = None) {
Review Comment:
I suggested for Serge to add this new field to avoid creating a new ThreadLocal. If we can produce the new `Origin` and store it in some existing place like e.g. `CurrentOrigin` instead of this new field, that sounds OK too -- up to you guys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433088058
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -57,6 +59,44 @@ case class Origin(
*/
trait WithOrigin {
def origin: Origin
+
+ /**
+ * Update query contexts in this object with translated positions. Uses reflection to
+ * generically update any object with query context.
+ */
+ def updateQueryContext(translator: Array[QueryContext] => Array[QueryContext]): WithOrigin = {
Review Comment:
Nowhere, removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416662934
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers in string literal contexts. This handles
+ * the namedParameterMarker case added to the stringLit grammar rule. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterValue(ctx: NamedParameterValueContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers in integer value contexts. This handles the
+ * namedParameterMarker case added to the integerValue grammar rule. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterIntegerValue(ctx: NamedParameterIntegerValueContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers in integer value contexts. This
+ * handles the QUESTION case added to the integerValue grammar rule (e.g., VARCHAR(?)). Note:
+ * The return value is not used; this method operates via side effects.
+ */
+ override def visitPositionalParameterIntegerValue(
+ ctx: PositionalParameterIntegerValueContext): AnyRef = withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Calculate the location of the parameter marker
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers in string literal contexts. Note: The
+ * return value is not used; this method operates via side effects.
+ */
+ override def visitPositionalParameterValue(ctx: PositionalParameterValueContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Calculate the location of the parameter marker
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: NamedParameterValueContext =>
+ visitNamedParameterValue(ctx)
+ case ctx: NamedParameterIntegerValueContext =>
+ visitNamedParameterIntegerValue(ctx)
+ case ctx: PositionalParameterIntegerValueContext =>
+ visitPositionalParameterIntegerValue(ctx)
+ case ctx: PositionalParameterValueContext =>
+ visitPositionalParameterValue(ctx)
+ case ctx: StringLiteralInContextContext =>
+ // For string literals in context, continue traversing to find any nested parameters
+ visitChildren(ctx)
+ case ruleNode: RuleNode =>
+ // Continue traversing children for rule nodes
+ visitChildren(ruleNode)
+ case _ =>
+ // For other types (like terminal nodes), don't traverse children
+ null
+ }
+ }
+
+ /**
+ * Visit all children of a node to find parameters.
+ */
+ override def visitChildren(node: RuleNode): AnyRef = {
+ if (node == null) return null
+
+ var result: AnyRef = null
Review Comment:
does the return value matter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2374379136
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -258,9 +285,15 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
* Create a comment string.
*/
override def visitCommentSpec(ctx: CommentSpecContext): String = withOrigin(ctx) {
- string(visitStringLit(ctx.stringLit))
+ string(visit(ctx.stringLit).asInstanceOf[Token])
Review Comment:
why do we need this change given we've added a new `visitStringLit` below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375135099
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
+ */
+class ParameterHandler {
+
+
+ // Thread-local storage for position mapper to ensure thread safety
+ private val positionMapperStorage = new ThreadLocal[Option[PositionMapper]]() {
+ override def initialValue(): Option[PositionMapper] = None
+ }
+
+ /**
+ * Get the current position mapper for translating error positions.
+ * This mapper can translate positions from substituted text back to original text.
+ *
+ * @return The current position mapper, if available
+ */
+ def getPositionMapper: Option[PositionMapper] = positionMapperStorage.get()
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param rule The substitution rule to use
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ rule: SubstitutionRule,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ positionMapperStorage.set(Some(PositionMapper.identity(sqlText)))
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText, rule,
+ namedParams = namedParams, positionalParams = positionalParams)
+ positionMapperStorage.set(Some(positionMapper))
+ substituted
+ }
+
+ /**
+ * Translate error context from substituted text positions to original text positions.
+ *
+ * @param context The error context with positions in substituted text
+ * @return The error context with positions mapped to original text
+ */
+ def translateErrorContext(context: SQLQueryContext): SQLQueryContext = {
+ positionMapperStorage.get() match {
+ case Some(mapper) =>
+ val translatedStartIndex = context.originStartIndex.map(mapper.mapToOriginal)
+ val translatedStopIndex = context.originStopIndex.map(mapper.mapToOriginal)
+
+ context.copy(
+ originStartIndex = translatedStartIndex,
+ originStopIndex = translatedStopIndex,
+ sqlText = Some(mapper.originalText) // Use original text instead of substituted
+ )
+ case None =>
+ context // No position mapper available, return as-is
+ }
+ }
+
+ /**
+ * Substitute parameters in SQL text based on the parameter context.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param context The parameter context (named or positional)
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteParameters(
+ sqlText: String,
+ context: ParameterContext,
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ context match {
+ case NamedParameterContext(params) =>
+ performSubstitution(sqlText, rule, namedParams = params.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
+ })
+
+ case PositionalParameterContext(params) =>
+ performSubstitution(sqlText, rule,
+ positionalParams = params.map(expr =>
+ LiteralToSqlConverter.convert(expr)).toList)
+
+ case HybridParameterContext(args, paramNames) =>
+ // Detect which parameter types are actually used in the query (single call)
+ val (hasPositional, hasNamed) = detectParameters(sqlText, rule)
+
+ // Validate that the query doesn't mix parameter types
+ if (hasPositional && hasNamed) {
+ throw org.apache.spark.sql.errors.QueryCompilationErrors
+ .invalidQueryMixedQueryParameters()
+ }
+
+ // Validate ALL_PARAMETERS_MUST_BE_NAMED: if query uses named parameters,
+ // all USING expressions must have names
+ if (hasNamed && !hasPositional) {
+ val unnamedExpressions = paramNames.zipWithIndex.collect {
+ case ("", index) => index // empty strings are unnamed
+ }
+ if (unnamedExpressions.nonEmpty) {
+ val unnamedExprs = unnamedExpressions.map(args(_)).map {
+ case literal: org.apache.spark.sql.catalyst.expressions.Literal =>
+ // Already a Literal from ResolveExecuteImmediate - use it directly
+ literal.asInstanceOf[Expression]
+ case expr: org.apache.spark.sql.catalyst.expressions.Expression =>
+ // Expression from Column object - use it directly
+ expr
+ case value =>
+ // Raw value - create new Literal
+ Literal(value).asInstanceOf[Expression]
+ }.toSeq
+ throw org.apache.spark.sql.errors.QueryCompilationErrors
+ .invalidQueryAllParametersMustBeNamed(unnamedExprs)
+ }
+ }
+
+ if (hasPositional && !hasNamed) {
+ // Query uses only positional parameters
+ // All arguments are already expressions at this point
+ val positionalParams = args.map(_.asInstanceOf[Expression]).toSeq
+ performSubstitution(sqlText, rule,
+ positionalParams = positionalParams.map(expr =>
+ LiteralToSqlConverter.convert(expr)).toList)
+ } else if (hasNamed && !hasPositional) {
+ // Query uses only named parameters
+ val namedParams = paramNames.zip(args).collect {
+ case (name, literal: org.apache.spark.sql.catalyst.expressions.Literal)
+ if name.nonEmpty =>
+ // Already a Literal from ResolveExecuteImmediate - use it directly
+ name -> literal.asInstanceOf[Expression]
+ case (name, expr: org.apache.spark.sql.catalyst.expressions.Expression)
+ if name.nonEmpty =>
+ // Expression from Column object - use it directly
+ name -> expr
+ case (name, value) if name.nonEmpty =>
+ // Raw value - create new Literal
+ name -> Literal(value).asInstanceOf[Expression]
+ }.toMap
+ performSubstitution(sqlText, rule,
+ namedParams = namedParams.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
+ })
+ } else {
+ // No parameters in query - return as-is
+ performSubstitution(sqlText, rule)
+ }
+ }
+ }
+
+ /**
+ * Substitute parameters in SQL text with optional context.
+ * If no context is provided, returns the original SQL text.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param contextOpt Optional parameter context
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteParametersIfNeeded(
+ sqlText: String,
+ contextOpt: Option[ParameterContext],
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ contextOpt match {
+ case Some(context) => substituteParameters(sqlText, context, rule)
+ case None => sqlText
+ }
+ }
+
+ /**
+ * Substitute parameters using named parameter map.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param paramMap Map of parameter names to expressions
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteNamedParameters(
+ sqlText: String,
+ paramMap: Map[String, Expression],
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ if (paramMap.isEmpty) return sqlText
+
+ val paramValues = paramMap.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
+ }
+ performSubstitution(sqlText, rule, namedParams = paramValues)
+ }
+
+ /**
+ * Substitute parameters using positional parameter list.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param paramList Sequence of parameter expressions
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substitutePositionalParameters(
+ sqlText: String,
+ paramList: Seq[Expression],
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ if (paramList.isEmpty) return sqlText
+
+ val paramValues = paramList.map(expr =>
+ LiteralToSqlConverter.convert(expr)).toList
+ performSubstitution(sqlText, rule, positionalParams = paramValues)
+ }
+
+ /**
+ * Determine the appropriate substitution rule based on SQL content.
+ *
+ * @param sqlText The SQL text to analyze
+ * @return The appropriate substitution rule
+ */
+ def determineSubstitutionRule(sqlText: String): SubstitutionRule = {
+ val trimmed = sqlText.trim
+ if (trimmed.length >= 5 && trimmed.regionMatches(true, 0, "BEGIN", 0, 5)) {
Review Comment:
it's case insensitive match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375184314
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ * @since 4.0.0
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
Review Comment:
shall we create a fake plan to resolve the parameter expressions? They should be foldable and has no reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2417837789
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers in string literal contexts. This handles
+ * the namedParameterMarker case added to the stringLit grammar rule. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterValue(ctx: NamedParameterValueContext): AnyRef =
Review Comment:
Yes, but ANTLR does not allow reusing the same visit, and it's short enough not to share the pieces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416637850
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -261,6 +288,12 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
string(visitStringLit(ctx.stringLit))
}
+ /**
+ * Visit a stringLit context by delegating to the appropriate labeled visitor.
+ */
+ def visitStringLit(ctx: StringLitContext): Token =
Review Comment:
nit: shall we place it at the beginning of these newly added visit methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416821047
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ * @since 4.0.0
Review Comment:
version doc is not needed as this is an internal API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433408649
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala:
##########
@@ -84,23 +86,44 @@ case class ResolveExecuteImmediate(sparkSession: SparkSession, catalogManager: C
// Extract the query string from the queryParam expression
val sqlString = extractQueryString(sqlStmtStr)
- // Parse and validate the query
- val parsedPlan = sparkSession.sessionState.sqlParser.parsePlan(sqlString)
- validateQuery(sqlString, parsedPlan)
+ // Create the origin for EXECUTE IMMEDIATE context - this will be used by expressions
+ // during parsing to set their queryContext, similar to how views work
+ val executeImmediateOrigin = Origin(
+ objectType = Some("EXECUTE IMMEDIATE"),
+ objectName = None, // No named object for EXECUTE IMMEDIATE, unlike views
+ sqlText = Some(sqlString),
+ startIndex = Some(0),
+ stopIndex = Some(sqlString.length - 1)
+ )
+
+ // Execute the query recursively with isolated local variable context and EXECUTE IMMEDIATE
+ // origin. The isolation must cover parsing, analysis, and execution phases.
+ // CurrentOrigin.withOrigin ensures expressions created during parsing get the proper context
+ val result = withIsolatedLocalVariableContext {
+ CurrentOrigin.withOrigin(executeImmediateOrigin) {
+ // Parse and validate the query inside the isolation context to ensure
+ // local variables are not accessible during parsing
+ if (args.isEmpty) {
+ val parsedPlan = sparkSession.sessionState.sqlParser.parsePlan(sqlString)
Review Comment:
We are blocking SQL Scripting here which is an existing limitations. I'm updating teh comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429685414
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala:
##########
@@ -80,10 +81,8 @@ class QueryExecution(
try {
analyzed
} catch {
- case e: AnalysisException =>
- // Because we do eager analysis for Dataframe, there will be no execution created after
- // AnalysisException occurs. So we need to explicitly create a new execution to post
- // start/end events to notify the listener and UI components.
+ case e: Throwable with WithOrigin =>
Review Comment:
what does this change fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429726444
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Set up origin with original SQL text for parameter-aware error reporting.
+ val currentOrigin = CurrentOrigin.get
+ val originToUse = if ((substitutionOccurred || hasParameters) && isTopLevel) {
+ // Set up origin with the substituted SQL text for proper error reporting
+ currentOrigin.copy(
+ sqlText = Some(variableSubstituted), // Use substituted SQL text
+ startIndex = Some(0),
+ stopIndex = Some(variableSubstituted.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged.
+ currentOrigin
+ }
+
+ CurrentOrigin.withOrigin(originToUse) {
+ super.parse(variableSubstituted)(toResult)
+ }
+ }
+
+ private def substituteParametersOrSetupCallback(
+ command: String,
+ context: ParameterContext): String = {
+
+ // Check legacy configuration - if true, skip parameter substitution during parsing.
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // In legacy mode, return original command without substitution.
+ command
+ } else {
+ ParameterHandler.substituteParameters(command, context)
+ }
+ }
+
+ /**
+ * Internal parse method for identifiers and data types that bypasses parameter logic.
+ * This ensures clean parsing without parameter substitution side effects.
Review Comment:
do we need to? The `def parse[T]` calls `parseInternal` without parameter context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2427704255
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
Review Comment:
We can whitelist the functions. I.e. Map and Array constructors.
I don't think it's that complicated it's scalar types and teh transitive closure of struct, map, and array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425619273
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Question mark is single character, so stopIndex = startIndex + 1
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers. This handles the namedParameterMarker case
+ * in the shared parameterMarker grammar rule. Note: The return value is not used; this method
+ * operates via side effects.
+ */
+ override def visitNamedParameterMarkerRule(ctx: NamedParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers. This handles the QUESTION case in the
+ * shared parameterMarker grammar rule. Note: The return value is not used; this method operates
+ * via side effects.
+ */
+ override def visitPositionalParameterMarkerRule(
+ ctx: PositionalParameterMarkerRuleContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Parameter marker is single character, so stopIndex = startIndex + 1
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = startIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: ParameterStringValueContext =>
+ // Handle parameter markers in string contexts
+ visit(ctx.parameterMarker())
+ case ctx: ParameterIntegerValueContext =>
+ // Handle parameter markers in integer contexts
+ visit(ctx.parameterMarker())
+ case ctx: NamedParameterMarkerRuleContext =>
+ visitNamedParameterMarkerRule(ctx)
+ case ctx: PositionalParameterMarkerRuleContext =>
+ visitPositionalParameterMarkerRule(ctx)
+ case ctx: StringLiteralInContextContext =>
+ // For string literals in context, continue traversing to find any nested parameters
+ visitChildren(ctx)
+ case ruleNode: RuleNode =>
+ // Continue traversing children for rule nodes
+ visitChildren(ruleNode)
+ case _ =>
+ // For other types (like terminal nodes), don't traverse children
+ null
+ }
+ }
+
+ /**
+ * Visit all children of a node to find parameters.
+ */
+ override def visitChildren(node: RuleNode): AnyRef = {
+ if (node == null) {
+ return null
+ }
+
+ var result: AnyRef = null
+ for (i <- 0 until node.getChildCount) {
+ val child = node.getChild(i)
+ if (child != null) {
+ val childResult = visit(child)
+ if (result == null) {
+ result = childResult
+ }
+ }
+ }
+ result
+ }
+
+ /**
+ * Visit terminal nodes (leaf nodes in the parse tree).
+ */
+ override def visitTerminal(node: TerminalNode): AnyRef = {
+ // Terminal nodes don't contain parameters themselves,
+ // but we need to handle them in the traversal
+ null
+ }
+}
+
+/**
+ * Data class to hold information about extracted parameters. This is defined here to make it
+ * available to the AST builder.
+ */
+case class ParameterInfo(namedParameters: Set[String], positionalParameters: List[Int]) {
Review Comment:
where do we use it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426087893
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -106,6 +106,7 @@ class SparkConnectPlanner(
import sessionHolder.session.toRichColumn
private[connect] def parser = session.sessionState.sqlParser
+ private val parameterHandler = new ParameterHandler()
Review Comment:
shall we create a new instance for each SQL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426708131
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ val parsed = sessionState.sqlParser.parsePlanWithParameters(sqlText, paramContext)
Review Comment:
SparkSession needs pasrePlanWithParameters
SparkConnect has a more complex protocol and nneeds teh ParameterHandler separate.
It also allows for modular unit testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2435990813
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterContext.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * Context for passing parameter values to the SQL parser.
+ * This allows parameter substitution to happen during the parsing phase
+ * rather than during analysis.
+ */
+sealed trait ParameterContext
+
+/**
+ * Context for named parameters (e.g., :paramName).
+ *
+ * @param params Map of parameter names to their expression values
+ */
+case class NamedParameterContext(params: Map[String, Expression]) extends ParameterContext
+
+/**
+ * Context for positional parameters (e.g., ?).
+ *
+ * @param params Sequence of expression values in order
+ */
+case class PositionalParameterContext(params: Seq[Expression]) extends ParameterContext
+
+/**
+ * Context that supports both named and positional parameters.
+ * This is used by EXECUTE IMMEDIATE where the parameter type is determined by the inner query.
+ *
+ * @param args Raw argument values from USING clause (already evaluated)
+ * @param paramNames Parameter names from USING clause (empty strings for positional)
+ */
+case class HybridParameterContext(
+ args: Array[_],
Review Comment:
shall we use `Seq[Expression]` to be consistent with the positional context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436008283
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -430,6 +431,77 @@ class SparkSession private(
| Everything else |
* ----------------- */
+ /**
+ * Resolves and validates parameter expressions using a fake Project plan.
+ * This ensures parameters are properly resolved and constant-folded before being
+ * passed to the pre-parser.
+ *
+ * @param args Map of parameter names to expressions
+ * @return Map of parameter names to resolved literal expressions
+ */
+ private[sql] def resolveAndValidateParameters(
+ args: Map[String, Expression]): Map[String, Literal] = {
+ if (args.isEmpty) {
+ return Map.empty
+ }
+
+ // Create a fake Project plan: Project(parameters, OneRowRelation)
+ val aliases = args.map { case (name, expr) =>
+ Alias(expr, name)()
+ }.toSeq
+ val fakePlan = Project(aliases, OneRowRelation())
+
+ // Analyze and optimize to resolve and constant-fold
+ val analyzed = sessionState.analyzer.execute(fakePlan)
+ val optimized = sessionState.optimizer.execute(analyzed)
+
+ // Extract the resolved expressions from the project list
+ val resolvedParams = optimized.asInstanceOf[Project].projectList.map { alias =>
+ val resolvedExpr = alias.asInstanceOf[Alias].child
+
+ // Validate: the expression tree must only contain allowed expression types.
+ // This mirrors the validation in BindParameters.checkArgs.
+ // The validation happens on the analyzed (but not yet optimized) plan to catch
+ // unsupported functions like str_to_map before they get folded.
+ val analyzedExpr = analyzed.asInstanceOf[Project].projectList
+ .find(_.asInstanceOf[Alias].name == alias.name)
+ .get.asInstanceOf[Alias].child
+
+ def isNotAllowed(expr: Expression): Boolean = expr.exists {
+ case _: Literal | _: CreateArray | _: CreateNamedStruct |
+ _: CreateMap | _: MapFromArrays | _: MapFromEntries | _: VariableReference => false
+ case a: Alias => isNotAllowed(a.child)
+ case _ => true
+ }
+
+ if (isNotAllowed(analyzedExpr)) {
Review Comment:
can we check it before we optimize the plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436002779
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala:
##########
@@ -37,7 +37,11 @@ class QueryPlanSuite extends SparkFunSuite {
CurrentOrigin.reset()
val mappedQuery = query mapExpressions {
- case _: Expression => Literal(1)
+ case _: Expression =>
+ // Reset CurrentOrigin to prevent contamination from previous operations
Review Comment:
there is no parsing in this suite, why are we doing it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2422359131
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -57,6 +68,51 @@ case class Origin(
*/
trait WithOrigin {
def origin: Origin
+
+ /**
+ * Update query contexts in this object with translated positions. Uses reflection to
+ * generically update any object with query context.
+ */
+ def updateQueryContext(translator: Array[QueryContext] => Array[QueryContext]): WithOrigin = {
+ try {
+ val thisClass = this.getClass
+
+ // Try to find query context using common method names
+ val contextMethodNames = Seq("getQueryContext", "context")
+
+ for (methodName <- contextMethodNames) {
+ try {
+ val getMethod = thisClass.getMethod(methodName)
+ val currentContexts = getMethod.invoke(this).asInstanceOf[Array[QueryContext]]
+ val translatedContexts = translator(currentContexts)
+
+ // Try to update the field in-place
+ val fieldName = if (methodName == "getQueryContext") "queryContext" else "context"
+ try {
+ val field = thisClass.getDeclaredField(fieldName)
+ field.setAccessible(true)
+ field.set(this, translatedContexts)
+ return this // Successfully updated in-place!
Review Comment:
can you skip this and the `this` on L109 would work instead equivalently?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -57,6 +68,51 @@ case class Origin(
*/
trait WithOrigin {
def origin: Origin
+
+ /**
+ * Update query contexts in this object with translated positions. Uses reflection to
+ * generically update any object with query context.
+ */
+ def updateQueryContext(translator: Array[QueryContext] => Array[QueryContext]): WithOrigin = {
+ try {
+ val thisClass = this.getClass
+
+ // Try to find query context using common method names
+ val contextMethodNames = Seq("getQueryContext", "context")
+
+ for (methodName <- contextMethodNames) {
+ try {
+ val getMethod = thisClass.getMethod(methodName)
+ val currentContexts = getMethod.invoke(this).asInstanceOf[Array[QueryContext]]
+ val translatedContexts = translator(currentContexts)
+
+ // Try to update the field in-place
+ val fieldName = if (methodName == "getQueryContext") "queryContext" else "context"
+ try {
+ val field = thisClass.getDeclaredField(fieldName)
+ field.setAccessible(true)
+ field.set(this, translatedContexts)
+ return this // Successfully updated in-place!
+ } catch {
+ case _: NoSuchFieldException | _: IllegalAccessException |
+ _: IllegalArgumentException =>
+ // Field update failed, continue to try other methods
+ }
+ } catch {
+ case _: NoSuchMethodException | _: IllegalAccessException |
Review Comment:
would it be simpler and equivalent to just combine all these exceptions into a single try/catch block, to avoid the nesting of three of them?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.trees.ParameterSubstitutionInfo
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+
Review Comment:
```suggestion
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +53,195 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Clear any stale substitution info from previous parsing operations to prevent contamination.
+ if (isTopLevel) {
+ CurrentOrigin.set(CurrentOrigin.get.copy(parameterSubstitutionInfo = None))
+ }
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Set up origin with original SQL text for parameter-aware error reporting.
+ val currentOrigin = CurrentOrigin.get
+ val originToUse = if ((substitutionOccurred || hasParameters) && isTopLevel) {
+ // Parameter substitution occurred or parameters detected in legacy mode.
+ // Set original SQL text for accurate error position mapping.
+ // IMPORTANT: Preserve any substitution info set by parameter substitution.
+ val baseOrigin = currentOrigin.copy(
+ sqlText = Some(command), // Use original SQL text, not substituted.
+ startIndex = Some(0),
+ stopIndex = Some(command.length - 1)
+ // parameterSubstitutionInfo is preserved by copy().
+ )
+
+ // Set up identity substitution info for legacy mode parameter error reporting.
+ if (hasParameters && !substitutionOccurred) {
+ // Legacy mode - set up identity info for proper error context.
+ val identityInfo = ParameterSubstitutionInfo(
+ originalSql = command,
+ isIdentity = true,
+ positionMapper = None
+ )
+ baseOrigin.copy(parameterSubstitutionInfo = Some(identityInfo))
+ } else {
+ baseOrigin
+ }
+ } else {
+ // No substitution or nested call - use existing origin unchanged.
+ currentOrigin
+ }
+
+ CurrentOrigin.withOrigin(originToUse) {
+ super.parse(variableSubstituted)(toResult)
+ }
+ }
+
+ private def substituteParametersOrSetupCallback(
+ command: String,
+ context: ParameterContext): String = {
+
+ // Check legacy configuration - if true, skip parameter substitution during parsing.
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // In legacy mode, set up identity mapping to ensure clean error context.
+ parameterHandler.setupSubstitutionContext(
+ command,
+ command, // Original = substituted in legacy mode.
+ PositionMapper.identity(command), // Identity mapper since no substitution.
+ true // isIdentity = true.
+ )
+ command
+ } else {
+ parameterHandler.substituteParameters(command, context)
+ }
+ }
+
+ /**
+ * Internal parse method for identifiers and data types that bypasses parameter logic.
+ * This ensures clean parsing without parameter substitution side effects.
+ */
+ private def parseIdentifierInternal[T](command: String)(toResult: SqlBaseParser => T): T = {
+ // Clear any stale substitution info to prevent contamination between tests/operations.
+ val currentOrigin = CurrentOrigin.get
+ if (currentOrigin.parameterSubstitutionInfo.isDefined) {
+ CurrentOrigin.set(currentOrigin.copy(parameterSubstitutionInfo = None))
+ }
+
+ val variableSubstituted = substitutor.substitute(command)
+ try {
+ super.parse(variableSubstituted)(toResult)
+ } finally {
+ // Ensure we clear any parameter substitution info that might have been set during parsing
Review Comment:
```suggestion
// Ensure we clear any parameter substitution info that might have been set during parsing.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425246673
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
Review Comment:
what does `previous` mean here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425237199
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -1666,9 +1680,19 @@ alterColumnAction
| dropDefault=DROP DEFAULT
;
+stringLitWithoutMarker
+ : STRING_LITERAL #stringLiteralValue
+ | {!double_quoted_identifiers}? DOUBLEQUOTED_STRING #doubleQuotedStringLiteralValue
+;
+
+parameterMarker
+ : {parameter_substitution_enabled}? namedParameterMarker #namedParameterMarkerRule
+ | {parameter_substitution_enabled}? QUESTION #positionalParameterMarkerRule
+ ;
+
stringLit
- : STRING_LITERAL
- | {!double_quoted_identifiers}? DOUBLEQUOTED_STRING
+ : stringLitWithoutMarker #stringLiteralInContext
+ | parameterMarker #parameterStringValue
Review Comment:
Correct me if I'm wrong: the pre-parser only allow parameter markers in `integerValue` and `stringLit`, and we still need the previous framework to bind parameters at analysis time for arbitrary expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2392010540
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ * @since 4.0.0
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
Review Comment:
I played with this, but it seems heavy, given how specialized the need is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2392021386
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ * @since 4.0.0
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ // Note: array, map, struct are naturally foldable but reach here as UnresolvedFunction
+ // map_from_arrays and map_from_entries are not naturally foldable
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ // Modern parameter substitution supports: legacy whitelist + foldable expressions
+ // Functions like str_to_map are intentionally not supported
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: Exception =>
+ // Fallback to manual conversion for cases where Literal.sql doesn't work
+ // (e.g., complex types that need constructor syntax)
+ lit match {
+ case Literal(null, _) => "NULL"
+ case Literal(value, dataType) => dataType match {
+ case ArrayType(elementType, _) => convertArrayLiteral(value, elementType)
+ case MapType(keyType, valueType, _) => convertMapLiteral(value, keyType, valueType)
+ case _: StructType =>
+ // Struct literals (row values) - convert to ROW constructor
+ value match {
+ case row: org.apache.spark.sql.catalyst.InternalRow =>
+ val structType = dataType.asInstanceOf[StructType]
+ val fieldValues = (0 until row.numFields).map { i =>
+ if (row.isNullAt(i)) {
+ "NULL"
+ } else {
+ val fieldType = structType.fields(i).dataType
+ val fieldValue = row.get(i, fieldType)
+ val fieldLiteral = Literal.create(fieldValue, fieldType)
+ convert(fieldLiteral)
+ }
+ }
+ s"ROW(${fieldValues.mkString(", ")})"
+ case _ => s"ROW(${value.toString})"
+ }
+ case _ =>
+ // For any other unsupported type, fall back to string representation
+ s"'${value.toString.replace("'", "''")}'"
+ }
+ }
+ }
+ }
+
+ private def convertArrayLiteral(value: Any, elementType: DataType): String = {
Review Comment:
The only difference is that this method handles a few more input types (like plain Scala Seq and Array), but since we're already calling lit.sql first in convertLiteral, we should only reach our custom method if Literal#sql fails.
I recall that there were tests using these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426063832
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+
+ // Use shared parser configuration to ensure consistency with main parser.
+ AbstractParser.configureParser(parser, sqlText, tokenStream, SQLConf.get)
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use shared two-stage parsing strategy for consistent error handling.
+ val ctx = AbstractParser.executeWithTwoStageStrategy(parser, tokenStream,
+ _.compoundOrSingleStatement())
+ val parameterLocations = astBuilder.extractParameterLocations(ctx)
+
+ // Substitute parameters in the original text.
+ val (substitutedSql, appliedSubstitutions) = substituteAtLocations(sqlText, parameterLocations,
+ namedParams, positionalParams)
+ val consumedPositionalParams = parameterLocations.positionalParameterLocations.length
+
+ // Create position mapper for error context translation.
+ val positionMapper = PositionMapper(sqlText, substitutedSql, appliedSubstitutions)
+
+ (substitutedSql, consumedPositionalParams, positionMapper)
+ }
+
+ /**
+ * Detects parameter markers in SQL text without performing substitution.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text to analyze
+ * @return A tuple of (hasPositionalParameters, hasNamedParameters)
+ */
+ def detectParameters(sqlText: String): (Boolean, Boolean) = {
Review Comment:
This detection means a repeated parsing. Why can't we have a global state in `SubstituteParmsAstBuilder`, and detect mixed parameter markers during real parsing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426377757
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426497706
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
Review Comment:
I do not undersstand teh issue. A literal is a literal. e.g. is TIMESTAMP'2025-12-01 00:00:00' a complex literal?
Where do we draw teh line and why?
Can you giv ean example opf SQL injection through a literal?
Or are you worried that, what is passed here is not actually a literal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426993689
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.trees.ParameterSubstitutionInfo
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ val identityMapper = PositionMapper.identity(sqlText)
+ setupSubstitutionContext(sqlText, sqlText, identityMapper, isIdentity = true)
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ setupSubstitutionContext(sqlText, substituted, positionMapper, isIdentity = false)
+ substituted
+ }
+
+ /**
+ * Set up parameter substitution context in CurrentOrigin for error reporting.
+ * This creates a ParameterSubstitutionInfo object with position mapping data.
+ *
+ * @param originalSql The original SQL text before substitution
+ * @param substitutedSql The SQL text after substitution
+ * @param positionMapper The mapper for translating positions
+ * @param isIdentity Whether this is an identity mapping (no substitution occurred)
+ */
+ private[sql] def setupSubstitutionContext(
+ originalSql: String,
+ substitutedSql: String,
+ positionMapper: PositionMapper,
+ isIdentity: Boolean): Unit = {
+
+ // Create parameter substitution info with the position mapper.
+ val substitutionInfo = ParameterSubstitutionInfo(
+ originalSql = originalSql,
+ isIdentity = isIdentity,
+ positionMapper = if (isIdentity) None else Some(positionMapper)
+ )
+
+ // Store the info in CurrentOrigin for position mapping during error reporting.
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val updatedOrigin = currentOrigin.copy(parameterSubstitutionInfo = Some(substitutionInfo))
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.set(updatedOrigin)
Review Comment:
I did. Here it seems that the `CurrentOrigin.set` is already an existing ThreadLocal. Here it seems we are updating the current `Origin` to add the parameter substitution info. This seems OK unless there is some problem with the approach? @cloud-fan what issue will the main parser have with the `Origin` if we do it this way, are we missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433141413
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ val parsed = sessionState.sqlParser.parsePlanWithParameters(sqlText, paramContext)
Review Comment:
I don't think so. Tehre is stuiff in parsePLanWithParameters which we need in legacy mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433609257
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,24 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
+ val parsedPlan = if (args.nonEmpty) {
+ // Use parameter context directly for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
Review Comment:
Yea we should at least resolve it here. I think it's fragile to pass unresolved parameter expressions to the pre-parser. Can we use the fake plan to resolve them? We also need to perform some validation as the previous parameter binding framework does: https://github.com/apache/spark/blob/b0285f8bbf8248ca5b9d9aebea087cb5037a4655/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/parameters.scala#L176-L188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433590831
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -90,6 +90,17 @@ class AstBuilder extends DataTypeAstBuilder
}
}
+ /**
+ * Get the integer value from an IntegerValueContext after parameter replacement.
+ * Asserts that parameter markers have been substituted before reaching AstBuilder.
+ */
+ protected def getIntegerValue(ctx: IntegerValueContext): Int = {
Review Comment:
nit: we can put it in the parent class `DataTypeAstBuilder`, which also needs to call it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2435993902
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Parameter expectation types.
+ */
+object ParameterExpectation extends Enumeration {
+ val Named, Positional, Unknown = Value
+}
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty,
+ expectationType: ParameterExpectation.Value = ParameterExpectation.Unknown,
+ allParametersAreNamed: Boolean = true,
+ originalArgs: Seq[Any] = Seq.empty,
+ originalParamNames: Seq[String] = Seq.empty):
+ (String, Int, PositionMapper) = {
Review Comment:
```suggestion
originalParamNames: Seq[String] = Seq.empty): (String, Int, PositionMapper) = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429717236
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Set up origin with original SQL text for parameter-aware error reporting.
+ val currentOrigin = CurrentOrigin.get
+ val originToUse = if ((substitutionOccurred || hasParameters) && isTopLevel) {
+ // Set up origin with the substituted SQL text for proper error reporting
+ currentOrigin.copy(
+ sqlText = Some(variableSubstituted), // Use substituted SQL text
+ startIndex = Some(0),
+ stopIndex = Some(variableSubstituted.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged.
+ currentOrigin
+ }
+
+ CurrentOrigin.withOrigin(originToUse) {
+ super.parse(variableSubstituted)(toResult)
+ }
+ }
+
+ private def substituteParametersOrSetupCallback(
+ command: String,
+ context: ParameterContext): String = {
+
+ // Check legacy configuration - if true, skip parameter substitution during parsing.
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
Review Comment:
the config is already checked by the caller side, we don't need to check again here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426027066
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.trees.ParameterSubstitutionInfo
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
Review Comment:
seems not exist anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429680434
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -483,20 +492,33 @@ class SparkSession private(
* such as `map()`, `array()`, `struct()`, in that case it is taken as is.
* @param tracker A tracker that can notify when query is ready for execution
*/
- private[sql] def sql(
+ private[sql] def sql(
Review Comment:
wrong indentation change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #52334:
URL: https://github.com/apache/spark/pull/52334#issuecomment-3412900763
We reviewed this extensively, the architecture is sound and the test coverage is thorough. The one remaining CI failure is `OracleIntegrationSuite` which is flaky and not related to this PR. Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2416790294
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -187,25 +187,93 @@ trait SparkParserUtils {
* Register the origin of the context. Any TreeNode created in the closure will be assigned the
* registered origin. This method restores the previously set origin after completion of the
* closure.
+ *
+ * This method is parameter substitution-aware. If parameter substitution occurred before
+ * parsing, it will automatically adjust the positions and SQL text to refer to the original SQL
+ * (before substitution) instead of the substituted SQL.
*/
def withOrigin[T](ctx: ParserRuleContext, sqlText: Option[String] = None)(f: => T): T = {
val current = CurrentOrigin.get
val text = sqlText.orElse(current.sqlText)
+
if (text.isEmpty) {
CurrentOrigin.set(position(ctx.getStart))
} else {
- CurrentOrigin.set(
- positionAndText(
- ctx.getStart,
- ctx.getStop,
- text.get,
- current.objectType,
- current.objectName))
+ // Check if parameter substitution occurred and adjust origin accordingly.
+ val adjustedOrigin = adjustOriginForParameterSubstitution(
+ ctx.getStart,
+ ctx.getStop,
+ text.get,
+ current.objectType,
+ current.objectName)
+
+ // Preserve any existing callback when setting the new origin.
+ val finalOrigin = if (current.parameterSubstitutionCallback.isDefined) {
+ adjustedOrigin.copy(parameterSubstitutionCallback = current.parameterSubstitutionCallback)
Review Comment:
why use a callback instead of update the Origin in place here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2417835303
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = scala.collection.mutable.Set[String]()
+ private val positionalParams = scala.collection.mutable.ListBuffer[Int]()
+ private val namedParamLocations =
+ scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = scala.collection.mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val startIndex = ctx.QUESTION().getSymbol.getStartIndex
+ positionalParams += startIndex
+
+ // Calculate the location of the question mark
+ val stopIndex = ctx.QUESTION().getSymbol.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers in string literal contexts. This handles
+ * the namedParameterMarker case added to the stringLit grammar rule. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterValue(ctx: NamedParameterValueContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about named parameter markers in integer value contexts. This handles the
+ * namedParameterMarker case added to the integerValue grammar rule. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterIntegerValue(ctx: NamedParameterIntegerValueContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ namedParamLocations.getOrElseUpdate(
+ paramName,
+ scala.collection.mutable.ListBuffer[ParameterLocation]()) += ParameterLocation(
+ startIndex,
+ stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers in integer value contexts. This
+ * handles the QUESTION case added to the integerValue grammar rule (e.g., VARCHAR(?)). Note:
+ * The return value is not used; this method operates via side effects.
+ */
+ override def visitPositionalParameterIntegerValue(
+ ctx: PositionalParameterIntegerValueContext): AnyRef = withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Calculate the location of the parameter marker
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about positional parameter markers in string literal contexts. Note: The
+ * return value is not used; this method operates via side effects.
+ */
+ override def visitPositionalParameterValue(ctx: PositionalParameterValueContext): AnyRef =
+ withOrigin(ctx) {
+ val paramIndex = positionalParams.size
+ positionalParams += paramIndex
+
+ // Calculate the location of the parameter marker
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ positionalParamLocations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Override visit to ensure we traverse all children to find parameters.
+ */
+ override def visit(tree: ParseTree): AnyRef = {
+ if (tree == null) return null
+
+ // Check if this is a parameter literal
+ tree match {
+ case ctx: NamedParameterLiteralContext =>
+ visitNamedParameterLiteral(ctx)
+ case ctx: PosParameterLiteralContext =>
+ visitPosParameterLiteral(ctx)
+ case ctx: NamedParameterValueContext =>
+ visitNamedParameterValue(ctx)
+ case ctx: NamedParameterIntegerValueContext =>
+ visitNamedParameterIntegerValue(ctx)
+ case ctx: PositionalParameterIntegerValueContext =>
+ visitPositionalParameterIntegerValue(ctx)
+ case ctx: PositionalParameterValueContext =>
+ visitPositionalParameterValue(ctx)
+ case ctx: StringLiteralInContextContext =>
+ // For string literals in context, continue traversing to find any nested parameters
+ visitChildren(ctx)
+ case ruleNode: RuleNode =>
+ // Continue traversing children for rule nodes
+ visitChildren(ruleNode)
+ case _ =>
+ // For other types (like terminal nodes), don't traverse children
+ null
+ }
+ }
+
+ /**
+ * Visit all children of a node to find parameters.
+ */
+ override def visitChildren(node: RuleNode): AnyRef = {
+ if (node == null) return null
+
+ var result: AnyRef = null
Review Comment:
I think you asked that before, which is why I changed it to Any to make it clear that it does not. :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425248128
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+/**
+ * Case class representing a text substitution.
+ */
+case class Substitution(start: Int, end: Int, replacement: String)
+
+/**
+ * Represents a range mapping from substituted positions to original positions. This is used for
+ * efficient O(k) position mapping where k = number of substitutions.
+ *
+ * @param substitutedStart
+ * Start position in substituted text (inclusive)
+ * @param substitutedEnd
+ * End position in substituted text (exclusive)
+ * @param originalStart
+ * Start position in original text
+ * @param offsetDelta
+ * Offset difference between original and substituted positions
+ */
+case class PositionRange(
+ substitutedStart: Int,
+ substitutedEnd: Int,
+ originalStart: Int,
+ offsetDelta: Int)
+
+/**
+ * Maps positions between original SQL text and substituted SQL text using sparse ranges.
+ *
+ * This implementation uses O(k) space and O(log k) lookup time where k = number of substitutions,
+ * instead of the previous O(n) space where n = SQL text length.
+ *
+ * @param originalText
+ * The original SQL text with parameter markers
+ * @param substitutedText
+ * The SQL text after parameter substitution
+ * @param substitutions
+ * List of substitutions that were applied
+ */
+class PositionMapper(
+ val originalText: String,
+ val substitutedText: String,
+ val substitutions: List[Substitution]) {
Review Comment:
shall we add an assert to make sure the substitutions have no overlap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2425748483
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -36,7 +46,8 @@ case class Origin(
objectType: Option[String] = None,
objectName: Option[String] = None,
stackTrace: Option[Array[StackTraceElement]] = None,
- pysparkErrorContext: Option[(String, String)] = None) {
+ pysparkErrorContext: Option[(String, String)] = None,
+ parameterSubstitutionInfo: Option[ParameterSubstitutionInfo] = None) {
Review Comment:
why do we need this new field? In the pre-parser, we can update the `sqlText` and other position/index fields and produce a new `Origin`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2375117010
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
Review Comment:
```suggestion
```
it's internal API, no need to have version doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429716031
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
+
+ // Step 3: Set up origin with original SQL text for parameter-aware error reporting.
+ val currentOrigin = CurrentOrigin.get
+ val originToUse = if ((substitutionOccurred || hasParameters) && isTopLevel) {
+ // Set up origin with the substituted SQL text for proper error reporting
+ currentOrigin.copy(
+ sqlText = Some(variableSubstituted), // Use substituted SQL text
+ startIndex = Some(0),
+ stopIndex = Some(variableSubstituted.length - 1)
+ )
+ } else {
+ // No substitution or nested call - use existing origin unchanged.
+ currentOrigin
+ }
+
+ CurrentOrigin.withOrigin(originToUse) {
+ super.parse(variableSubstituted)(toResult)
+ }
+ }
+
+ private def substituteParametersOrSetupCallback(
Review Comment:
what does `OrSetupCallback` mean here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2427665865
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
Review Comment:
For array/struct/map, the generated SQL can be quite complicated and is not a literal in the context of a SQL (contains function invocation and CAST).
@dtenedor yes we are re-parsing the injected literal string. What worries me is that the behavior can be undefined and we can never test all the cases. We put a complicated SQL clause to places that require integer/string literals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2427656599
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -106,6 +106,7 @@ class SparkConnectPlanner(
import sessionHolder.session.toRichColumn
private[connect] def parser = session.sessionState.sqlParser
+ private val parameterHandler = new ParameterHandler()
Review Comment:
then shall we make it an scala `object`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429594567
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:
##########
@@ -138,7 +159,18 @@ class DataTypeAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
}
} else {
val badType = typeCtx.unsupportedType.getText
- val params = typeCtx.INTEGER_VALUE().asScala.toList
+ val params = typeCtx
+ .integerValue()
+ .asScala
+ .map { intVal =>
+ // Assert that parameter markers have been substituted before reaching DataTypeAstBuilder
+ assert(
+ intVal.getChild(0).getClass.getSimpleName != "ParameterIntegerValueContext",
Review Comment:
```suggestion
intVal.parameterMarker() == null,
```
does the above work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2392114181
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -448,16 +448,26 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args: Array[_], tracker: QueryPlanningTracker): DataFrame =
withActive {
val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- // Check for SQL scripting with positional parameters before creating parameterized query
- if (parsedPlan.isInstanceOf[CompoundBody]) {
- throw SqlScriptingErrors.positionalParametersAreNotSupportedWithSqlScripting()
+ val parsedPlan = if (args.nonEmpty) {
+ // Set parameter context for parsing
+ val paramContext = PositionalParameterContext(args.map(lit(_).expr).toSeq)
+ ThreadLocalParameterContext.withContext(paramContext) {
+ val parsed = sessionState.sqlParser.parsePlan(sqlText)
Review Comment:
The flow is:
1. SparkSession sets up parameter context in ThreadLocal
2. SparkSession calls sqlParser.parsePlan(sqlText)
3. SparkSqlParser.parse automatically detects the ThreadLocal parameter context
4. SparkSqlParser calls the pre-parser (ParameterHandler.substituteParameters)
5. Pre-parser returns substituted SQL text (e.g., "SELECT ?" → "SELECT 42")
6. SparkSqlParser continues with normal ANTLR parsing on the substituted text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426385554
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+
+import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
+import org.apache.spark.sql.catalyst.util.SparkParserUtils.withOrigin
+
+/**
+ * AST builder for extracting parameter markers and their locations from SQL parse trees. This
+ * builder traverses the parse tree and collects parameter information for substitution.
+ */
+class SubstituteParmsAstBuilder extends SqlBaseParserBaseVisitor[AnyRef] {
+
+ private val namedParams = mutable.Set[String]()
+ private val positionalParams = mutable.ListBuffer[Int]()
+ private val namedParamLocations = mutable.Map[String, mutable.ListBuffer[ParameterLocation]]()
+ private val positionalParamLocations = mutable.ListBuffer[ParameterLocation]()
+
+ /**
+ * Extract parameter location information from a parse context. This method traverses the parse
+ * tree and collects parameter locations for substitution.
+ */
+ def extractParameterLocations(ctx: ParseTree): ParameterLocationInfo = {
+ // Clear previous state
+ namedParams.clear()
+ positionalParams.clear()
+ namedParamLocations.clear()
+ positionalParamLocations.clear()
+
+ // Visit the context to collect parameters and their locations
+ visit(ctx)
+
+ ParameterLocationInfo(
+ namedParamLocations.view.mapValues(_.toList).toMap,
+ positionalParamLocations.toList)
+ }
+
+ /**
+ * Collect information about a named parameter in a literal context. Note: The return value is
+ * not used; this method operates via side effects.
+ */
+ override def visitNamedParameterLiteral(ctx: NamedParameterLiteralContext): AnyRef =
+ withOrigin(ctx) {
+ val paramName = ctx.namedParameterMarker().identifier().getText
+ namedParams += paramName
+
+ // Calculate the location of the entire parameter (including the colon)
+ val startIndex = ctx.getStart.getStartIndex
+ val stopIndex = ctx.getStop.getStopIndex + 1
+ val locations =
+ namedParamLocations.getOrElseUpdate(paramName, mutable.ListBuffer[ParameterLocation]())
+ locations += ParameterLocation(startIndex, stopIndex)
+
+ null // Return value not used
+ }
+
+ /**
+ * Collect information about a positional parameter in a literal context. Note: The return value
+ * is not used; this method operates via side effects.
+ */
+ override def visitPosParameterLiteral(ctx: PosParameterLiteralContext): AnyRef =
Review Comment:
It is easiest to let them all be handled in the preparser. The original code remains for now for safetyy. But I see no reason to keep it. If at soem point we want to move into plan sharing thsi code gets more interesting again. But then we we would introduce templates/markers that weren't even given by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426083025
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ */
+object LiteralToSqlConverter {
Review Comment:
I'm a bit worried about puting complex SQL string into the original SQL statement. The previous way of parameter binding happens at the analysis phase and changes the plan tree directly, which is 100% safe from SQL injection. The pre-parser accepts parameter markers in two more places: integer literal and string literal. I think it's better to only handle this two new places by the pre-parser, and still use the previous analysis-time parameter binding for others. It's safe to do SQL string manipulation only for integer and string literals, but arbitrary SQL generated by complex literals looks risky to me.
cc @dtenedor as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2426035585
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.trees.ParameterSubstitutionInfo
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ val identityMapper = PositionMapper.identity(sqlText)
+ setupSubstitutionContext(sqlText, sqlText, identityMapper, isIdentity = true)
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ setupSubstitutionContext(sqlText, substituted, positionMapper, isIdentity = false)
+ substituted
+ }
+
+ /**
+ * Set up parameter substitution context in CurrentOrigin for error reporting.
+ * This creates a ParameterSubstitutionInfo object with position mapping data.
+ *
+ * @param originalSql The original SQL text before substitution
+ * @param substitutedSql The SQL text after substitution
+ * @param positionMapper The mapper for translating positions
+ * @param isIdentity Whether this is an identity mapping (no substitution occurred)
+ */
+ private[sql] def setupSubstitutionContext(
+ originalSql: String,
+ substitutedSql: String,
+ positionMapper: PositionMapper,
+ isIdentity: Boolean): Unit = {
+
+ // Create parameter substitution info with the position mapper.
+ val substitutionInfo = ParameterSubstitutionInfo(
+ originalSql = originalSql,
+ isIdentity = isIdentity,
+ positionMapper = if (isIdentity) None else Some(positionMapper)
+ )
+
+ // Store the info in CurrentOrigin for position mapping during error reporting.
+ val currentOrigin = org.apache.spark.sql.catalyst.trees.CurrentOrigin.get
+ val updatedOrigin = currentOrigin.copy(parameterSubstitutionInfo = Some(substitutionInfo))
+ org.apache.spark.sql.catalyst.trees.CurrentOrigin.set(updatedOrigin)
+ }
+
+ /**
+ * Convert a value to an Expression, handling different input types safely.
+ * Uses pattern matching for type safety instead of unsafe casting.
+ *
+ * @param value The value to convert (can be Literal, Expression, or raw value)
+ * @return An Expression representing the value
+ */
+ private def convertToExpression(value: Any): Expression = value match {
+ case literal: Literal =>
+ // Already a Literal from ResolveExecuteImmediate - use it directly.
+ literal
+ case expr: Expression =>
+ // Expression from Column object - use it directly.
+ expr
+ case null =>
+ // Handle null values explicitly.
+ Literal(null)
+ case other =>
+ // Raw value - create new Literal with proper error handling.
+ try {
+ Literal(other)
+ } catch {
+ case _: RuntimeException =>
+ // Fallback for unsupported types.
+ throw new IllegalArgumentException(
+ s"Cannot convert value of type ${other.getClass.getSimpleName} to Expression: $other")
+ }
+ }
+
+ /**
+ * Convert an Expression to SQL string representation.
+ *
+ * @param expr The expression to convert
+ * @return SQL string representation
+ */
+ private def convertToSql(expr: Expression): String = {
+ LiteralToSqlConverter.convert(expr)
+ }
+
+ /**
+ * Substitute parameters in SQL text based on the parameter context.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param context The parameter context (named or positional)
+ * @return The SQL text with parameters substituted
+ * @throws IllegalArgumentException if context is null
+ */
+ def substituteParameters(
+ sqlText: String,
+ context: ParameterContext): String = {
+ require(context != null, "Parameter context cannot be null")
+
+ context match {
+ case NamedParameterContext(params) =>
+ performSubstitution(sqlText, namedParams = params.map { case (name, expr) =>
+ (name, convertToSql(expr))
+ })
+
+ case PositionalParameterContext(params) =>
+ performSubstitution(sqlText,
+ positionalParams = params.map(expr =>
+ convertToSql(expr)).toList)
+
+ case HybridParameterContext(args, paramNames) =>
+ handleHybridParameters(sqlText, args.toSeq, paramNames.toSeq)
+ }
+ }
+
+ /**
+ * Handle hybrid parameter context which can contain both named and positional parameters.
+ * Validates parameter consistency and routes to appropriate substitution method.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param args The parameter values
+ * @param paramNames The parameter names (empty string for positional parameters)
+ * @return The SQL text with parameters substituted
+ */
+ private def handleHybridParameters(
+ sqlText: String,
+ args: Seq[Any],
+ paramNames: Seq[String]): String = {
+ // Detect which parameter types are actually used in the query.
+ val (hasPositional, hasNamed) = detectParameters(sqlText)
+
+ // Validate that the query doesn't mix parameter types.
+ if (hasPositional && hasNamed) {
+ throw QueryCompilationErrors.invalidQueryMixedQueryParameters()
+ }
+
+ // Validate ALL_PARAMETERS_MUST_BE_NAMED: if query uses named parameters,
+ // all USING expressions must have names.
+ if (hasNamed && !hasPositional) {
+ validateAllParametersNamed(args, paramNames)
+ }
+
+ if (hasPositional && !hasNamed) {
+ // Query uses only positional parameters.
+ val positionalParams = args.map(convertToExpression)
+ performSubstitution(sqlText,
+ positionalParams = positionalParams.map(convertToSql).toList)
+ } else if (hasNamed && !hasPositional) {
+ // Query uses only named parameters.
+ val namedParams = paramNames.zip(args).collect {
+ case (name, value) if name.nonEmpty =>
+ name -> convertToExpression(value)
+ }.toMap
+ performSubstitution(sqlText,
+ namedParams = namedParams.map { case (name, expr) =>
+ (name, convertToSql(expr))
+ })
+ } else {
+ // No parameters in query - return as-is.
+ performSubstitution(sqlText)
Review Comment:
```suggestion
sqlText
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2433344335
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,9 +54,152 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext), isTopLevel = true)(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None, isTopLevel = true)(toResult)
}
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param isTopLevel Whether this is a top-level parse (vs identifier/data type parsing)
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext],
+ isTopLevel: Boolean)
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Check if parameter substitution should occur.
+ val (paramSubstituted, substitutionOccurred, hasParameters) = parameterContext match {
+ case Some(context) if isTopLevel =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: skip parameter substitution but still detect parameters for context.
+ // Parameters detected but substitution skipped in legacy mode.
+ // Position mapping will be set up below.
+ (command, false, true)
+ } else {
+ // Modern mode: perform parameter substitution if parameters are present.
+ val substituted = substituteParametersOrSetupCallback(command, context)
+ (substituted, substituted != command, true) // Track if substitution occurred.
+ }
+ case _ =>
+ // No parameter context or not top-level - no parameter substitution.
+ (command, false, false)
+ }
+
+ // Step 2: Apply existing variable substitution.
+ val variableSubstituted = substitutor.substitute(paramSubstituted)
Review Comment:
No problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436032064
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -50,8 +54,110 @@ class SparkSqlParser extends AbstractSqlParser {
private val substitutor = new VariableSubstitution()
+ /**
+ * Parse SQL with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL with parameters.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ def parseWithParameters[T](
+ command: String,
+ parameterContext: ParameterContext)
+ (toResult: SqlBaseParser => T): T = {
+ parseInternal(command, Some(parameterContext))(toResult)
+ }
+
+ /**
+ * Parse SQL plan with explicit parameter context, avoiding thread-local usage.
+ * This is the preferred method for parsing SQL plans with parameters.
+ *
+ * @param sqlText The SQL text to parse
+ * @param parameterContext The parameter context containing parameter values
+ * @return The parsed logical plan
+ */
+ override def parsePlanWithParameters(
+ sqlText: String,
+ parameterContext: ParameterContext): LogicalPlan = {
+ parseWithParameters(sqlText, parameterContext) { parser =>
+ val ctx = parser.compoundOrSingleStatement()
+ withErrorHandling(ctx, Some(sqlText)) {
+ astBuilder.visitCompoundOrSingleStatement(ctx) match {
+ case compoundBody: CompoundPlanStatement => compoundBody
+ case plan: LogicalPlan => plan
+ case _ =>
+ val position = Origin(None, None)
+ throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ }
+ }
+ }
+ }
+
protected override def parse[T](command: String)(toResult: SqlBaseParser => T): T = {
- super.parse(substitutor.substitute(command))(toResult)
+ parseInternal(command, None)(toResult)
+ }
+
+ /**
+ * Internal parse method that handles both parameter substitution and regular parsing.
+ *
+ * @param command The SQL text to parse
+ * @param parameterContext Optional parameter context for parameter substitution
+ * @param toResult Function to convert the parser result
+ * @return The parsed result
+ */
+ private def parseInternal[T](
+ command: String,
+ parameterContext: Option[ParameterContext])
+ (toResult: SqlBaseParser => T): T = {
+
+ // Step 1: Apply variable substitution to expand any variable references.
+ val variableSubstituted = substitutor.substitute(command)
+
+ // Step 2: Apply parameter substitution if a parameter context is provided.
+ val (paramSubstituted, positionMapper, hasParameters) = parameterContext match {
+ case Some(context) =>
+ if (SQLConf.get.legacyParameterSubstitutionConstantsOnly) {
+ // Legacy mode: Parameters are detected but substitution is deferred to analysis phase.
+ (variableSubstituted, None, true)
+ } else {
+ // Modern mode: Perform parameter substitution during parsing.
+ val (substituted, mapper) =
+ ParameterHandler.substituteParameters(variableSubstituted, context)
+ (substituted, mapper, true)
+ }
+ case None =>
+ // No parameter context provided; skip parameter substitution.
+ (variableSubstituted, None, false)
+ }
+
+ // Step 3: Set up the origin with SQL text and position mapper to enable
+ // parameter-aware error reporting.
+ val currentOrigin = CurrentOrigin.get
+ val originToUse = if (positionMapper.isDefined || hasParameters) {
+ // Set up origin with the substituted SQL text and position mapper for
+ // proper error reporting.
+ currentOrigin.copy(
+ sqlText = Some(paramSubstituted),
+ startIndex = Some(0),
+ stopIndex = Some(paramSubstituted.length - 1),
+ positionMapper = positionMapper
+ )
+ } else {
+ // No substitution occurred; use the existing origin unchanged.
+ currentOrigin
+ }
+
+ // Parse with the origin containing position mapper, ensuring cleanup afterwards.
+ CurrentOrigin.withOrigin(originToUse) {
+ try {
Review Comment:
we can remove the try-catch, as `CurrentOrigin.withOrigin` already does it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2436034193
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:
##########
@@ -488,15 +573,36 @@ class SparkSession private(
args: Map[String, Any],
tracker: QueryPlanningTracker): DataFrame =
withActive {
- val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
- val parsedPlan = sessionState.sqlParser.parsePlan(sqlText)
- if (args.nonEmpty) {
- NameParameterizedQuery(parsedPlan, args.transform((_, v) => lit(v).expr))
- } else {
- parsedPlan
+ // Always set parameter context if we have actual parameters
+ if (args.nonEmpty) {
+ // Resolve and validate parameters first
+ val resolvedParams = resolveAndValidateParameters(args.transform((_, v) => lit(v).expr))
+ val paramContext = NamedParameterContext(resolvedParams)
+ val plan = tracker.measurePhase(QueryPlanningTracker.PARSING) {
+ val parsedPlan = sessionState.sqlParser.parsePlanWithParameters(sqlText, paramContext)
+ // In legacy mode, wrap the parsed plan with NameParameterizedQuery
+ // so that the BindParameters analyzer rule can bind the parameters
+ if (sessionState.conf.legacyParameterSubstitutionConstantsOnly) {
+ NameParameterizedQuery(parsedPlan, paramContext.params)
+ } else {
+ parsedPlan
+ }
+ }
+
+ Dataset.ofRows(self, plan, tracker)
+ } else {
+ // Reset position mapper to prevent contamination from previous parameterized sql() calls
Review Comment:
I think this is no longer needed, as we reset it right after the parsing: https://github.com/apache/spark/pull/52334/files#diff-77a9aad2da3dc60210a2c4d2f3165d5f1d0acd54ca4811072a053225170ed748R153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2429670820
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala:
##########
@@ -84,23 +86,44 @@ case class ResolveExecuteImmediate(sparkSession: SparkSession, catalogManager: C
// Extract the query string from the queryParam expression
val sqlString = extractQueryString(sqlStmtStr)
- // Parse and validate the query
- val parsedPlan = sparkSession.sessionState.sqlParser.parsePlan(sqlString)
- validateQuery(sqlString, parsedPlan)
+ // Create the origin for EXECUTE IMMEDIATE context - this will be used by expressions
+ // during parsing to set their queryContext, similar to how views work
+ val executeImmediateOrigin = Origin(
+ objectType = Some("EXECUTE IMMEDIATE"),
+ objectName = None, // No named object for EXECUTE IMMEDIATE, unlike views
+ sqlText = Some(sqlString),
+ startIndex = Some(0),
+ stopIndex = Some(sqlString.length - 1)
+ )
+
+ // Execute the query recursively with isolated local variable context and EXECUTE IMMEDIATE
+ // origin. The isolation must cover parsing, analysis, and execution phases.
+ // CurrentOrigin.withOrigin ensures expressions created during parsing get the proper context
+ val result = withIsolatedLocalVariableContext {
+ CurrentOrigin.withOrigin(executeImmediateOrigin) {
+ // Parse and validate the query inside the isolation context to ensure
+ // local variables are not accessible during parsing
+ if (args.isEmpty) {
+ val parsedPlan = sparkSession.sessionState.sqlParser.parsePlan(sqlString)
Review Comment:
we never use this `parsedPlan`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2415179182
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2200,7 +2200,8 @@ class AstBuilder extends DataTypeAstBuilder
Limit(expression(ctx.expression), query)
case ctx: SampleByPercentileContext =>
- val fraction = ctx.percentage.getText.toDouble
+ val fraction = if (ctx.DECIMAL_VALUE() != null) { ctx.DECIMAL_VALUE().getText.toDouble }
Review Comment:
you can do this to skip repeating the token:
```
Option(ctx.DECIMAL_VALUE()).map { v =>
v.getText.toDouble
}.getOrElse {
ctx.integerValue().getText.toDouble
}
```
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -234,4 +276,71 @@ trait SparkParserUtils {
}
}
-object SparkParserUtils extends SparkParserUtils
+object SparkParserUtils extends SparkParserUtils {
+
+ /**
+ * Callback type for parameter substitution integration.
+ *
+ * This callback allows the parameter substitution system to provide origin adjustment without
+ * creating circular dependencies between modules.
+ *
+ * @param startToken
+ * The start token from substituted SQL
+ * @param stopToken
+ * The stop token from substituted SQL
+ * @param substitutedSql
+ * The substituted SQL text
+ * @param objectType
+ * The object type
+ * @param objectName
+ * The object name
+ * @return
+ * Some(origin) if parameter substitution should be applied, None otherwise
+ */
+ type ParameterSubstitutionCallback =
+ (Token, Token, String, Option[String], Option[String]) => Option[Origin]
+
+ /**
+ * Thread-local callback for parameter substitution integration. This is set by the parameter
+ * handler when parameter substitution occurs.
+ */
+ private val parameterSubstitutionCallbackStorage =
+ new ThreadLocal[Option[ParameterSubstitutionCallback]]() {
+ override def initialValue(): Option[ParameterSubstitutionCallback] = None
+ }
+
+ /**
+ * Get the current parameter substitution callback.
+ */
+ def parameterSubstitutionCallback: Option[ParameterSubstitutionCallback] = {
+ parameterSubstitutionCallbackStorage.get()
+ }
+
+ /**
+ * Set the parameter substitution callback for the current thread. This should be called by
+ * parameter handlers before parsing.
+ */
+ def setParameterSubstitutionCallback(callback: ParameterSubstitutionCallback): Unit = {
+ parameterSubstitutionCallbackStorage.set(Some(callback))
+ }
+
+ /**
+ * Clear the parameter substitution callback for the current thread.
+ */
+ def clearParameterSubstitutionCallback(): Unit = {
+ parameterSubstitutionCallbackStorage.remove()
+ }
+
+ /**
+ * Execute a block with a parameter substitution callback set.
+ */
+ def withParameterSubstitutionCallback[T](
Review Comment:
This is not used anywhere in the PR as of now?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
+ val ctx = try {
+ try {
+ // First attempt: SLL mode with bail error strategy
+ parser.setErrorHandler(new SparkParserBailErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ } catch {
+ case e: ParseCancellationException =>
+ // Second attempt: LL mode with full error strategy
+ tokenStream.seek(0) // rewind input stream
+ parser.reset()
+ parser.setErrorHandler(new SparkParserErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ parser.compoundOrSingleStatement()
+ }
+ } catch {
+ case e: Throwable => throw e
+ }
+ val parameterLocations = astBuilder.extractParameterLocations(ctx)
+
+ // Substitute parameters in the original text
+ val (substitutedSql, appliedSubstitutions) = substituteAtLocations(sqlText, parameterLocations,
+ namedParams, positionalParams)
+ val consumedPositionalParams = parameterLocations.positionalParameterLocations.length
+
+ // Create position mapper for error context translation
+ val positionMapper = PositionMapper(sqlText, substitutedSql, appliedSubstitutions)
+
+ (substitutedSql, consumedPositionalParams, positionMapper)
+ }
+
+ /**
+ * Detects parameter markers in SQL text without performing substitution.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text to analyze
+ * @return A tuple of (hasPositionalParameters, hasNamedParameters)
+ */
+ def detectParameters(sqlText: String): (Boolean, Boolean) = {
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (false, false)
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
+ val ctx = try {
+ try {
+ // First attempt: SLL mode with bail error strategy
+ parser.setErrorHandler(new SparkParserBailErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ } catch {
+ case e: ParseCancellationException =>
+ // Second attempt: LL mode with full error strategy
+ tokenStream.seek(0) // rewind input stream
+ parser.reset()
+ parser.setErrorHandler(new SparkParserErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ parser.compoundOrSingleStatement()
+ }
+ } catch {
+ case e: Throwable => throw e
Review Comment:
do we need this try/catch block if we just throw every exception again? We could just omit it for the same behavior?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): (String, Int, PositionMapper) = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ return (sqlText, 0, PositionMapper.identity(sqlText))
+ }
+
+ val lexer = new SqlBaseLexer(new UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ lexer.addErrorListener(ParseErrorListener)
+
+ val tokenStream = new CommonTokenStream(lexer)
+ val parser = new SqlBaseParser(tokenStream)
+ // Match main parser configuration for consistent error messages
+ parser.addParseListener(PostProcessor)
+ parser.addParseListener(UnclosedCommentProcessor(sqlText, tokenStream))
+ parser.removeErrorListeners()
+ parser.addErrorListener(ParseErrorListener)
+ parser.legacy_setops_precedence_enabled = SQLConf.get.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled = SQLConf.get.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = SQLConf.get.enforceReservedKeywords
+ parser.double_quoted_identifiers = SQLConf.get.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled = !SQLConf.get.legacyParameterSubstitutionConstantsOnly
+
+ val astBuilder = new SubstituteParmsAstBuilder()
+
+ // Use the same two-stage parsing strategy as the main parser for consistent error messages
+ val ctx = try {
+ try {
+ // First attempt: SLL mode with bail error strategy
+ parser.setErrorHandler(new SparkParserBailErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ } catch {
+ case e: ParseCancellationException =>
+ // Second attempt: LL mode with full error strategy
+ tokenStream.seek(0) // rewind input stream
+ parser.reset()
+ parser.setErrorHandler(new SparkParserErrorStrategy())
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ parser.compoundOrSingleStatement()
+ }
+ } catch {
+ case e: Throwable => throw e
Review Comment:
This just re-throws the exception unchanged; do we need the try/catch block at all? The behavior could be the same without it?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.antlr.v4.runtime.{CharStreams, CommonTokenStream}
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.internal.SQLConf
+
+
+/**
+ * A parameter substitution parser that replaces parameter markers in SQL text with their values.
+ * This parser finds parameter markers and substitutes them with provided values to produce
+ * a modified SQL string ready for execution.
+ */
+class SubstituteParamsParser extends Logging {
+
+ /**
+ * Substitute parameter markers in SQL text with provided values.
+ * Always uses compoundOrSingleStatement parsing which can handle all SQL constructs.
+ *
+ * @param sqlText The original SQL text containing parameter markers
+ * @param namedParams Map of named parameter values (paramName -> value)
+ * @param positionalParams List of positional parameter values in order
+ * @return A tuple of (modified SQL string with parameters substituted,
+ * number of consumed positional parameters)
+ */
+ def substitute(
+ sqlText: String,
Review Comment:
please fix formatting here?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2257,7 +2258,7 @@ class AstBuilder extends DataTypeAstBuilder
override def visitVersion(ctx: VersionContext): Option[String] = {
if (ctx != null) {
- if (ctx.INTEGER_VALUE != null) {
+ if (ctx.INTEGER_VALUE() != null) {
Review Comment:
same here, you can do:
```
Option(ctx.INTEGER_VALUE()).map { v =>
Some(v.getText)
}.getOrElse {
Option(string(visitStringLit(ctx.stringLit())))
}
```
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -234,4 +276,71 @@ trait SparkParserUtils {
}
}
-object SparkParserUtils extends SparkParserUtils
+object SparkParserUtils extends SparkParserUtils {
+
+ /**
+ * Callback type for parameter substitution integration.
+ *
+ * This callback allows the parameter substitution system to provide origin adjustment without
+ * creating circular dependencies between modules.
+ *
+ * @param startToken
+ * The start token from substituted SQL
+ * @param stopToken
+ * The stop token from substituted SQL
+ * @param substitutedSql
+ * The substituted SQL text
+ * @param objectType
+ * The object type
+ * @param objectName
+ * The object name
+ * @return
+ * Some(origin) if parameter substitution should be applied, None otherwise
+ */
+ type ParameterSubstitutionCallback =
+ (Token, Token, String, Option[String], Option[String]) => Option[Origin]
+
+ /**
+ * Thread-local callback for parameter substitution integration. This is set by the parameter
+ * handler when parameter substitution occurs.
+ */
+ private val parameterSubstitutionCallbackStorage =
+ new ThreadLocal[Option[ParameterSubstitutionCallback]]() {
Review Comment:
I wonder if there is any way we can set the callback using the parsing API rather than a thread local like this. This could make it more complex to support parallel parsing later, for example. If the parsing entry point is the `parsePlan` method of AbstractSqlParser, that introduces a call site with stack frames creating state, currently starting with the SQL string to parse. Is there any way we could pass this callback as part of that state into the stack frames instead of using a thread local?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala:
##########
@@ -90,15 +90,17 @@ abstract class AbstractSqlParser extends AbstractParser with ParserInterface {
}
/** Creates LogicalPlan for a given SQL string. */
- override def parsePlan(sqlText: String): LogicalPlan = parse(sqlText) { parser =>
- val ctx = parser.compoundOrSingleStatement()
- withErrorHandling(ctx, Some(sqlText)) {
- astBuilder.visitCompoundOrSingleStatement(ctx) match {
- case compoundBody: CompoundPlanStatement => compoundBody
- case plan: LogicalPlan => plan
- case _ =>
- val position = Origin(None, None)
- throw QueryParsingErrors.sqlStatementUnsupportedError(sqlText, position)
+ override def parsePlan(sqlText: String): LogicalPlan = {
Review Comment:
It looks like this part of the code in the file has not changed? If not, let's revert it to simplify the PR and help make it easier for people's Spark forks (if any) to merge it later?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -36,7 +36,51 @@ case class Origin(
objectType: Option[String] = None,
objectName: Option[String] = None,
stackTrace: Option[Array[StackTraceElement]] = None,
- pysparkErrorContext: Option[(String, String)] = None) {
+ pysparkErrorContext: Option[(String, String)] = None,
+ parameterSubstitutionCallback: Option[Any] = None) { // Store callback to avoid dependencies.
Review Comment:
I am wondering what dependencies we have to avoid that make it necessary to pass this as a callback instead of just invoking the necessary code directly? The parser lives in the Catalyst package, and it should be possible to import and directly call/use any other code in the Catalyst package. Can we do that instead of passing a callback around? It would simplify the logic a lot.
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -234,4 +302,15 @@ trait SparkParserUtils {
}
}
-object SparkParserUtils extends SparkParserUtils
+object SparkParserUtils extends SparkParserUtils {
+
+ /**
+ * Type alias for parameter substitution callback function. Takes (startToken, stopToken,
+ * substitutedSql, objectType, objectName) and returns an optional Origin.
+ */
+ type ParameterSubstitutionCallback =
+ (Token, Token, String, Option[String], Option[String]) => Option[Origin]
+
+ // Note: Thread-local callback mechanism removed - parameter substitution info
Review Comment:
OK, it is good that we're not using a ThreadLocal to hold the callback anymore because that's a global variable that is brittle to threading changes. We can remove this comment now :)
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:
##########
@@ -187,25 +187,93 @@ trait SparkParserUtils {
* Register the origin of the context. Any TreeNode created in the closure will be assigned the
* registered origin. This method restores the previously set origin after completion of the
* closure.
+ *
+ * This method is parameter substitution-aware. If parameter substitution occurred before
+ * parsing, it will automatically adjust the positions and SQL text to refer to the original SQL
+ * (before substitution) instead of the substituted SQL.
*/
def withOrigin[T](ctx: ParserRuleContext, sqlText: Option[String] = None)(f: => T): T = {
val current = CurrentOrigin.get
val text = sqlText.orElse(current.sqlText)
+
if (text.isEmpty) {
CurrentOrigin.set(position(ctx.getStart))
} else {
- CurrentOrigin.set(
- positionAndText(
- ctx.getStart,
- ctx.getStop,
- text.get,
- current.objectType,
- current.objectName))
+ // Check if parameter substitution occurred and adjust origin accordingly.
+ val adjustedOrigin = adjustOriginForParameterSubstitution(
+ ctx.getStart,
+ ctx.getStop,
+ text.get,
+ current.objectType,
+ current.objectName)
+
+ // Preserve any existing callback when setting the new origin.
+ val finalOrigin = if (current.parameterSubstitutionCallback.isDefined) {
+ adjustedOrigin.copy(parameterSubstitutionCallback = current.parameterSubstitutionCallback)
+ } else {
+ adjustedOrigin
+ }
+
+ CurrentOrigin.set(finalOrigin)
}
try {
f
} finally {
- CurrentOrigin.set(current)
+ // When restoring origin, preserve any callback that was added during parsing.
+ val currentAfterParsing = CurrentOrigin.get
+ val originToRestore =
+ if (currentAfterParsing.parameterSubstitutionCallback.isDefined ||
+ current.parameterSubstitutionCallback.isDefined) {
+ // Either the current or the original has a callback - preserve it.
+ val callbackToPreserve = currentAfterParsing.parameterSubstitutionCallback
+ .orElse(current.parameterSubstitutionCallback)
+ current.copy(parameterSubstitutionCallback = callbackToPreserve)
+ } else {
+ // Neither has a callback - restore as normal.
+ current
+ }
+
+ CurrentOrigin.set(originToRestore)
+ }
+ }
+
+ /**
+ * Adjust origin information to account for parameter substitution.
+ *
+ * If parameter substitution occurred, this method maps positions from the substituted SQL back
+ * to the original SQL and uses the original SQL text in the origin.
+ *
+ * @param startToken
+ * The start token from the substituted SQL
+ * @param stopToken
+ * The stop token from the substituted SQL
+ * @param substitutedSql
+ * The SQL text after substitution
+ * @param objectType
+ * The object type for the origin
+ * @param objectName
+ * The object name for the origin
+ * @return
+ * Origin with positions and text adjusted for parameter substitution
+ */
+ private def adjustOriginForParameterSubstitution(
+ startToken: Token,
+ stopToken: Token,
+ substitutedSql: String,
+ objectType: Option[String],
+ objectName: Option[String]): Origin = {
+
+ // Try to get parameter substitution callback from CurrentOrigin.
+ CurrentOrigin.get.parameterSubstitutionCallback match {
+ case Some(callback) =>
+ // Cast the callback from Any to the proper type.
+ val typedCallback = callback.asInstanceOf[SparkParserUtils.ParameterSubstitutionCallback]
Review Comment:
can we just store it as an `Option[SparkParserUtils.ParameterSubstitutionCallback]` instead of `Option[Any]` for better type safety?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala:
##########
@@ -49,10 +53,132 @@ class SparkSqlParser extends AbstractSqlParser {
val astBuilder = new SparkSqlAstBuilder()
private val substitutor = new VariableSubstitution()
+ private[execution] val parameterHandler = new ParameterHandler()
+
+ // Thread-local flag to track whether we're in a top-level parse operation
+ // This is used to prevent parameter substitution during identifier/data type parsing
+ private val isTopLevelParse = new ThreadLocal[Boolean] {
Review Comment:
This is also introducing a new ThreadLocal. Can we avoid it?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LiteralToSqlConverter.scala:
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.{InternalRow}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedFunction
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.types._
+
+/**
+ * Utility for converting Catalyst literal expressions to their SQL string representation.
+ *
+ * This object provides a specialized implementation for converting Spark SQL literal
+ * expressions to their equivalent SQL text representation. It is used by the parameter
+ * substitution system for EXECUTE IMMEDIATE and other parameterized queries.
+ *
+ * Key features:
+ * - Handles all Spark SQL data types for literal values
+ * - Supports both Scala collections and Spark internal data structures
+ * - Proper SQL escaping and formatting
+ * - Optimized for literal expressions only
+ *
+ * Supported data types:
+ * - Primitives: String, Integer, Long, Float, Double, Boolean, Decimal
+ * - Temporal: Date, Timestamp, TimestampNTZ, Interval
+ * - Collections: Array, Map (including nested structures)
+ * - Special: Null values, Binary data
+ * - Complex: Nested arrays, maps of arrays, arrays of maps
+ *
+ * @example Basic usage:
+ * {{{
+ * val result1 = LiteralToSqlConverter.convert(Literal(42))
+ * // result1: "42"
+ *
+ * val result2 = LiteralToSqlConverter.convert(Literal("hello"))
+ * // result2: "'hello'"
+ *
+ * val arrayLit = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+ * val result3 = LiteralToSqlConverter.convert(arrayLit)
+ * // result3: "ARRAY(1, 2, 3)"
+ * }}}
+ *
+ * @example Complex types:
+ * {{{
+ * val mapLit = Literal.create(Map("key" -> "value"), MapType(StringType, StringType))
+ * val result = LiteralToSqlConverter.convert(mapLit)
+ * // result: "MAP('key', 'value')"
+ * }}}
+ *
+ * @note This utility is thread-safe and can be used concurrently.
+ * @note Only supports Literal expressions - all parameter values must be pre-evaluated.
+ * @see [[ParameterHandler]] for the main parameter handling entry point
+ * @since 4.0.0
+ */
+object LiteralToSqlConverter {
+
+ /**
+ * Convert an expression to its SQL string representation.
+ *
+ * This method handles both simple literals and complex expressions that result from
+ * parameter evaluation. For complex types like arrays and maps, the expressions are
+ * evaluated to internal data structures that need to be converted back to SQL constructors.
+ *
+ * @param expr The expression to convert (typically a Literal, but may be other expressions
+ * for complex types)
+ * @return SQL string representation of the expression value
+ */
+ def convert(expr: Expression): String = expr match {
+ case lit: Literal => convertLiteral(lit)
+
+ // Special handling for UnresolvedFunction expressions that don't naturally evaluate
+ // Only handle functions that are whitelisted in legacy mode but don't eval() naturally
+ case UnresolvedFunction(name, children, _, _, _, _, _) =>
+ val functionName = name.mkString(".")
+ functionName.toLowerCase(java.util.Locale.ROOT) match {
+ case "array" | "map" | "struct" | "map_from_arrays" | "map_from_entries" =>
+ // Convert whitelisted functions to SQL function call syntax
+ val childrenSql = children.map(convert).mkString(", ")
+ s"${functionName.toUpperCase(java.util.Locale.ROOT)}($childrenSql)"
+ case _ =>
+ // Non-whitelisted function - not supported in parameter substitution
+ throw QueryCompilationErrors.unsupportedParameterExpression(expr)
+ }
+
+ case _ =>
+ // For non-literal expressions, they should be resolved before reaching this converter
+ // If we get an unresolved expression, it indicates a problem in the calling code
+ if (!expr.resolved) {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter received unresolved expression: " +
+ s"${expr.getClass.getSimpleName}. All expressions should be resolved before " +
+ s"parameter conversion.")
+ }
+ if (expr.foldable) {
+ val value = expr.eval()
+ val dataType = expr.dataType
+ convertLiteral(Literal.create(value, dataType))
+ } else {
+ throw SparkException.internalError(
+ s"LiteralToSqlConverter cannot convert non-foldable expression: " +
+ s"${expr.getClass.getSimpleName}. All parameter values should be evaluable to " +
+ s"literals before conversion.")
+ }
+ }
+
+ private def convertLiteral(lit: Literal): String = {
+ // For simple cases, delegate to the existing Literal.sql method
+ // which already has the correct logic for most data types
+ try {
+ lit.sql
+ } catch {
+ case _: Exception =>
Review Comment:
This catches all possible exceptions and throws away their contents. It could be dangerous if it runs into a new exception type we don't expect in the future and we lose the information. Could we make this catch more specific subclasses of exceptions instead for better safety?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala:
##########
@@ -57,6 +101,44 @@ case class Origin(
*/
trait WithOrigin {
def origin: Origin
+
+ /**
+ * Update query contexts in this object with translated positions. Uses reflection to
+ * generically update any object with query context.
+ */
+ def updateQueryContext(translator: Array[QueryContext] => Array[QueryContext]): WithOrigin = {
+ try {
+ val thisClass = this.getClass
+
+ // Try to find query context using common method names
+ val contextMethodNames = Seq("getQueryContext", "context")
+
+ for (methodName <- contextMethodNames) {
+ try {
+ val getMethod = thisClass.getMethod(methodName)
+ val currentContexts = getMethod.invoke(this).asInstanceOf[Array[QueryContext]]
+ val translatedContexts = translator(currentContexts)
+
+ // Try to update the field in-place
+ val fieldName = if (methodName == "getQueryContext") "queryContext" else "context"
+ try {
+ val field = thisClass.getDeclaredField(fieldName)
+ field.setAccessible(true)
+ field.set(this, translatedContexts)
+ return this // Successfully updated in-place!
+ } catch {
+ case _: Exception => // Field update failed, continue to try other methods
+ }
+ } catch {
+ case _: Exception => // Method doesn't exist, try next
+ }
+ }
+
+ this // No update possible, return unchanged
+ } catch {
+ case _: Exception => this // Return unchanged if anything fails
Review Comment:
This is catching all possible exceptions, and dropping the contents of the exceptions. This could be dangerous if something unexpected happens and we lose the contents of the exception to know what it was. Could we possibly catch more specific exception subclasses instead for better safety, letting everything not explicitly enumerated fall through instead?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Uses CompoundOrSingleStatement parsing for all SQL constructs
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations with position-aware error context
+ *
+ * The handler integrates with the parser through callback mechanisms stored in
+ * CurrentOrigin to ensure error positions are correctly mapped back to the original SQL text.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Optional context:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = Some(NamedParameterContext(Map("param1" -> Literal(42))))
+ * val result = handler.substituteParametersIfNeeded("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ */
+class ParameterHandler {
+
+ // Compiled regex pattern for efficient parameter marker detection.
+ private val parameterMarkerPattern = java.util.regex.Pattern.compile("[?:]")
+
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely.
+ if (!parameterMarkerPattern.matcher(sqlText).find()) {
+ val identityMapper = PositionMapper.identity(sqlText)
+ setupSubstitutionContext(sqlText, sqlText, identityMapper, isIdentity = true)
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText,
+ namedParams = namedParams, positionalParams = positionalParams)
+
+ setupSubstitutionContext(sqlText, substituted, positionMapper, isIdentity = false)
+ substituted
+ }
+
+ /**
+ * Set up position mapping context for error reporting.
+ * Creates a callback function and stores it in CurrentOrigin for position translation
+ * between original and substituted SQL text.
+ *
+ * @param originalSql The original SQL text before parameter substitution
+ * @param substitutedSql The SQL text after parameter substitution
+ * @param positionMapper The position mapper for translating error positions
+ * @param isIdentity Whether this is an identity mapping (no substitution occurred)
+ */
+ private[sql] def setupSubstitutionContext(
+ originalSql: String,
+ substitutedSql: String,
+ positionMapper: PositionMapper,
+ isIdentity: Boolean): Unit = {
+
+ // Create callback function for position mapping.
+ // The positionMapper is captured in the closure for efficient position translation.
+ val callback: (org.antlr.v4.runtime.Token, org.antlr.v4.runtime.Token, String,
Review Comment:
If I understand correctly, this is the only actual parameter substitution callback we're proposing to pass in. This code is in the Catalyst package; could we just call it directly instead? That could simplify a lot of things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407241112
##########
sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala:
##########
@@ -833,4 +820,867 @@ class ParametersSuite extends QueryTest with SharedSparkSession {
checkAnswer(spark.sql("execute immediate 'select ?' using :param", Map("param" -> 2)), Row(2))
checkAnswer(spark.sql("execute immediate 'select :param' using ? as param", Array(3)), Row(3))
}
+
+ test("named parameters in DDL data type specifications") {
+ withTable("ddl_datatype_named") {
+ // Test VARCHAR length parameter
+ spark.sql("CREATE TABLE ddl_datatype_named (name VARCHAR(:length)) USING PARQUET",
+ Map("length" -> 100))
+ val schema = spark.table("ddl_datatype_named").schema
+ checkColType(schema("name"), VarcharType(100))
+ }
+
+ withTable("ddl_decimal_named") {
+ // Test DECIMAL precision and scale parameters
+ spark.sql(
+ "CREATE TABLE ddl_decimal_named (amount DECIMAL(:precision, :scale)) USING PARQUET",
+ Map("precision" -> 10, "scale" -> 2))
+ val schema = spark.table("ddl_decimal_named").schema
+ assert(schema("amount").dataType == DecimalType(10, 2))
+ }
+
+ withTable("ddl_char_named") {
+ // Test CHAR length parameter
+ spark.sql("CREATE TABLE ddl_char_named (code CHAR(:length)) USING PARQUET",
+ Map("length" -> 5))
+ val schema = spark.table("ddl_char_named").schema
+ checkColType(schema("code"), CharType(5))
+ }
+ }
+
+ test("positional parameters in DDL data type specifications") {
+ withTable("ddl_datatype_pos") {
+ // Test VARCHAR length parameter
+ spark.sql("CREATE TABLE ddl_datatype_pos (name VARCHAR(?)) USING PARQUET",
+ Array(150))
+ val schema = spark.table("ddl_datatype_pos").schema
+ checkColType(schema("name"), VarcharType(150))
+ }
+
+ withTable("ddl_decimal_pos") {
+ // Test DECIMAL precision and scale parameters
+ spark.sql(
+ "CREATE TABLE ddl_decimal_pos (amount DECIMAL(?, ?)) USING PARQUET",
+ Array(12, 3))
+ val schema = spark.table("ddl_decimal_pos").schema
+ assert(schema("amount").dataType == DecimalType(12, 3))
+ }
+
+ withTable("ddl_char_pos") {
+ // Test CHAR length parameter
+ spark.sql("CREATE TABLE ddl_char_pos (code CHAR(?)) USING PARQUET",
+ Array(8))
+ val schema = spark.table("ddl_char_pos").schema
+ checkColType(schema("code"), CharType(8))
+ }
+ }
+
+
Review Comment:
nit: empty lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "miland-db (via GitHub)" <gi...@apache.org>.
miland-db commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2407243292
##########
sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala:
##########
@@ -780,9 +767,9 @@ class ParametersSuite extends QueryTest with SharedSparkSession {
}
checkAnswer(spark.sql(query(":cte"), args = Map("cte" -> "t1")),
- Seq(Row(1), Row(2), Row(3), Row(4), Row(5)))
+ Array(Row(1), Row(2), Row(3), Row(4), Row(5)))
Review Comment:
Is this change needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53573][WIP][SQL] Use Pre-processor for generalized parameter marker handling [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #52334:
URL: https://github.com/apache/spark/pull/52334#discussion_r2388940258
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParameterHandler.scala:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.catalyst.trees.SQLQueryContext
+import org.apache.spark.sql.catalyst.util.LiteralToSqlConverter
+
+/**
+ * Handler for parameter substitution across different Spark SQL contexts.
+ *
+ * This class consolidates the common parameter handling logic used by SparkSqlParser,
+ * SparkConnectPlanner, and ExecuteImmediate. It provides a single, consistent API
+ * for all parameter substitution operations in Spark SQL.
+ *
+ * Key features:
+ * - Automatic parameter type detection (named vs positional)
+ * - Automatic substitution rule selection (Statement vs CompoundOrSingleStatement)
+ * - Consistent error handling and validation
+ * - Support for complex data types (arrays, maps, nested structures)
+ * - Thread-safe operations
+ *
+ * The handler uses the Strategy pattern internally to delegate to appropriate
+ * parameter substitution strategies based on the parameter types detected in the SQL.
+ *
+ * @example Basic usage:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ * val result = handler.substituteParameters("SELECT :param1", context)
+ * // result: "SELECT 42"
+ * }}}
+ *
+ * @example Automatic rule detection:
+ * {{{
+ * val handler = new ParameterHandler()
+ * val context = NamedParameterContext(Map("param1" -> Literal(42)))
+ *
+ * // Regular statement
+ * val result1 = handler.substituteParametersWithAutoRule("SELECT :param1", context)
+ *
+ * // SQL scripting block - automatically uses CompoundOrSingleStatement rule
+ * val result2 = handler.substituteParametersWithAutoRule("BEGIN SELECT :param1; END", context)
+ * }}}
+ *
+ * @see [[SubstituteParamsParser]] for the underlying parameter substitution logic
+ * @since 4.0.0
+ */
+class ParameterHandler {
+
+
+ // Thread-local storage for position mapper to ensure thread safety
+ private val positionMapperStorage = new ThreadLocal[Option[PositionMapper]]() {
+ override def initialValue(): Option[PositionMapper] = None
+ }
+
+ /**
+ * Get the current position mapper for translating error positions.
+ * This mapper can translate positions from substituted text back to original text.
+ *
+ * @return The current position mapper, if available
+ */
+ def getPositionMapper: Option[PositionMapper] = positionMapperStorage.get()
+
+ /**
+ * Helper method to perform parameter substitution and store position mapper.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param rule The substitution rule to use
+ * @param namedParams Optional named parameters map
+ * @param positionalParams Optional positional parameters list
+ * @return The SQL text with parameters substituted
+ */
+ private def performSubstitution(
+ sqlText: String,
+ rule: SubstitutionRule,
+ namedParams: Map[String, String] = Map.empty,
+ positionalParams: List[String] = List.empty): String = {
+
+ // Quick pre-check: if there are no parameter markers in the text, skip parsing entirely
+ if (!sqlText.contains("?") && !sqlText.contains(":")) {
+ positionMapperStorage.set(Some(PositionMapper.identity(sqlText)))
+ return sqlText
+ }
+
+ val substitutor = new SubstituteParamsParser()
+ val (substituted, _, positionMapper) = substitutor.substitute(sqlText, rule,
+ namedParams = namedParams, positionalParams = positionalParams)
+ positionMapperStorage.set(Some(positionMapper))
+ substituted
+ }
+
+ /**
+ * Translate error context from substituted text positions to original text positions.
+ *
+ * @param context The error context with positions in substituted text
+ * @return The error context with positions mapped to original text
+ */
+ def translateErrorContext(context: SQLQueryContext): SQLQueryContext = {
+ positionMapperStorage.get() match {
+ case Some(mapper) =>
+ val translatedStartIndex = context.originStartIndex.map(mapper.mapToOriginal)
+ val translatedStopIndex = context.originStopIndex.map(mapper.mapToOriginal)
+
+ context.copy(
+ originStartIndex = translatedStartIndex,
+ originStopIndex = translatedStopIndex,
+ sqlText = Some(mapper.originalText) // Use original text instead of substituted
+ )
+ case None =>
+ context // No position mapper available, return as-is
+ }
+ }
+
+ /**
+ * Substitute parameters in SQL text based on the parameter context.
+ *
+ * @param sqlText The SQL text containing parameter markers
+ * @param context The parameter context (named or positional)
+ * @param rule The substitution rule to use (default: Statement)
+ * @return The SQL text with parameters substituted
+ */
+ def substituteParameters(
+ sqlText: String,
+ context: ParameterContext,
+ rule: SubstitutionRule = SubstitutionRule.Statement): String = {
+
+ context match {
+ case NamedParameterContext(params) =>
+ performSubstitution(sqlText, rule, namedParams = params.map { case (name, expr) =>
+ (name, LiteralToSqlConverter.convert(expr))
Review Comment:
Constant expression. I'll revisit the naming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org