You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "learningchess2003 (via GitHub)" <gi...@apache.org> on 2023/06/01 22:39:52 UTC
[GitHub] [spark] learningchess2003 opened a new pull request, #41429: [SPARK-43922] Add named parameter support in parser for function calls
learningchess2003 opened a new pull request, #41429:
URL: https://github.com/apache/spark/pull/41429
### What changes were proposed in this pull request?
We plan on adding two new tokens called ```namedArgumentExpression``` and ```functionArgument``` which would enable this feature. When parsing this logic, we also make changes to ASTBuilder such that it can detect if the argument passed is a named argument or a positional one.
### Why are the changes needed?
This is part of a larger project to implement named parameter support for user defined functions, built-in functions, and table valued functions.
### Does this PR introduce _any_ user-facing change?
Yes, the user would be able to call functions with argument lists that contain named arguments.
### How was this patch tested?
We add tests in the PlanParserSuite that will verify that the plan parsed is as intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1226973650
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unevaluable unary expression specifically for named argument functions
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * NamedArgumentExpression is expected to be resolved
+ * and replaced in class extending [[NamedArgumentFunction]]
Review Comment:
please remove that
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2176,7 +2185,15 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
val name = ctx.functionName.getText
val isDistinct = Option(ctx.setQuantifier()).exists(_.DISTINCT != null)
// Call `toSeq`, otherwise `ctx.argument.asScala.map(expression)` is `Buffer` in Scala 2.13
- val arguments = ctx.argument.asScala.map(expression).toSeq match {
+ val arguments = ctx.argument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
+ val key = e.namedArgumentExpression.key.strictIdentifier
Review Comment:
I think you can just use `e.namedArgumentExpression.key.getText` for this since it is a single-part identifier in the grammar.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2176,7 +2185,15 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
val name = ctx.functionName.getText
val isDistinct = Option(ctx.setQuantifier()).exists(_.DISTINCT != null)
// Call `toSeq`, otherwise `ctx.argument.asScala.map(expression)` is `Buffer` in Scala 2.13
- val arguments = ctx.argument.asScala.map(expression).toSeq match {
+ val arguments = ctx.argument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
+ val key = e.namedArgumentExpression.key.strictIdentifier
+ val value = e.namedArgumentExpression.value
+ NamedArgumentExpression(key.getText, expression(value))
Review Comment:
This reveals the NamedArgumentExpression to the analyzer, but we haven't yet implemented resolution support for it. This should be OK since it is `Unevaluable`; let's add a couple analyzer or execution tests with the new syntax to show the errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1590434238
> This is part of a larger project to implement named parameter support for user defined functions, built-in functions, and table valued functions.
@learningchess2003 is there a design doc for the implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anchovYu commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "anchovYu (via GitHub)" <gi...@apache.org>.
anchovYu commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1228817177
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1580,6 +1580,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Named arguments support is currently disabled. Enable suppport using ALLOW_NAMED_FUNCTION_ARGUMENTS."
Review Comment:
Resolve since it's outdated.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -193,6 +193,13 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase {
messageParameters = Map("configKey" -> configKey))
}
+ def namedArgumentsNotEnabledError(argumentName: String): Throwable = {
+ new AnalysisException(
+ errorClass = "NAMED_ARGUMENTS_SUPPORT_DISABLED",
+ messageParameters = Map("message" -> (argumentName + " is a named argument."))
Review Comment:
Resolve since it's outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1234075810
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1513,6 +1513,19 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
}
}
+ private def extractExpression(expr: FunctionArgumentContext, funcName: String) : Expression = {
+ Option(expr.namedArgumentExpression).map { n =>
+ if (SQLConf.get.getConf(SQLConf.ALLOW_NAMED_FUNCTION_ARGUMENTS)) {
Review Comment:
```suggestion
if (conf.getConf(SQLConf.ALLOW_NAMED_FUNCTION_ARGUMENTS)) {
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -193,6 +193,14 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase {
messageParameters = Map("configKey" -> configKey))
}
+ def namedArgumentsNotEnabledError(functionName: String, argumentName: String): Throwable = {
+ new AnalysisException(
+ errorClass = "NAMED_ARGUMENTS_SUPPORT_DISABLED",
+ messageParameters = Map("functionName" -> toSQLId(functionName),
+ "argument" -> toSQLId(argumentName))
Review Comment:
nit:
```suggestion
messageParameters = Map(
"functionName" -> toSQLId(functionName),
"argument" -> toSQLId(argumentName))
```
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1580,6 +1580,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
Review Comment:
Sounds like it can be enabled. If so, could you describe how in the error message.
> ... are not supported here
If it is just not supported, could you add a sub-class to `UNSUPPORTED_FEATURE`
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1513,6 +1513,19 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
}
}
+ private def extractExpression(expr: FunctionArgumentContext, funcName: String) : Expression = {
+ Option(expr.namedArgumentExpression).map { n =>
+ if (SQLConf.get.getConf(SQLConf.ALLOW_NAMED_FUNCTION_ARGUMENTS)) {
+ NamedArgumentExpression(n.key.getText, expression(n.value))
+ } else {
+ throw QueryCompilationErrors.namedArgumentsNotEnabledError(
+ funcName, n.key.getText)
Review Comment:
```suggestion
throw QueryCompilationErrors.namedArgumentsNotEnabledError(funcName, n.key.getText)
```
##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryParsingErrorsSuite.scala:
##########
@@ -20,16 +20,33 @@ package org.apache.spark.sql.errors
import org.apache.spark.SparkThrowable
import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.plans.SQLHelper
+import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
import org.apache.spark.sql.test.SharedSparkSession
// Turn of the length check because most of the tests check entire error messages
// scalastyle:off line.size.limit
-class QueryParsingErrorsSuite extends QueryTest with SharedSparkSession {
+class QueryParsingErrorsSuite extends QueryTest with SharedSparkSession with SQLHelper {
private def parseException(sqlText: String): SparkThrowable = {
intercept[ParseException](sql(sqlText).collect())
}
+ test("NAMED_ARGUMENTS_SUPPORT_DISABLED: named arguments not turned on") {
+ withSQLConf("spark.sql.allowNamedFunctionArguments" -> "false") {
+ checkError(
+ exception = parseException("SELECT * FROM encode(value => 'abc', charset => 'utf-8')"),
+ errorClass = "NAMED_ARGUMENTS_SUPPORT_DISABLED",
+ parameters = Map("functionName" -> toSQLId("encode"), "argument" -> toSQLId("value"))
Review Comment:
nit: just enough to use the expected values instead of `toSQLId`
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * This represents an argument expression to a function call accompanied with an
+ * explicit reference to the corresponding argument name as a string. In this way,
+ * the analyzer can make sure that the provided values match up to the arguments
+ * as intended, and the arguments may appear in any order.
+ * This unary expression is unevaluable because we intend to replace it with
+ * the provided value itself during query analysis (after possibly rearranging
+ * the parsed argument list to match up the names to the expected function
+ * signature).
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * Example usage with the "encode" scalar function:
+ * SELECT encode("abc", charset => "utf-8");
+ * The second argument generates NamedArgumentExpression("charset", Literal("utf-8"))
+ * SELECT encode(charset => "utf-8", value => "abc");
+ *
+ * @param key The name of the function argument
+ * @param value The value of the function argument
+ */
+case class NamedArgumentExpression(key: String, value: Expression)
+ extends UnaryExpression with Unevaluable {
Review Comment:
see https://github.com/databricks/scala-style-guide#spacing-and-indentation
```suggestion
extends UnaryExpression with Unevaluable {
```
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -329,6 +330,27 @@ class ExpressionParserSuite extends AnalysisTest {
parameters = Map("error" -> "'x'", "hint" -> ": extra input 'x'"))
}
+ test("function expressions with named arguments") {
+ assertEqual("encode(value => 'abc', charset => 'utf-8')",
+ $"encode".function(NamedArgumentExpression("value", Literal("abc")),
+ NamedArgumentExpression("charset", Literal("utf-8"))))
Review Comment:
Please, fix indentation here and below according to https://github.com/databricks/scala-style-guide
```suggestion
$"encode".function(NamedArgumentExpression("value", Literal("abc")),
NamedArgumentExpression("charset", Literal("utf-8"))))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1243178821
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -329,6 +330,27 @@ class ExpressionParserSuite extends AnalysisTest {
parameters = Map("error" -> "'x'", "hint" -> ": extra input 'x'"))
}
+ test("function expressions with named arguments") {
+ assertEqual("encode(value => 'abc', charset => 'utf-8')",
+ $"encode".function(NamedArgumentExpression("value", Literal("abc")),
+ NamedArgumentExpression("charset", Literal("utf-8"))))
+ assertEqual("encode('abc', charset => 'utf-8')",
+ $"encode".function(Literal("abc"),
+ NamedArgumentExpression("charset", Literal("utf-8"))))
Review Comment:
Please, fix indentation here and below as you did above.
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1659,6 +1659,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Cannot call function <functionName> because named argument references are not enabled here. In this case, the named argument reference was <argument>. Set spark.sql.allowNamedFunctionArguments to true to turn on feature."
Review Comment:
Could you quote the config name and its value at the functions `toSQLConf()`/`toSQLConfVal` do.
```suggestion
"Cannot call function <functionName> because named argument references are not enabled here. In this case, the named argument reference was <argument>. Set \"spark.sql.allowNamedFunctionArguments\" to \"true\" to turn on feature."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 closed pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 closed pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
URL: https://github.com/apache/spark/pull/41429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1226984342
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2176,7 +2185,15 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
val name = ctx.functionName.getText
val isDistinct = Option(ctx.setQuantifier()).exists(_.DISTINCT != null)
// Call `toSeq`, otherwise `ctx.argument.asScala.map(expression)` is `Buffer` in Scala 2.13
- val arguments = ctx.argument.asScala.map(expression).toSeq match {
+ val arguments = ctx.argument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
+ val key = e.namedArgumentExpression.key.strictIdentifier
Review Comment:
I think you can just use `e.namedArgumentExpression.key.getText` for this since it is a single-part identifier in the grammar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1227109912
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -329,6 +329,18 @@ class ExpressionParserSuite extends AnalysisTest {
parameters = Map("error" -> "'x'", "hint" -> ": extra input 'x'"))
}
+ test("function expressions with named arguments") {
+ assertEqual("encode(value => 'abc', charset => 'utf-8')",
Review Comment:
Please also add positive tests with:
* constant but non-literal values, e.g. `'utf' || '-8'` or `abs(42)`
* double parentheses in a named argument, e.g. `arg1 => ((select 1))`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1599741003
@MaxGekk Addressed comments. Let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244387537
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1527,6 +1527,18 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
}
}
+ private def extractExpression(expr: FunctionArgumentContext, funcName: String) : Expression = {
Review Comment:
how about `extractFuncArgument`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1588242655
@dtenedor The configuration has been added. Other comments has been addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244077471
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1659,6 +1659,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Cannot call function <functionName> because named argument references are not enabled here. In this case, the named argument reference was <argument>. Set spark.sql.allowNamedFunctionArguments to true to turn on feature."
Review Comment:
Oh, sorry, I just realized that you meant it for functionName. This is good. I will commit this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1612189356
@MaxGekk Sounds good. I've added the end-to-end tests. In the next PR after this, most of the error messages will be replaced with the intended results. Right now, nothing is really supported yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1610265555
@cloud-fan Thanks for the review! Addressed the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1227284125
##########
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -921,7 +930,7 @@ primaryExpression
| LEFT_PAREN namedExpression (COMMA namedExpression)+ RIGHT_PAREN #rowConstructor
| LEFT_PAREN query RIGHT_PAREN #subqueryExpression
| IDENTIFIER_KW LEFT_PAREN expression RIGHT_PAREN #identifierClause
- | functionName LEFT_PAREN (setQuantifier? argument+=expression (COMMA argument+=expression)*)? RIGHT_PAREN
+ | functionName LEFT_PAREN (setQuantifier? argument+=functionArgument (COMMA argument+=functionArgument)*)? RIGHT_PAREN
Review Comment:
can you maybe fix formatting of these three lines so they break before the 94th column that the above labeks (e.g. `#identifierClause`) start on? This could make the parser logic easier to read.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1544,8 +1544,17 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
if (name.length > 1) {
throw QueryParsingErrors.invalidTableValuedFunctionNameError(name, ctx)
}
+ val args = func.functionArgument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
Review Comment:
You can simplify using `Option` and `map`:
```
Option(e.namedArgumentExpression).map { n =>
NamedArgumentExpression(n.get.getText, expression(nvalue))
}.getOrElse {
expression(e)
}
```
Same on L2188 below.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -329,6 +329,18 @@ class ExpressionParserSuite extends AnalysisTest {
parameters = Map("error" -> "'x'", "hint" -> ": extra input 'x'"))
}
+ test("function expressions with named arguments") {
+ assertEqual("encode(value => 'abc', charset => 'utf-8')",
Review Comment:
Great, thanks!
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala:
##########
@@ -1412,6 +1412,31 @@ class PlanParserSuite extends AnalysisTest {
assertEqual("select a, b from db.c; ;; ;", table("db", "c").select($"a", $"b"))
}
+ test("table valued function with named arguments") {
+ // All named arguments
Review Comment:
Great, thanks!
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1544,8 +1544,17 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
if (name.length > 1) {
throw QueryParsingErrors.invalidTableValuedFunctionNameError(name, ctx)
}
+ val args = func.functionArgument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
+ val key = e.namedArgumentExpression.key.getText
+ val value = e.namedArgumentExpression.value
+ NamedArgumentExpression(key, expression(value))
Review Comment:
Can you control this behavior with a SQLConf? You can create a new entry like `ALLOW_NAMED_FUNCTION_ARGUMENTS` and return an error here if the config is not set to true:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unevaluable unary expression specifically for named argument functions
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * Example usage in encode:
Review Comment:
```suggestion
* Example usage with the "encode" scalar function:
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unevaluable unary expression specifically for named argument functions
Review Comment:
```suggestion
* This represents an argument expression to a function call accompanied with an
* explicit reference to the corresponding argument name as a string. In this way,
* the analyzer can make sure that the provided values match up to the arguments
* as intended, and the arguments may appear in any order.
* This unary expression is unevaluable because we intend to replace it with
* the provided value itself during query analysis (after possibly rearranging
* the parsed argument list to match up the names to the expected function
* signature).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anchovYu commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "anchovYu (via GitHub)" <gi...@apache.org>.
anchovYu commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1228808365
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1580,6 +1580,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Named arguments support is currently disabled. Enable suppport using ALLOW_NAMED_FUNCTION_ARGUMENTS."
Review Comment:
Should write the full name of the configuration: `spark.sql.allowNamedFunctionArguments`.
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1580,6 +1580,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Cannot call function ‘<functionName>’ because named argument references are not supported here. In this case, the named argument reference was ‘<argument>’.\n"
Review Comment:
```suggestion
"Cannot call function <functionName> because named argument references are not supported here. In this case, the named argument reference was <argument>"
```
We should avoid quoting directly in the error message. Instead, in the `sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala`, try to wrap those idenfitiers using `toSQLId`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -193,6 +193,13 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase {
messageParameters = Map("configKey" -> configKey))
}
+ def namedArgumentsNotEnabledError(argumentName: String): Throwable = {
+ new AnalysisException(
+ errorClass = "NAMED_ARGUMENTS_SUPPORT_DISABLED",
+ messageParameters = Map("message" -> (argumentName + " is a named argument."))
Review Comment:
This error class doesn't have any message parameters `<message>`, the map should be empty.
##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryParsingErrorsSuite.scala:
##########
@@ -20,16 +20,27 @@ package org.apache.spark.sql.errors
import org.apache.spark.SparkThrowable
import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.plans.SQLHelper
import org.apache.spark.sql.test.SharedSparkSession
// Turn of the length check because most of the tests check entire error messages
// scalastyle:off line.size.limit
-class QueryParsingErrorsSuite extends QueryTest with SharedSparkSession {
+class QueryParsingErrorsSuite extends QueryTest with SharedSparkSession with SQLHelper {
private def parseException(sqlText: String): SparkThrowable = {
intercept[ParseException](sql(sqlText).collect())
}
+ test("NAMED_ARGUMENTS_SUPPORT_DISABLED: named arguments not turned on") {
+ withSQLConf("spark.sql.allowNamedFunctionArguments" -> "false") {
+ checkError(
+ exception = parseException("SELECT * FROM encode(value => 'abc', charset => 'utf-8')"),
Review Comment:
Could you add an extra test for TVF?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244385979
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * This represents an argument expression to a function call accompanied with an
+ * explicit reference to the corresponding argument name as a string. In this way,
+ * the analyzer can make sure that the provided values match up to the arguments
+ * as intended, and the arguments may appear in any order.
+ * This unary expression is unevaluable because we intend to replace it with
+ * the provided value itself during query analysis (after possibly rearranging
+ * the parsed argument list to match up the names to the expected function
+ * signature).
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * Example usage with the "encode" scalar function:
+ * SELECT encode("abc", charset => "utf-8");
+ * The second argument generates NamedArgumentExpression("charset", Literal("utf-8"))
+ * SELECT encode(charset => "utf-8", value => "abc");
+ *
+ * @param key The name of the function argument
+ * @param value The value of the function argument
+ */
+case class NamedArgumentExpression(key: String, value: Expression)
+ extends UnaryExpression with Unevaluable {
+ override def nullable: Boolean = value.nullable
Review Comment:
nit: we can remove this line as `UnaryExpression` already provides an implementation of `nullable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1228957857
##########
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -862,6 +862,15 @@ expression
: booleanExpression
;
+namedArgumentExpression
+ : key=identifier FAT_ARROW value=expression
Review Comment:
identifier can be quoted and unquoted
Shall we make it `IDENTIFIER`(unquoted) only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1226973650
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unevaluable unary expression specifically for named argument functions
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * NamedArgumentExpression is expected to be resolved
+ * and replaced in class extending [[NamedArgumentFunction]]
Review Comment:
please remove that
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2176,7 +2185,15 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
val name = ctx.functionName.getText
val isDistinct = Option(ctx.setQuantifier()).exists(_.DISTINCT != null)
// Call `toSeq`, otherwise `ctx.argument.asScala.map(expression)` is `Buffer` in Scala 2.13
- val arguments = ctx.argument.asScala.map(expression).toSeq match {
+ val arguments = ctx.argument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
+ val key = e.namedArgumentExpression.key.strictIdentifier
+ val value = e.namedArgumentExpression.value
+ NamedArgumentExpression(key.getText, expression(value))
Review Comment:
This reveals the NamedArgumentExpression to the analyzer, but we haven't yet implemented resolution support for it. This should be OK since it is `Unevaluable`; let's add a couple analyzer or execution tests with the new syntax to show the errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anchovYu commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "anchovYu (via GitHub)" <gi...@apache.org>.
anchovYu commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1226991300
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unevaluable unary expression specifically for named argument functions
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * NamedArgumentExpression is expected to be resolved
+ * and replaced in class extending [[NamedArgumentFunction]]
+ *
+ * Example usage in encode:
+ * SELECT encode("abc", charset => "utf-8");
Review Comment:
Add the corresponding `NamedArgumentExpression` in the example:
```suggestion
* SELECT encode("abc", charset => "utf-8");
The second argument generates a `NamedArgumentExpression("charset", ......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1602973893
@MaxGekk Can you check if these are the changes you had in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1227111940
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala:
##########
@@ -1412,6 +1412,31 @@ class PlanParserSuite extends AnalysisTest {
assertEqual("select a, b from db.c; ;; ;", table("db", "c").select($"a", $"b"))
}
+ test("table valued function with named arguments") {
+ // All named arguments
Review Comment:
Can you add some negative test cases that return errors? You can look here [1] for some examples, and here are some I can think of:
```
select * from my_tvf(arg1 -> 'value1')
select * from my_tvf(arg1 = => 'value1')
select * from my_tvf(arg1 ==> 'value1')
select * from my_tvf((arg1 => 'value1'))
select * from my_tvf(arg1 'value1')
select * from my_tvf(arg1 => )
select * from my_tvf(arg1 => , 42)
select * from my_tvf(my_tvf.arg1 => 'value1')
select * from my_tvf(arg1 => table t1)
select * from my_tvf(group => 'abc')
```
The last one checks that a reserved keyword like `GROUP` or `ORDER` will work as a named argument name.
[1] https://github.com/google/zetasql/blob/master/zetasql/parser/testdata/named_arguments.test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244387537
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1527,6 +1527,18 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
}
}
+ private def extractExpression(expr: FunctionArgumentContext, funcName: String) : Expression = {
Review Comment:
how about `extractNamedArgument`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1609912887
@MaxGekk Sorry about the confusion. I realized your suggestion for quoting the spark.sql.allowNamedFunctionArguments config was accurate. That was my mistake. I committed your suggestion. Let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244063839
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1659,6 +1659,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Cannot call function <functionName> because named argument references are not enabled here. In this case, the named argument reference was <argument>. Set spark.sql.allowNamedFunctionArguments to true to turn on feature."
Review Comment:
ok, let's pass the config name from the code and quote it by `toSQLConf`. This works to me too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1236010450
##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryParsingErrorsSuite.scala:
##########
@@ -20,16 +20,33 @@ package org.apache.spark.sql.errors
import org.apache.spark.SparkThrowable
import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.plans.SQLHelper
+import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
import org.apache.spark.sql.test.SharedSparkSession
// Turn of the length check because most of the tests check entire error messages
// scalastyle:off line.size.limit
-class QueryParsingErrorsSuite extends QueryTest with SharedSparkSession {
+class QueryParsingErrorsSuite extends QueryTest with SharedSparkSession with SQLHelper {
private def parseException(sqlText: String): SparkThrowable = {
intercept[ParseException](sql(sqlText).collect())
}
+ test("NAMED_ARGUMENTS_SUPPORT_DISABLED: named arguments not turned on") {
+ withSQLConf("spark.sql.allowNamedFunctionArguments" -> "false") {
+ checkError(
+ exception = parseException("SELECT * FROM encode(value => 'abc', charset => 'utf-8')"),
+ errorClass = "NAMED_ARGUMENTS_SUPPORT_DISABLED",
+ parameters = Map("functionName" -> toSQLId("encode"), "argument" -> toSQLId("value"))
Review Comment:
Yeah, tried it without them and it would fail. The parameters are expected to have the extra quotation marks around them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1588158845
@dtenedor Thanks for the comments! I've added the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1227338001
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1544,8 +1544,22 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
if (name.length > 1) {
throw QueryParsingErrors.invalidTableValuedFunctionNameError(name, ctx)
}
+ val args = func.functionArgument.asScala.map { e =>
+ Option(e.namedArgumentExpression).map { n =>
+ if (SQLConf.get.getConf(SQLConf.ALLOW_NAMED_FUNCTION_ARGUMENTS)) {
+ NamedArgumentExpression(n.key.getText, expression(n.value))
+ } else {
+ throw new ParseException(
+ errorClass = "Named arguments not enabled.",
Review Comment:
This is not quite how the error classes work :) instead you need to:
1. Add a new entry in error-classes.json [1]
2. Add a corresponding function in QueryCompilationErrors.scala [2]
3. Call that function here like `throw QueryCompilationErrors.namedArgumentsNotEnabled(n.key.getText)`
4. Exercise this behavior in a unit test to make sure we really get this error message if the configuration is not enabled.
[1] https://github.com/apache/spark/blob/master/core/src/main/resources/error/error-classes.json#L1326
[2] https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1230023875
##########
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -862,6 +862,15 @@ expression
: booleanExpression
;
+namedArgumentExpression
+ : key=identifier FAT_ARROW value=expression
Review Comment:
@gengliangwang After looking at the design doc, would it be fine to keep it like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on PR #41429:
URL: https://github.com/apache/spark/pull/41429#issuecomment-1613811183
I messed up my Git structure with the fork, so migrating to another PR #41796. @MaxGekk just to let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anchovYu commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "anchovYu (via GitHub)" <gi...@apache.org>.
anchovYu commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1227011434
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2176,7 +2185,15 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
val name = ctx.functionName.getText
val isDistinct = Option(ctx.setQuantifier()).exists(_.DISTINCT != null)
// Call `toSeq`, otherwise `ctx.argument.asScala.map(expression)` is `Buffer` in Scala 2.13
- val arguments = ctx.argument.asScala.map(expression).toSeq match {
+ val arguments = ctx.argument.asScala.map { e =>
+ if (e.namedArgumentExpression != null) {
+ val key = e.namedArgumentExpression.key.strictIdentifier
+ val value = e.namedArgumentExpression.value
+ NamedArgumentExpression(key.getText, expression(value))
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1227109912
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -329,6 +329,18 @@ class ExpressionParserSuite extends AnalysisTest {
parameters = Map("error" -> "'x'", "hint" -> ": extra input 'x'"))
}
+ test("function expressions with named arguments") {
+ assertEqual("encode(value => 'abc', charset => 'utf-8')",
Review Comment:
Please also add positive tests with:
* constant but non-literal values, e.g. `'utf' || '-8'` or `abs(42)`
* double parentheses in a named argument, e.g. `arg1 => ((select 1))`
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/NamedArgumentExpression.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types.DataType
+
+/**
+ * An unevaluable unary expression specifically for named argument functions
+ *
+ * SQL Syntax: key => value
+ * SQL grammar: key=identifier FAT_ARROW value=expression
+ *
+ * Example usage in encode:
+ * SELECT encode("abc", charset => "utf-8");
+ * The second argument generates NamedArgumentExpression("charset", Literal("utf-8"))
+ * SELECT encode(charset => "utf-8", value => "abc");
+ *
+ * @param key The name of the function argument
+ * @param value The value of the function argument
+ */
+case class NamedArgumentExpression(key: String, value: Expression)
+ extends UnaryExpression with Unevaluable {
+ override def nullable: Boolean = value.nullable
+
+ override def dataType: DataType = value.dataType
+
+ override def toString: String = s"""$key => $value"""
Review Comment:
```suggestion
override def toString: String = s"$key => $value"
```
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala:
##########
@@ -1412,6 +1412,31 @@ class PlanParserSuite extends AnalysisTest {
assertEqual("select a, b from db.c; ;; ;", table("db", "c").select($"a", $"b"))
}
+ test("table valued function with named arguments") {
+ // All named arguments
Review Comment:
Can you add some negative test cases that return errors? You can look here [1] for some examples, and here are some I can think of:
```
select * from my_tvf(arg1 -> 'value1')
select * from my_tvf(arg1 = => 'value1')
select * from my_tvf(arg1 ==> 'value1')
select * from my_tvf((arg1 => 'value1'))
select * from my_tvf(arg1 'value1')
select * from my_tvf(arg1 => )
select * from my_tvf(arg1 => , 42)
select * from my_tvf(my_tvf.arg1 => 'value1')
select * from my_tvf(arg1 => table t1)
select * from my_tvf(group => 'abc')
```
The last one checks that a reserved keyword like `GROUP` or `ORDER` will work as a named argument name.
[1] https://github.com/google/zetasql/blob/master/zetasql/parser/testdata/named_arguments.test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1228958219
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -2176,7 +2188,18 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
val name = ctx.functionName.getText
val isDistinct = Option(ctx.setQuantifier()).exists(_.DISTINCT != null)
// Call `toSeq`, otherwise `ctx.argument.asScala.map(expression)` is `Buffer` in Scala 2.13
- val arguments = ctx.argument.asScala.map(expression).toSeq match {
+ val arguments = ctx.argument.asScala.map { e =>
+ Option(e.namedArgumentExpression).map { n =>
Review Comment:
Shall we move this to a function for better readability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244054887
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1659,6 +1659,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Cannot call function <functionName> because named argument references are not enabled here. In this case, the named argument reference was <argument>. Set spark.sql.allowNamedFunctionArguments to true to turn on feature."
Review Comment:
@MaxGekk I think @anchovYu said that it is preferred not to add the quotes. She was one who implemented the error-classes framework, and she said it was standard practice to call toSQLId() instead of adding the quotes in the error string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244389813
##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -875,6 +875,18 @@ class QueryExecutionErrorsSuite
sqlState = "XX000")
}
+ test("INTERNAL_ERROR: Calling eval on Unevaluable NamedArgumentExpression") {
Review Comment:
I don't think we need to test it. `Unevaluable` is not added by this PR and has been there for a long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] learningchess2003 commented on a diff in pull request #41429: [SPARK-43922] Add named parameter support in parser for function calls
Posted by "learningchess2003 (via GitHub)" <gi...@apache.org>.
learningchess2003 commented on code in PR #41429:
URL: https://github.com/apache/spark/pull/41429#discussion_r1244077471
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -1659,6 +1659,11 @@
"Not allowed to implement multiple UDF interfaces, UDF class <className>."
]
},
+ "NAMED_ARGUMENTS_SUPPORT_DISABLED" : {
+ "message" : [
+ "Cannot call function <functionName> because named argument references are not enabled here. In this case, the named argument reference was <argument>. Set spark.sql.allowNamedFunctionArguments to true to turn on feature."
Review Comment:
Oh, sorry, I just realized that you didn't mean it for functionName. This is good. I will commit the suggestion you had.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org