You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dtenedor (via GitHub)" <gi...@apache.org> on 2024/11/25 20:07:09 UTC
[PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
dtenedor opened a new pull request, #48961:
URL: https://github.com/apache/spark/pull/48961
### What changes were proposed in this pull request?
This PR extends the Spark SQL grammar to allow an optional trailing comma at the end of SELECT lists, and other named expression lists.
This behavior is controlled by the configuration `spark.sql.optionalTrailingCommaInNamedExpressionLists` (`SQLConf.get.OPTIONAL_TRAILING_COMMA_IN_NAMED_EXPRESSION_LISTS`). The new configuration is true by default as of its introduction in this PR.
For example, this query was not valid syntax before, but is now:
```
SELECT 1, 2, FROM t
```
### Why are the changes needed?
After deliberation, we decided it was useful to have this support to help with migration from other legacy databases.
### Does this PR introduce _any_ user-facing change?
Yes, see above.
### How was this patch tested?
This PR adds unit test cases with the configuration enabled and disabled.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48961:
URL: https://github.com/apache/spark/pull/48961#discussion_r1858653629
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -5268,6 +5268,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val OPTIONAL_TRAILING_COMMA_IN_NAMED_EXPRESSION_LISTS =
+ buildConf("spark.sql.optionalTrailingCommaInNamedExpressionLists")
Review Comment:
The config name confuses slightly. I would think if it is `true`, the trailing comma is optional otherwise is mandatory ;-).
```suggestion
buildConf("spark.sql.allowTrailingCommaInNamedExpressionLists")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #48961:
URL: https://github.com/apache/spark/pull/48961#issuecomment-2521395784
After investigating this more, I find that it is not actually possible to make this change without causing unacceptable ambiguity in the grammar. Closing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor closed pull request #48961: [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists
URL: https://github.com/apache/spark/pull/48961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #48961:
URL: https://github.com/apache/spark/pull/48961#issuecomment-2498931299
cc @srielau @gengliangwang @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #48961:
URL: https://github.com/apache/spark/pull/48961#discussion_r1859245269
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala:
##########
@@ -1974,4 +1974,18 @@ class PlanParserSuite extends AnalysisTest {
assert(unresolvedRelation2.options == CaseInsensitiveStringMap.empty)
assert(unresolvedRelation2.isStreaming)
}
+
+ test("SPARK-50418: Support an optional trailing comma at the end of SELECT lists") {
+ withSQLConf(SQLConf.OPTIONAL_TRAILING_COMMA_IN_NAMED_EXPRESSION_LISTS.key -> "true") {
+ assertEqual("select 1, ", OneRowRelation().select(1))
+ assertEqual("select 1, 2", OneRowRelation().select(1, 2))
Review Comment:
nit: let's also test `select 1, 2,`
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala:
##########
@@ -1974,4 +1974,18 @@ class PlanParserSuite extends AnalysisTest {
assert(unresolvedRelation2.options == CaseInsensitiveStringMap.empty)
assert(unresolvedRelation2.isStreaming)
}
+
+ test("SPARK-50418: Support an optional trailing comma at the end of SELECT lists") {
+ withSQLConf(SQLConf.OPTIONAL_TRAILING_COMMA_IN_NAMED_EXPRESSION_LISTS.key -> "true") {
+ assertEqual("select 1, ", OneRowRelation().select(1))
+ assertEqual("select 1, 2", OneRowRelation().select(1, 2))
+ }
+ withSQLConf(SQLConf.OPTIONAL_TRAILING_COMMA_IN_NAMED_EXPRESSION_LISTS.key -> "false") {
+ assertEqual("select 1, 2", OneRowRelation().select(1, 2))
+ checkError(
+ exception = parseException("select 1,"),
Review Comment:
nit: let's also test `select 1, 2,`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48961:
URL: https://github.com/apache/spark/pull/48961#discussion_r1859872436
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -5268,6 +5268,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val OPTIONAL_TRAILING_COMMA_IN_NAMED_EXPRESSION_LISTS =
+ buildConf("spark.sql.optionalTrailingCommaInNamedExpressionLists")
Review Comment:
named expression list is not a user-facing concept, how about `spark.sql.allowTrailingCommaInSelectList`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50418][SQL] Allow an optional trailing comma at the end of SELECT lists [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48961:
URL: https://github.com/apache/spark/pull/48961#discussion_r1859879714
##########
sql/core/src/test/resources/sql-tests/results/postgreSQL/window_part3.sql.out:
##########
@@ -425,7 +425,7 @@ org.apache.spark.sql.catalyst.parser.ParseException
"sqlState" : "42601",
"messageParameters" : {
"error" : "'BY'",
- "hint" : ": extra input 'BY'"
+ "hint" : ""
Review Comment:
why do we lose the hint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org