You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "MaxGekk (via GitHub)" <gi...@apache.org> on 2023/09/14 18:08:26 UTC
[GitHub] [spark] MaxGekk opened a new pull request, #42931: [WIP][SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
MaxGekk opened a new pull request, #42931:
URL: https://github.com/apache/spark/pull/42931
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
To fix the issue demonstrated by the example:
```scala
spark.sql("select element_at(?, 1)", Array(array(lit(1)))).collect()
java.lang.UnsupportedOperationException: literal unresolved_function {
[info] function_name: "array"
[info] arguments {
[info] literal {
[info] integer: 1
[info] }
[info] }
[info] }
[info] not supported (yet).
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
By new checks:
```
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.ClientE2ETestSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #42931:
URL: https://github.com/apache/spark/pull/42931#issuecomment-1724026847
Merging to master. Thank you, @HyukjinKwon and @cloud-fan for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #42931:
URL: https://github.com/apache/spark/pull/42931#discussion_r1328271078
##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -65,6 +65,12 @@ message SqlCommand {
// (Optional) A sequence of literal expressions for positional parameters in the SQL query text.
repeated Expression.Literal pos_args = 3;
+
+ // (Optional) A map of parameter names to expressions.
+ map<string, Expression> arguments = 4;
+
+ // (Optional) A sequence of expressions for positional parameters in the SQL query text.
Review Comment:
@grundprinzip @HyukjinKwon Is there any way to deprecate some fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #42931:
URL: https://github.com/apache/spark/pull/42931#discussion_r1328337587
##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -65,6 +65,12 @@ message SqlCommand {
// (Optional) A sequence of literal expressions for positional parameters in the SQL query text.
repeated Expression.Literal pos_args = 3;
+
+ // (Optional) A map of parameter names to expressions.
+ map<string, Expression> arguments = 4;
+
+ // (Optional) A sequence of expressions for positional parameters in the SQL query text.
Review Comment:
Would this be enough `[deprecated=true]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #42931:
URL: https://github.com/apache/spark/pull/42931#discussion_r1328222454
##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -65,6 +65,12 @@ message SqlCommand {
// (Optional) A sequence of literal expressions for positional parameters in the SQL query text.
repeated Expression.Literal pos_args = 3;
+
+ // (Optional) A map of parameter names to expressions.
+ map<string, Expression> arguments = 4;
+
+ // (Optional) A sequence of expressions for positional parameters in the SQL query text.
Review Comment:
shall we also mention that the `args` and `pos_args` are deprecated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #42931:
URL: https://github.com/apache/spark/pull/42931#discussion_r1328222151
##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -65,6 +65,12 @@ message SqlCommand {
// (Optional) A sequence of literal expressions for positional parameters in the SQL query text.
repeated Expression.Literal pos_args = 3;
+
+ // (Optional) A map of parameter names to expressions.
+ map<string, Expression> arguments = 4;
Review Comment:
`named_arguments`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #42931:
URL: https://github.com/apache/spark/pull/42931#discussion_r1328222237
##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -65,6 +65,12 @@ message SqlCommand {
// (Optional) A sequence of literal expressions for positional parameters in the SQL query text.
repeated Expression.Literal pos_args = 3;
+
+ // (Optional) A map of parameter names to expressions.
+ map<string, Expression> arguments = 4;
+
+ // (Optional) A sequence of expressions for positional parameters in the SQL query text.
Review Comment:
shall we doc that this can't coexist with `named_arguments`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #42931: [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
URL: https://github.com/apache/spark/pull/42931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org