You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/24 07:48:09 UTC
[GitHub] [spark] beliefer opened a new pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
beliefer opened a new pull request #30479:
URL: https://github.com/apache/spark/pull/30479
### What changes were proposed in this pull request?
In Spark, decode(bin, charset) - Decodes the first argument using the second argument character set.
Unfortunately this is NOT what any other SQL vendor understands `DECODE` to do.
`DECODE` generally is a short hand for a simple case expression:
```
SELECT DECODE(c1, 1, 'Hello', 2, 'World', '!') FROM (VALUES (1), (2), (3)) AS T(c1)
=>
(Hello),
(World)
(!)
```
There are some mainstream database support the syntax.
**Oracle**
https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/DECODE.html#GUID-39341D91-3442-4730-BD34-D3CF5D4701CE
**Vertica**
https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/String/DECODE.htm?tocpath=SQL%20Reference%20Manual%7CSQL%20Functions%7CString%20Functions%7C_____10
**DB2**
https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.sqls.doc/ids_sqs_1447.htm
**Redshift**
https://docs.aws.amazon.com/redshift/latest/dg/r_DECODE_expression.html
### Why are the changes needed?
It is very useful.
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
Jenkins test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732991020
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733666278
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740594440
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132427/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732914434
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741798395
**[Test build #132484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132484/testReport)** for PR 30479 at commit [`eea967a`](https://github.com/apache/spark/commit/eea967a4aa4ad18760c5a65c275c1b3f68971f85).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732914434
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732790339
**[Test build #131628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131628/testReport)** for PR 30479 at commit [`cd81faf`](https://github.com/apache/spark/commit/cd81faf6ea86a34bb79514566bf08e6cb22cd7a7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Decode(fakeChild: Expression, params: Seq[Expression]) extends RuntimeReplaceable `
* `case class StringDecode(bin: Expression, charset: Expression)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741633094
**[Test build #132460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132460/testReport)** for PR 30479 at commit [`1614e62`](https://github.com/apache/spark/commit/1614e6206b75b35ca8b80f266e94bffa6a48c222).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741567638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37062/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-742994211
@cloud-fan Thanks for your work
@maropu @HyukjinKwon Thanks for your review too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740513400
**[Test build #132427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132427/testReport)** for PR 30479 at commit [`6a1b919`](https://github.com/apache/spark/commit/6a1b919feb8e589ee54d8d16d5508836af3e0589).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539072466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
Review comment:
If rename the `existing` decode func will break the behavior.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732799386
**[Test build #131645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131645/testReport)** for PR 30479 at commit [`338d28b`](https://github.com/apache/spark/commit/338d28b03cc9d5882ec807c432449b81a579b7c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741567638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37062/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539061140
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
Review comment:
hm, IMHO it would be good to rename the existing `decode` func or assign a new name into the new `decode` func because the generated doc for `decode` will be complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540705955
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
Review comment:
We can probably rename the function in ansi mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732856036
**[Test build #131645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131645/testReport)** for PR 30479 at commit [`338d28b`](https://github.com/apache/spark/commit/338d28b03cc9d5882ec807c432449b81a579b7c8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741695048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37086/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740452443
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539078950
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
Review comment:
Yea, so I think we need add a legacy config, update the migration guide, ... if we rename it. Anyway, the latter approach (assigning a new name into the new `decode` func) looks better if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732725257
**[Test build #131628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131628/testReport)** for PR 30479 at commit [`cd81faf`](https://github.com/apache/spark/commit/cd81faf6ea86a34bb79514566bf08e6cb22cd7a7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-742987078
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741522857
**[Test build #132460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132460/testReport)** for PR 30479 at commit [`1614e62`](https://github.com/apache/spark/commit/1614e6206b75b35ca8b80f266e94bffa6a48c222).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740603657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132426/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740567233
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37027/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741695025
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37086/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732790562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740603657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132426/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539132890
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(encode('abc', 'utf-8'), 'utf-8');
+ abc
+ > SELECT _FUNC_(2, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ San Francisco
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ Non domestic
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
+ NULL
+ """,
+ since = "3.1.0")
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740540392
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37027/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741522857
**[Test build #132460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132460/testReport)** for PR 30479 at commit [`1614e62`](https://github.com/apache/spark/commit/1614e6206b75b35ca8b80f266e94bffa6a48c222).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732790562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733149740
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540712652
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2097,7 +2157,7 @@ case class UnBase64(child: Expression)
""",
since = "1.5.0")
// scalastyle:on line.size.limit
-case class Decode(bin: Expression, charset: Expression)
+case class StringDecode(bin: Expression, charset: Expression)
Review comment:
I'd suggest the following followup:
1. add a new function name for `StringDecode`. Probably "string_decode".
2. require `decode` function to take more than 2 parameters, under ansi mode(or a legacy config)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r538960499
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2082,64 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 => StringDecode(Literal.create(null, StringType), Literal.create(null, StringType))
+ case 1 => StringDecode(params.head, Literal.create(null, StringType))
Review comment:
Let's preserve the `AnalysisException(s"Invalid number of arguments for function decode. ")`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741621624
> Just a question; the other systems don't have a function like Spark decode/encode?
Only PostgreSQL have the function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540712600
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(encode('abc', 'utf-8'), 'utf-8');
+ abc
+ > SELECT _FUNC_(2, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ San Francisco
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ Non domestic
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
+ NULL
+ """,
+ since = "3.2.0")
+// scalastyle:on line.size.limit
+case class Decode(params: Seq[Expression], child: Expression) extends RuntimeReplaceable {
Review comment:
The new decode could keep compatible with old one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540708892
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(encode('abc', 'utf-8'), 'utf-8');
+ abc
+ > SELECT _FUNC_(2, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ San Francisco
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ Non domestic
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
+ NULL
+ """,
+ since = "3.2.0")
+// scalastyle:on line.size.limit
+case class Decode(params: Seq[Expression], child: Expression) extends RuntimeReplaceable {
Review comment:
So this replaces the existing `decode` in SQL side only, right? shall we update migration guide?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732725257
**[Test build #131628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131628/testReport)** for PR 30479 at commit [`cd81faf`](https://github.com/apache/spark/commit/cd81faf6ea86a34bb79514566bf08e6cb22cd7a7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r538420019
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2082,64 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 => StringDecode(Literal.create(null, StringType), Literal.create(null, StringType))
+ case 1 => StringDecode(params.head, Literal.create(null, StringType))
Review comment:
do we really allow 0 or 1 parameter before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741579331
Just a question; the other systems don't have a function like Spark decode/encode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741657553
**[Test build #132484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132484/testReport)** for PR 30479 at commit [`eea967a`](https://github.com/apache/spark/commit/eea967a4aa4ad18760c5a65c275c1b3f68971f85).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733512705
**[Test build #131747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131747/testReport)** for PR 30479 at commit [`1cf326b`](https://github.com/apache/spark/commit/1cf326bb88c8b743ef629fe6bf062de299ebae9b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732799386
**[Test build #131645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131645/testReport)** for PR 30479 at commit [`338d28b`](https://github.com/apache/spark/commit/338d28b03cc9d5882ec807c432449b81a579b7c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540712600
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(encode('abc', 'utf-8'), 'utf-8');
+ abc
+ > SELECT _FUNC_(2, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ San Francisco
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ Non domestic
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
+ NULL
+ """,
+ since = "3.2.0")
+// scalastyle:on line.size.limit
+case class Decode(params: Seq[Expression], child: Expression) extends RuntimeReplaceable {
Review comment:
The new `decode` could keep compatible with old one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740513400
**[Test build #132427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132427/testReport)** for PR 30479 at commit [`6a1b919`](https://github.com/apache/spark/commit/6a1b919feb8e589ee54d8d16d5508836af3e0589).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539074944
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
Review comment:
The new `decode` function is the same as other function uses `RuntimeReplaceable`. The parameter list is not fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30479:
URL: https://github.com/apache/spark/pull/30479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540709293
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2097,7 +2157,7 @@ case class UnBase64(child: Expression)
""",
since = "1.5.0")
// scalastyle:on line.size.limit
-case class Decode(bin: Expression, charset: Expression)
+case class StringDecode(bin: Expression, charset: Expression)
Review comment:
Do we have an alternative for the previous `decode` after this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741552517
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37062/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733658841
**[Test build #131747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131747/testReport)** for PR 30479 at commit [`1cf326b`](https://github.com/apache/spark/commit/1cf326bb88c8b743ef629fe6bf062de299ebae9b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Decode(params: Seq[Expression], fakeChild: Expression) extends RuntimeReplaceable `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539110966
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
Review comment:
This PR could avoid migration and support the new `decode` as well. cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740558334
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37027/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733149740
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r540710626
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2097,7 +2157,7 @@ case class UnBase64(child: Expression)
""",
since = "1.5.0")
// scalastyle:on line.size.limit
-case class Decode(bin: Expression, charset: Expression)
+case class StringDecode(bin: Expression, charset: Expression)
Review comment:
This only renames the expression, not the function. This PR just adds an overload of the `decode` function when there are more than 3 parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733512705
**[Test build #131747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131747/testReport)** for PR 30479 at commit [`1cf326b`](https://github.com/apache/spark/commit/1cf326bb88c8b743ef629fe6bf062de299ebae9b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539063086
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
Review comment:
Could you follow the other format? e.g., https://github.com/apache/spark/blob/c88eddac3bf860d04bba91fc913f8b2069a94153/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L479-L486
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740589612
**[Test build #132427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132427/testReport)** for PR 30479 at commit [`6a1b919`](https://github.com/apache/spark/commit/6a1b919feb8e589ee54d8d16d5508836af3e0589).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741634078
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132460/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740545835
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741695048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37086/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741634078
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132460/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741680088
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37086/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740567233
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37027/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740505451
**[Test build #132426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132426/testReport)** for PR 30479 at commit [`80c56a1`](https://github.com/apache/spark/commit/80c56a1d5278ab8b622e9836ccef9db56cb8a9d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-733666278
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740530909
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741539001
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37062/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740594440
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132427/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740505451
**[Test build #132426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132426/testReport)** for PR 30479 at commit [`80c56a1`](https://github.com/apache/spark/commit/80c56a1d5278ab8b622e9836ccef9db56cb8a9d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740545860
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30479:
URL: https://github.com/apache/spark/pull/30479#discussion_r539120389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2082,6 +2083,65 @@ case class UnBase64(child: Expression)
}
}
+object Decode {
+ def createExpr(params: Seq[Expression]): Expression = {
+ params.length match {
+ case 0 | 1 =>
+ throw new AnalysisException("Invalid number of arguments for function decode. " +
+ s"Expected: 2; Found: ${params.length}")
+ case 2 => StringDecode(params.head, params.last)
+ case _ =>
+ val input = params.head
+ val other = params.tail
+ val itr = other.iterator
+ var default: Expression = Literal.create(null, StringType)
+ val branches = ArrayBuffer.empty[(Expression, Expression)]
+ while (itr.hasNext) {
+ val search = itr.next
+ if (itr.hasNext) {
+ val condition = EqualTo(input, search)
+ branches += ((condition, itr.next))
+ } else {
+ default = search
+ }
+ }
+ CaseWhen(branches.seq, default)
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ |_FUNC_(bin, charset) - Decodes the first argument using the second argument character set.
+ |
+ |_FUNC_(expr, search, result [, search, result ] ... [, default]) - Decode compares expr
+ | to each search value one by one. If expr is equal to a search, returns the corresponding result.
+ | If no match is found, then Oracle returns default. If default is omitted, returns null.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(encode('abc', 'utf-8'), 'utf-8');
+ abc
+ > SELECT _FUNC_(2, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ San Francisco
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
+ Non domestic
+ > SELECT _FUNC_(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
+ NULL
+ """,
+ since = "3.1.0")
Review comment:
BTW, new feature should arrive in 3.2.0 because branch-3.1 is out now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-732991020
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741657553
**[Test build #132484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132484/testReport)** for PR 30479 at commit [`eea967a`](https://github.com/apache/spark/commit/eea967a4aa4ad18760c5a65c275c1b3f68971f85).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741815129
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132484/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740603086
**[Test build #132426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132426/testReport)** for PR 30479 at commit [`80c56a1`](https://github.com/apache/spark/commit/80c56a1d5278ab8b622e9836ccef9db56cb8a9d6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Decode(params: Seq[Expression], child: Expression) extends RuntimeReplaceable `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-740545860
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30479: [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30479:
URL: https://github.com/apache/spark/pull/30479#issuecomment-741815129
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132484/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org