You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/14 12:00:30 UTC
[GitHub] [spark] iRakson opened a new pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
iRakson opened a new pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
`hash()` and `xxhash64()` cannot be used on elements of `Maptype`. A new configuration `spark.sql.legacy.useHashOnMapType` is introduced to allow users to restore the previous behaviour.
When `spark.sql.legacy.useHashOnMapType` is set to false:
```
scala> spark.sql("select hash(map())");
org.apache.spark.sql.AnalysisException: cannot resolve 'hash(map())' due to data type mismatch: input to function hash cannot contain elements of MapType; line 1 pos 7;
'Project [unresolvedalias(hash(map(), 42), None)]
+- OneRowRelation
```
when `spark.sql.legacy.useHashOnMapType` is set to true :
```
scala> spark.sql("set spark.sql.legacy.useHashOnMapType=true");
res3: org.apache.spark.sql.DataFrame = [key: string, value: string]
scala> spark.sql("select hash(map())").first()
res4: org.apache.spark.sql.Row = [42]
```
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
As discussed in Jira, SparkSql's map hashcodes depends on their order of insertion which is not consistent with the normal scala behaviour which might confuse users.
Code snippet from JIRA :
```
val a = spark.createDataset(Map(1->1, 2->2) :: Nil)
val b = spark.createDataset(Map(2->2, 1->1) :: Nil)
// Demonstration of how Scala Map equality is unaffected by insertion order:
assert(Map(1->1, 2->2).hashCode() == Map(2->2, 1->1).hashCode())
assert(Map(1->1, 2->2) == Map(2->2, 1->1))
assert(a.first() == b.first())
// In contrast, this will print two different hashcodes:
println(Seq(a, b).map(_.selectExpr("hash(*)").first()))
```
Also `MapType` is prohibited for aggregation / joins / equality comparisons #7819 and set operations #17236.
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce any user-facing change?
Yes. Now users cannot use hash functions on elements of `mapType`. To restore the previous behaviour set `spark.sql.legacy.useHashOnMapType` to true.
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
### How was this patch tested?
UT added.
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591298975
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586865216
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118532/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591298357
**[Test build #118960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118960/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586305692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23189/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588047727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23414/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591246456
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591298975
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589570072
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23529/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586260125
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590951662
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118920/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591419638
**[Test build #118960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118960/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586566197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118466/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r384915402
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,21 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.exists(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
Review comment:
indentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588047393
**[Test build #118662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118662/testReport)** for PR 27580 at commit [`ab27a0f`](https://github.com/apache/spark/commit/ab27a0fc9b0f0599a7e1b92aa584b5a6e111a88e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586566197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118466/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586305684
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591298988
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23708/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r384918063
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,21 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.exists(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
Review comment:
@dongjoon-hyun should i raise a followup for these changes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590846449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23668/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586566625
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591292690
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r381088173
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -2121,6 +2121,26 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-27619: Throw analysis exception when hash and xxhash64 is used on MapType") {
+ Seq("hash", "xxhash64").foreach {
+ case hashExpression =>
+ intercept[AnalysisException] {
+ spark.createDataset(Map(1 -> 10, 2 -> 20) :: Nil).selectExpr(s"$hashExpression(*)")
+ }
+ }
+ }
+
+ test("SPARK-27619: when spark.sql.legacy.useHashOnMapType is true, hash can be used on Maptype") {
+ Seq("hash", "xxhash64").foreach {
+ case hashExpression =>
Review comment:
I modified the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586563658
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23223/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586834124
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379995971
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType. Logically same maps " +
+ "may have different hashcode, thus hash expressions are prohibited on MapType " +
+ "elements. To restore previous behavior set spark.sql.legacy.useHashOnMapType to true.")
Review comment:
nit: `spark.sql.legacy.useHashOnMapType` -> `SQLConf.LEGACY_USE_HASH_ON_MAPTYPE.key`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586868947
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23309/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589515559
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591420590
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118960/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379998771
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType. Logically same maps " +
+ "may have different hashcode, thus hash expressions are prohibited on MapType " +
+ "elements. To restore previous behavior set spark.sql.legacy.useHashOnMapType to true.")
Review comment:
I used `spark.sql.legacy.useHashOnMapType` to maintain consistency with what i updated in migration guide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588103475
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586833776
**[Test build #118536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118536/testReport)** for PR 27580 at commit [`3bcc470`](https://github.com/apache/spark/commit/3bcc47001a68f53e29d43c3a305751dd19ad92ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586260510
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591292386
**[Test build #118946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118946/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586566173
**[Test build #118466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118466/testReport)** for PR 27580 at commit [`944fac1`](https://github.com/apache/spark/commit/944fac1e07cbde83d4dae7b41ec9023cd25f069c).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379961159
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2188,6 +2188,12 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_USE_HASH_ON_MAPTYPE = buildConf("spark.sql.legacy.useHashOnMapType")
+ .doc("When set to true, hash expressions can be applied on elements of MapType. Otherwise, " +
+ "an analysis exception will be thrown.")
+ .booleanConf.
Review comment:
nit: the doc at the end. It has to be in the next line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586566193
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588087085
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118662/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591246456
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591246166
**[Test build #118946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118946/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591292706
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118946/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586566193
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r384920729
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2205,6 +2205,12 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_USE_HASH_ON_MAPTYPE = buildConf("spark.sql.legacy.useHashOnMapType")
Review comment:
ah "allow" is more precise. @iRakson feel free to send a followup to address all the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r384915586
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2205,6 +2205,12 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_USE_HASH_ON_MAPTYPE = buildConf("spark.sql.legacy.useHashOnMapType")
Review comment:
Maybe, `allowHashOnMapType`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590846045
**[Test build #118920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118920/testReport)** for PR 27580 at commit [`5c5ae20`](https://github.com/apache/spark/commit/5c5ae20c30f53e2df018177c8f376e0070ba2d97).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586833776
**[Test build #118536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118536/testReport)** for PR 27580 at commit [`3bcc470`](https://github.com/apache/spark/commit/3bcc47001a68f53e29d43c3a305751dd19ad92ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586821334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23287/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588087077
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379960516
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -2121,6 +2121,26 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-27619: Throw analysis exception when hash and xxhash64 is used on MapType") {
+ Seq("hash", "xxhash64").foreach {
+ case hashExpression =>
+ intercept[AnalysisException] {
+ spark.createDataset(Map(1 -> 10, 2 -> 20) :: Nil).selectExpr(s"$hashExpression(*)")
+ }
+ }
+ }
+
+ test("SPARK-27619: when spark.sql.legacy.useHashOnMapType is true, hash can be used on Maptype") {
+ Seq("hash", "xxhash64").foreach {
+ case hashExpression =>
Review comment:
`hashExpression` seems not used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586821331
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586563647
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586868937
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589569688
**[Test build #118778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118778/testReport)** for PR 27580 at commit [`ab27a0f`](https://github.com/apache/spark/commit/ab27a0fc9b0f0599a7e1b92aa584b5a6e111a88e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586864322
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591420575
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590846436
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590951662
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118920/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586989896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118554/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591298357
**[Test build #118960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118960/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589675680
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586407641
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586821334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23287/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589570066
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r383835740
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
Review comment:
`forall` -> `exists`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586563647
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591298988
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23708/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586834127
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23291/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586305692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23189/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586304987
**[Test build #118432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118432/testReport)** for PR 27580 at commit [`7a66814`](https://github.com/apache/spark/commit/7a668140ac0951e646944baa6a9458ab71559462).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586867925
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586834124
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-587432960
gentle ping @cloud-fan @HyukjinKwon @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590846449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23668/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591555095
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379961053
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ protected def hasMapType(dt: DataType): Boolean = {
Review comment:
Seems like it doesn't need to be `protected` at this moment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379404394
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,19 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ protected def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType. To restore previous " +
+ s"behavior set spark.sql.legacy.useHashOnMapType to true.")
Review comment:
we should also warn users about the consequence: logically same maps may have different hashcode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586260510
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590846045
**[Test build #118920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118920/testReport)** for PR 27580 at commit [`5c5ae20`](https://github.com/apache/spark/commit/5c5ae20c30f53e2df018177c8f376e0070ba2d97).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r384915364
##########
File path: docs/sql-migration-guide.md
##########
@@ -224,6 +224,8 @@ license: |
- Since Spark 3.0, when casting string value to integral types(tinyint, smallint, int and bigint), datetime types(date, timestamp and interval) and boolean type, the leading and trailing whitespaces (<= ASCII 32) will be trimmed before converted to these type values, e.g. `cast(' 1\t' as int)` results `1`, `cast(' 1\t' as boolean)` results `true`, `cast('2019-10-10\t as date)` results the date value `2019-10-10`. In Spark version 2.4 and earlier, while casting string to integrals and booleans, it will not trim the whitespaces from both ends, the foregoing results will be `null`, while to datetimes, only the trailing spaces (= ASCII 32) will be removed.
+ - Since Spark 3.0, An analysis exception will be thrown when hash expressions are applied on elements of MapType. To restore the behavior before Spark 3.0, set `spark.sql.legacy.useHashOnMapType` to true.
Review comment:
nit.
```
`An` -> `an`
to true -> to `true`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590951648
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379996197
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -2121,6 +2121,25 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-27619: Throw analysis exception when hash and xxhash64 is used on MapType") {
+ Seq("hash", "xxhash64").foreach {
+ case hashExpression =>
+ intercept[AnalysisException] {
+ spark.createDataset(Map(1 -> 10, 2 -> 20) :: Nil).selectExpr(s"$hashExpression(*)")
+ }
+ }
+ }
+
+ test("SPARK-27619: when spark.sql.legacy.useHashOnMapType is true, hash can be used on Maptype") {
+ Seq("hash", "xxhash64").foreach {
+ case hashExpression =>
+ withSQLConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE.key -> true.toString()) {
Review comment:
nit: `true.toString()` -> `"true"`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589675695
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118778/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586260125
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586563658
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23223/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r383836114
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType. Logically same maps " +
Review comment:
`Logically` -> `In Spark, `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586298292
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586305684
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589674813
**[Test build #118778 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118778/testReport)** for PR 27580 at commit [`ab27a0f`](https://github.com/apache/spark/commit/ab27a0fc9b0f0599a7e1b92aa584b5a6e111a88e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586834127
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23291/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586821331
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590846436
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591246463
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23695/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379960602
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ protected def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType. Logically same maps " +
+ s"may have different hashcode, thus hash expressions are prohibited on MapType " +
Review comment:
nit: leading `s` seems unnecessary in this line and the next line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586563396
**[Test build #118466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118466/testReport)** for PR 27580 at commit [`944fac1`](https://github.com/apache/spark/commit/944fac1e07cbde83d4dae7b41ec9023cd25f069c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591292706
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118946/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586868473
**[Test build #118554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118554/testReport)** for PR 27580 at commit [`3bcc470`](https://github.com/apache/spark/commit/3bcc47001a68f53e29d43c3a305751dd19ad92ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588047720
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586821074
**[Test build #118532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118532/testReport)** for PR 27580 at commit [`477e154`](https://github.com/apache/spark/commit/477e154c3abc2d64874203c70a22017224c893b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591292690
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589568209
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586988947
**[Test build #118554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118554/testReport)** for PR 27580 at commit [`3bcc470`](https://github.com/apache/spark/commit/3bcc47001a68f53e29d43c3a305751dd19ad92ad).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591246463
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23695/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586563396
**[Test build #118466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118466/testReport)** for PR 27580 at commit [`944fac1`](https://github.com/apache/spark/commit/944fac1e07cbde83d4dae7b41ec9023cd25f069c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591246166
**[Test build #118946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118946/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586989881
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586868947
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23309/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591787034
+1, LGTM. You can ignore the above comment because those are too minor, @iRakson .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586864336
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118536/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590951648
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589569688
**[Test build #118778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118778/testReport)** for PR 27580 at commit [`ab27a0f`](https://github.com/apache/spark/commit/ab27a0fc9b0f0599a7e1b92aa584b5a6e111a88e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586864322
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586865216
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118532/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586259638
cc @dongjoon-hyun @cloud-fan @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586260822
OK to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27580: [SPARK-27619][SQL]MapType
should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591297728
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586407641
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588087085
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118662/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591420590
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118960/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588047720
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27580: [SPARK-27619][SQL]MapType
should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591136602
@iRakson It seems a valid test failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586865205
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586864092
**[Test build #118536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118536/testReport)** for PR 27580 at commit [`3bcc470`](https://github.com/apache/spark/commit/3bcc47001a68f53e29d43c3a305751dd19ad92ad).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586864336
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118536/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586865205
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590214816
gentle ping @maropu @HyukjinKwon @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589675680
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586407339
**[Test build #118432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118432/testReport)** for PR 27580 at commit [`7a66814`](https://github.com/apache/spark/commit/7a668140ac0951e646944baa6a9458ab71559462).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586261267
sounds reasonable, cc @hvanhovell
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591787034
+1, late LGTM. You can ignore the above comment because those are too minor, @iRakson .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r383836608
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,20 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ private def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType. Logically same maps " +
+ "may have different hashcode, thus hash expressions are prohibited on MapType " +
+ "elements. To restore previous behavior set spark.sql.legacy.useHashOnMapType to true.")
Review comment:
We should not hardcode the config name. Better to use `${SQLConf.LEGACY_USE_HASH_ON_MAPTYPE.key}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586821074
**[Test build #118532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118532/testReport)** for PR 27580 at commit [`477e154`](https://github.com/apache/spark/commit/477e154c3abc2d64874203c70a22017224c893b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586407650
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118432/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586407650
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118432/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-591420575
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588047727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23414/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586868473
**[Test build #118554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118554/testReport)** for PR 27580 at commit [`3bcc470`](https://github.com/apache/spark/commit/3bcc47001a68f53e29d43c3a305751dd19ad92ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586304987
**[Test build #118432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118432/testReport)** for PR 27580 at commit [`7a66814`](https://github.com/apache/spark/commit/7a668140ac0951e646944baa6a9458ab71559462).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-590951240
**[Test build #118920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118920/testReport)** for PR 27580 at commit [`5c5ae20`](https://github.com/apache/spark/commit/5c5ae20c30f53e2df018177c8f376e0070ba2d97).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586989896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118554/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586820216
All review comments are handled. Please check once. @HyukjinKwon @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586989881
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588086870
**[Test build #118662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118662/testReport)** for PR 27580 at commit [`ab27a0f`](https://github.com/apache/spark/commit/ab27a0fc9b0f0599a7e1b92aa584b5a6e111a88e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588087077
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#discussion_r379397803
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##########
@@ -249,10 +247,18 @@ abstract class HashExpression[E] extends Expression {
override def nullable: Boolean = false
+ protected def hasMapType(dt: DataType): Boolean = {
+ dt.existsRecursively(_.isInstanceOf[MapType])
+ }
+
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
+ } else if (children.forall(child => hasMapType(child.dataType)) &&
+ !SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
+ TypeCheckResult.TypeCheckFailure(
+ s"input to function $prettyName cannot contain elements of MapType")
Review comment:
the error message should mention the legacy config and how to restore the old behavior
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589570066
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586868937
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-588047393
**[Test build #118662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118662/testReport)** for PR 27580 at commit [`ab27a0f`](https://github.com/apache/spark/commit/ab27a0fc9b0f0599a7e1b92aa584b5a6e111a88e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589675695
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118778/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-589570072
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23529/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580:
[SPARK-27619][SQL]MapType should be prohibited in hash expressions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
URL: https://github.com/apache/spark/pull/27580#issuecomment-586864084
**[Test build #118532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118532/testReport)** for PR 27580 at commit [`477e154`](https://github.com/apache/spark/commit/477e154c3abc2d64874203c70a22017224c893b7).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org