You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yangw1234 <gi...@git.apache.org> on 2016/10/10 07:32:20 UTC
[GitHub] spark pull request #15416: [SPARK-17849] Fix NPE problem when using grouping...
GitHub user yangw1234 opened a pull request:
https://github.com/apache/spark/pull/15416
[SPARK-17849] Fix NPE problem when using grouping sets
## What changes were proposed in this pull request?
Prior this pr, the following code would cause an NPE:
`case class point(a:String, b:String, c:String, d: Int)`
`val data = Seq(
point("1","2","3", 1),
point("4","5","6", 1),
point("7","8","9", 1)
)`
`sc.parallelize(data).toDF().registerTempTable("table")`
`spark.sql("select a, b, c, count(d) from table group by a, b, c GROUPING SETS ((a)) ").show()`
The reason is that when the grouping_id() behavior was changed in #10677, some code (which should be changed) was left out.
Take the above code for example, prior #10677, the bit mask for set "(a)" was `001`, while after #10677 the bit mask was changed to `011`. However, the `nonNullBitmask` was not changed accordingly.
This pr will fix this problem.
## How was this patch tested?
add integration tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yangw1234/spark groupingid
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15416.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15416
----
commit a6d4651d63c5d31dcd7b1ca6d48be9707316e33b
Author: wangyang <wa...@haizhi.com>
Date: 2016-10-10T04:19:31Z
fix grouping id bug
commit ab82b211afc9c9d2030c25b4f953db6f4b1bb62e
Author: wangyang <wa...@haizhi.com>
Date: 2016-10-10T07:03:35Z
add test
commit c849b6a02ee65e3a818abc03f301d82b36ec7f43
Author: wangyang <wa...@haizhi.com>
Date: 2016-10-10T07:31:51Z
add test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] Fix NPE problem when using grouping sets
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on the issue:
https://github.com/apache/spark/pull/15416
cc @davies Would you help reviewing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #3309 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3309/consoleFull)** for PR 15416 at commit [`42f7a63`](https://github.com/apache/spark/commit/42f7a63e0f43695e5ce8d85b5bf7425aa02a8b8b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on the issue:
https://github.com/apache/spark/pull/15416
scala style fixed.
I didn't notice. Sorry for the delay. @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #68203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68203/consoleFull)** for PR 15416 at commit [`0ad7aba`](https://github.com/apache/spark/commit/0ad7aba8dfb997f5d8e65a874123a92466c47e1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82720505
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,14 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ // The left most bit in the bitmasks corresponds to the last expression in groupByAliases
+ // with 0 indicating this expression is in the grouping set. The following line of code
+ // calculates the bit mask representing the expressions that exist in all the grouping sets.
+ val nonNullBitmask = ~ x.bitmasks.reduce(_ | _)
--- End diff --
Could you remove the '~' here, and use `(nonNullBitmask & (1 << (attrLength - idx - 1))) == 1`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #66970 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66970/consoleFull)** for PR 15416 at commit [`0ad7aba`](https://github.com/apache/spark/commit/0ad7aba8dfb997f5d8e65a874123a92466c47e1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82562070
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ---
@@ -2189,6 +2189,24 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
}
}
+ test("SPARK-17849: grouping set throws NPE") {
--- End diff --
@rxin done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/15416
LGTM - Merging to master/2.1/2.0. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/15416
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82726593
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,14 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ // The rightmost bit in the bitmasks corresponds to the last expression in groupByAliases with 0
+ // indicating this expression is in the grouping set. The following line of code calculates the
+ // bitmask representing the expressions that exist in all the grouping sets (also indicated by 0).
+ val nonNullBitmask = x.bitmasks.reduce(_ | _)
--- End diff --
done @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #3337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3337/consoleFull)** for PR 15416 at commit [`69f6e4f`](https://github.com/apache/spark/commit/69f6e4f1bc37afd6b3ca529c8b0f0afec891459a).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82723882
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,14 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ // The rightmost bit in the bitmasks corresponds to the last expression in groupByAliases with 0
+ // indicating this expression is in the grouping set. The following line of code calculates the
+ // bitmask representing the expressions that exist in all the grouping sets (also indicated by 0).
+ val nonNullBitmask = x.bitmasks.reduce(_ | _)
--- End diff --
Should we call this `nullBitmask` now? (1 means it's nullable)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #3309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3309/consoleFull)** for PR 15416 at commit [`42f7a63`](https://github.com/apache/spark/commit/42f7a63e0f43695e5ce8d85b5bf7425aa02a8b8b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] Fix NPE problem when using grouping sets
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on the issue:
https://github.com/apache/spark/pull/15416
also cc @hvanhovell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the issue:
https://github.com/apache/spark/pull/15416
@yangw1234 Thanks for working on this, could you also double check that all the places that use bitmasks are correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82715416
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,11 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ val nonNullBitmask = ~ x.bitmasks.reduce(_ | _)
--- End diff --
Ok, I'll do it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/15416
I'll merge after a successfull test run
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/15416
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #66970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66970/consoleFull)** for PR 15416 at commit [`0ad7aba`](https://github.com/apache/spark/commit/0ad7aba8dfb997f5d8e65a874123a92466c47e1a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15416
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66970/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the issue:
https://github.com/apache/spark/pull/15416
LGTM, pending tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #68203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68203/consoleFull)** for PR 15416 at commit [`0ad7aba`](https://github.com/apache/spark/commit/0ad7aba8dfb997f5d8e65a874123a92466c47e1a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68203/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15416
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on the issue:
https://github.com/apache/spark/pull/15416
@rxin @davies Will this patch be merged in 2.0.2? Kind of need this to upgrade our production environment. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on the issue:
https://github.com/apache/spark/pull/15416
@davies Other places all seem to be correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82721927
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,14 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ // The left most bit in the bitmasks corresponds to the last expression in groupByAliases
+ // with 0 indicating this expression is in the grouping set. The following line of code
+ // calculates the bit mask representing the expressions that exist in all the grouping sets.
+ val nonNullBitmask = ~ x.bitmasks.reduce(_ | _)
--- End diff --
Do you mean `((nonNullBitmask >> (attrLength - idx - 1)) & 1) == 1`? We can only test on `0` if we left shift `1`, right? @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15416
@yangw1234 can you fix the scala styles?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82664096
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,11 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ val nonNullBitmask = ~ x.bitmasks.reduce(_ | _)
--- End diff --
This is generally lacking documentation. Could you add a line or two to explain how the bitmasks are structured.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82556934
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ---
@@ -2189,6 +2189,24 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
}
}
+ test("SPARK-17849: grouping set throws NPE") {
--- End diff --
maybe we can move this into SQLQueryTestSuite, by creating a new grouping_set.q file??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82715694
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,11 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ val nonNullBitmask = ~ x.bitmasks.reduce(_ | _)
--- End diff --
@hvanhovell @rxin comments are added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15416
**[Test build #3337 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3337/consoleFull)** for PR 15416 at commit [`69f6e4f`](https://github.com/apache/spark/commit/69f6e4f1bc37afd6b3ca529c8b0f0afec891459a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/15416
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15416: [SPARK-17849] [SQL] Fix NPE problem when using grouping ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15416
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15416: [SPARK-17849] [SQL] Fix NPE problem when using gr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15416#discussion_r82710433
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -298,10 +298,11 @@ class Analyzer(
case other => Alias(other, other.toString)()
}
- val nonNullBitmask = x.bitmasks.reduce(_ & _)
+ val nonNullBitmask = ~ x.bitmasks.reduce(_ | _)
--- End diff --
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org