You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pwoody <gi...@git.apache.org> on 2017/11/01 16:08:56 UTC
[GitHub] spark pull request #19629: [SPARK-22408][SQL] RelationalGroupedDataset's dis...
GitHub user pwoody opened a pull request:
https://github.com/apache/spark/pull/19629
[SPARK-22408][SQL] RelationalGroupedDataset's distinct pivot value calculation launches unnecessary stages
## What changes were proposed in this pull request?
Adding a global limit on top of the distinct values before sorting and collecting will reduce the overall work in the case where we have more distinct values. We will also eagerly perform a collect rather than a take because we know we only have at most (maxValues + 1) rows.
## How was this patch tested?
Existing tests cover sorted order
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwoody/spark SPARK-22408
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19629.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19629
----
commit aa809e39baf222e698315a5efb2d583cab99aad7
Author: Patrick Woody <pw...@palantir.com>
Date: 2017-11-01T15:44:51Z
SPARK-22408: reduce work of calculating pivot distinct values
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/19629
Jenkins, test this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on the issue:
https://github.com/apache/spark/pull/19629
diff LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19629: [SPARK-22408][SQL] RelationalGroupedDataset's dis...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19629
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19629
**[Test build #3971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3971/testReport)** for PR 19629 at commit [`aa809e3`](https://github.com/apache/spark/commit/aa809e39baf222e698315a5efb2d583cab99aad7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/19629
Merging in master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by pwoody <gi...@git.apache.org>.
Github user pwoody commented on the issue:
https://github.com/apache/spark/pull/19629
@rxin @aray since you were the last few to touch around here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19629
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19629: [SPARK-22408][SQL] RelationalGroupedDataset's distinct p...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19629
**[Test build #3971 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3971/testReport)** for PR 19629 at commit [`aa809e3`](https://github.com/apache/spark/commit/aa809e39baf222e698315a5efb2d583cab99aad7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org