You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/05/30 07:26:03 UTC
[GitHub] spark pull request #21460: [SPARK-23442][SQL] Improvement reading from parti...
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/21460
[SPARK-23442][SQL] Improvement reading from partitioned and bucketed table.
## What changes were proposed in this pull request?
For a partitioned and bucketed table. With the increasing number of partitions, the amount of data is getting larger and larger. Reading this table always uses the `bucket number` of tasks.
This PR changes the logic to `bucket number` * `partition number` when reading partitioned and bucketed table.
## How was this patch tested?
manual tests.
```scala
spark.range(10000).selectExpr(
"id as key",
"id % 5 as t1",
"id % 10 as p").repartition(5, col("p")).write.partitionBy("p").bucketBy(5,
"key").sortBy("t1").saveAsTable("spark_23442")
```
```scala
// All partition: partition number = 5 * 10 = 50
spark.sql("select count(distinct t1) from spark_23442 ").show
```
```scala
// Filtered 1/2 partition: partition number = 5 * (10 / 2) = 25
spark.sql("select count(distinct t1) from spark_23442 where p >= 5 ").show
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-23442
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21460.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21460
----
commit 58e4e098016051f41103464040ba24bbee28b2cf
Author: Yuming Wang <yu...@...>
Date: 2018-05-30T06:53:52Z
Improvement reading from partitioned and bucketed table.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3699/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Increase reading tasks when reading f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21460
**[Test build #91293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91293/testReport)** for PR 21460 at commit [`58e4e09`](https://github.com/apache/spark/commit/58e4e098016051f41103464040ba24bbee28b2cf).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Increase reading tasks when reading f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91293/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21460
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21460
**[Test build #91293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91293/testReport)** for PR 21460 at commit [`58e4e09`](https://github.com/apache/spark/commit/58e4e098016051f41103464040ba24bbee28b2cf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21460
**[Test build #91291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91291/testReport)** for PR 21460 at commit [`58e4e09`](https://github.com/apache/spark/commit/58e4e098016051f41103464040ba24bbee28b2cf).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Increase reading tasks when reading f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3697/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21460: [SPARK-23442][SQL] Increase reading tasks when re...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum closed the pull request at:
https://github.com/apache/spark/pull/21460
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Increase reading tasks when reading f...
Posted by pnpranavrao <gi...@git.apache.org>.
Github user pnpranavrao commented on the issue:
https://github.com/apache/spark/pull/21460
The reason `createBucketedReadRDD` works the way currently (accumulate all buckets across partitions into a partition with `id` equal to `bucketId`) is to skip shuffle when joining on the bucketed columns. Your suggested change would break this.
I opened this [JIRA](https://issues.apache.org/jira/browse/SPARK-23442) for the same need. We need to generate different physical plans for join and non-join usecases, and not assume if a datasource is bucketed, it will be used for a join involving the bucketed columns like now.
The workaround right now is to turn off bucketing with a SparkConf flag, but I'm working on using the both partitioning and bucketing information to plan queries.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21460
**[Test build #91291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91291/testReport)** for PR 21460 at commit [`58e4e09`](https://github.com/apache/spark/commit/58e4e098016051f41103464040ba24bbee28b2cf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21460
(I'd avoid saying just an improvement in something in the PR title btw)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21460: [SPARK-23442][SQL] Improvement reading from partitioned ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91291/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org