You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2017/10/25 02:12:28 UTC
[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/19569
[SPARK-22348][SQL] The table cache providing ColumnarBatch should also do partition batch pruning
## What changes were proposed in this pull request?
We enable table cache `InMemoryTableScanExec` to provide `ColumnarBatch` now. But the cached batches are retrieved without pruning. In this case, we still need to do partition batch pruning.
## How was this patch tested?
Existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 SPARK-22348
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19569.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19569
----
commit f2c8266b4074bc387c4b46bb5b46cca82ff3929d
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date: 2017-10-25T02:11:09Z
Do partition batch pruning for ColumnarBatch path too.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19569
Good catch, thank you. Would it be possible to add a test case for pruning with table cache?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19569
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83027/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19569
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19569
@kiszk Thanks. I've roughly checked existing tests. Seems that there are related ones for pruning the table cache. Let me see if I can add one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19569
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83032/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19569
**[Test build #83031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83031/testReport)** for PR 19569 at commit [`b9e238c`](https://github.com/apache/spark/commit/b9e238cc42ea53a4e43af1db2ef2a36854cbc6e3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19569
good catch! merging to master, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19569
**[Test build #83027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83027/testReport)** for PR 19569 at commit [`f2c8266`](https://github.com/apache/spark/commit/f2c8266b4074bc387c4b46bb5b46cca82ff3929d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19569
**[Test build #83032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83032/testReport)** for PR 19569 at commit [`568e791`](https://github.com/apache/spark/commit/568e7917ef2fb04500d32fa254240f1f5f1ab9a5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19569
**[Test build #83027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83027/testReport)** for PR 19569 at commit [`f2c8266`](https://github.com/apache/spark/commit/f2c8266b4074bc387c4b46bb5b46cca82ff3929d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...
Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/19569#discussion_r146748032
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
@@ -201,35 +193,50 @@ case class InMemoryTableScanExec(
schema)
partitionFilter.initialize(index)
+ // Do partition batch pruning if enabled
+ if (inMemoryPartitionPruningEnabled) {
+ cachedBatchIterator.filter { cachedBatch =>
+ if (!partitionFilter.eval(cachedBatch.stats)) {
+ logDebug {
+ val statsString = schemaIndex.map { case (a, i) =>
+ val value = cachedBatch.stats.get(i, a.dataType)
+ s"${a.name}: $value"
+ }.mkString(", ")
+ s"Skipping partition based on stats $statsString"
+ }
+ false
+ } else {
+ true
+ }
+ }
+ } else {
+ cachedBatchIterator
+ }
+ }
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ val numOutputRows = longMetric("numOutputRows")
+
+ if (enableAccumulators) {
+ readPartitions.setValue(0)
+ readBatches.setValue(0)
+ }
+
+ // Using these variables here to avoid serialization of entire objects (if referenced directly)
+ // within the map Partitions closure.
+ val relOutput: AttributeSeq = relation.output
+ val buffers = filteredCachedBatches()
--- End diff --
this seems to be a leftover of copy-paste
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19569
**[Test build #83031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83031/testReport)** for PR 19569 at commit [`b9e238c`](https://github.com/apache/spark/commit/b9e238cc42ea53a4e43af1db2ef2a36854cbc6e3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19569#discussion_r146749535
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
@@ -201,35 +193,50 @@ case class InMemoryTableScanExec(
schema)
partitionFilter.initialize(index)
+ // Do partition batch pruning if enabled
+ if (inMemoryPartitionPruningEnabled) {
+ cachedBatchIterator.filter { cachedBatch =>
+ if (!partitionFilter.eval(cachedBatch.stats)) {
+ logDebug {
+ val statsString = schemaIndex.map { case (a, i) =>
+ val value = cachedBatch.stats.get(i, a.dataType)
+ s"${a.name}: $value"
+ }.mkString(", ")
+ s"Skipping partition based on stats $statsString"
+ }
+ false
+ } else {
+ true
+ }
+ }
+ } else {
+ cachedBatchIterator
+ }
+ }
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ val numOutputRows = longMetric("numOutputRows")
+
+ if (enableAccumulators) {
+ readPartitions.setValue(0)
+ readBatches.setValue(0)
+ }
+
+ // Using these variables here to avoid serialization of entire objects (if referenced directly)
+ // within the map Partitions closure.
+ val relOutput: AttributeSeq = relation.output
+ val buffers = filteredCachedBatches()
--- End diff --
Removed. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19569
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19569
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19569
also cc @cloud-fan for review.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19569
**[Test build #83032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83032/testReport)** for PR 19569 at commit [`568e791`](https://github.com/apache/spark/commit/568e7917ef2fb04500d32fa254240f1f5f1ab9a5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19569
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83031/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19569
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org