You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2017/10/25 02:12:28 UTC

[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

GitHub user viirya opened a pull request:

    https://github.com/apache/spark/pull/19569

    [SPARK-22348][SQL] The table cache providing ColumnarBatch should also do partition batch pruning

    ## What changes were proposed in this pull request?
    
    We enable table cache `InMemoryTableScanExec` to provide `ColumnarBatch` now. But the cached batches are retrieved without pruning. In this case, we still need to do partition batch pruning.
    
    ## How was this patch tested?
    
    Existing tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/viirya/spark-1 SPARK-22348

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19569.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19569
    
----
commit f2c8266b4074bc387c4b46bb5b46cca82ff3929d
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date:   2017-10-25T02:11:09Z

    Do partition batch pruning for ColumnarBatch path too.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Good catch, thank you. Would it be possible to add a test case for pruning with table cache?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83027/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    @kiszk Thanks. I've roughly checked existing tests. Seems that there are related ones for pruning the table cache. Let me see if I can add one.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83032/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    **[Test build #83031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83031/testReport)** for PR 19569 at commit [`b9e238c`](https://github.com/apache/spark/commit/b9e238cc42ea53a4e43af1db2ef2a36854cbc6e3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    good catch! merging to master, thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    **[Test build #83027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83027/testReport)** for PR 19569 at commit [`f2c8266`](https://github.com/apache/spark/commit/f2c8266b4074bc387c4b46bb5b46cca82ff3929d).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    **[Test build #83032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83032/testReport)** for PR 19569 at commit [`568e791`](https://github.com/apache/spark/commit/568e7917ef2fb04500d32fa254240f1f5f1ab9a5).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    **[Test build #83027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83027/testReport)** for PR 19569 at commit [`f2c8266`](https://github.com/apache/spark/commit/f2c8266b4074bc387c4b46bb5b46cca82ff3929d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19569#discussion_r146748032
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -201,35 +193,50 @@ case class InMemoryTableScanExec(
             schema)
           partitionFilter.initialize(index)
     
    +      // Do partition batch pruning if enabled
    +      if (inMemoryPartitionPruningEnabled) {
    +        cachedBatchIterator.filter { cachedBatch =>
    +          if (!partitionFilter.eval(cachedBatch.stats)) {
    +            logDebug {
    +              val statsString = schemaIndex.map { case (a, i) =>
    +                val value = cachedBatch.stats.get(i, a.dataType)
    +                s"${a.name}: $value"
    +              }.mkString(", ")
    +              s"Skipping partition based on stats $statsString"
    +            }
    +            false
    +          } else {
    +            true
    +          }
    +        }
    +      } else {
    +        cachedBatchIterator
    +      }
    +    }
    +  }
    +
    +  protected override def doExecute(): RDD[InternalRow] = {
    +    val numOutputRows = longMetric("numOutputRows")
    +
    +    if (enableAccumulators) {
    +      readPartitions.setValue(0)
    +      readBatches.setValue(0)
    +    }
    +
    +    // Using these variables here to avoid serialization of entire objects (if referenced directly)
    +    // within the map Partitions closure.
    +    val relOutput: AttributeSeq = relation.output
    +    val buffers = filteredCachedBatches()
    --- End diff --
    
    this seems to be a leftover of copy-paste


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    **[Test build #83031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83031/testReport)** for PR 19569 at commit [`b9e238c`](https://github.com/apache/spark/commit/b9e238cc42ea53a4e43af1db2ef2a36854cbc6e3).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19569#discussion_r146749535
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -201,35 +193,50 @@ case class InMemoryTableScanExec(
             schema)
           partitionFilter.initialize(index)
     
    +      // Do partition batch pruning if enabled
    +      if (inMemoryPartitionPruningEnabled) {
    +        cachedBatchIterator.filter { cachedBatch =>
    +          if (!partitionFilter.eval(cachedBatch.stats)) {
    +            logDebug {
    +              val statsString = schemaIndex.map { case (a, i) =>
    +                val value = cachedBatch.stats.get(i, a.dataType)
    +                s"${a.name}: $value"
    +              }.mkString(", ")
    +              s"Skipping partition based on stats $statsString"
    +            }
    +            false
    +          } else {
    +            true
    +          }
    +        }
    +      } else {
    +        cachedBatchIterator
    +      }
    +    }
    +  }
    +
    +  protected override def doExecute(): RDD[InternalRow] = {
    +    val numOutputRows = longMetric("numOutputRows")
    +
    +    if (enableAccumulators) {
    +      readPartitions.setValue(0)
    +      readBatches.setValue(0)
    +    }
    +
    +    // Using these variables here to avoid serialization of entire objects (if referenced directly)
    +    // within the map Partitions closure.
    +    val relOutput: AttributeSeq = relation.output
    +    val buffers = filteredCachedBatches()
    --- End diff --
    
    Removed. Thanks.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19569


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    also cc @cloud-fan for review.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    **[Test build #83032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83032/testReport)** for PR 19569 at commit [`568e791`](https://github.com/apache/spark/commit/568e7917ef2fb04500d32fa254240f1f5f1ab9a5).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83031/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19569
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org