You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by CodingCat <gi...@git.apache.org> on 2017/11/24 07:42:51 UTC

[GitHub] spark pull request #19810: Partition level pruning 2

GitHub user CodingCat opened a pull request:

    https://github.com/apache/spark/pull/19810

    Partition level pruning 2

    ## What changes were proposed in this pull request?
    
    In the current implementation of Spark, InMemoryTableExec read all data in a cached table, filter CachedBatch according to stats and pass data to the downstream operators. This implementation makes it inefficient to reside the whole table in memory to serve various queries against different partitions of the table, which occupies a certain portion of our users' scenarios.
    
    The following is an example of such a use case:
    
    store_sales is a 1TB-sized table in cloud storage, which is partitioned by 'location'. The first query, Q1, wants to output several metrics A, B, C for all stores in all locations. After that, a small team of 3 data scientists wants to do some causal analysis for the sales in different locations. To avoid unnecessary I/O and parquet/orc parsing overhead, they want to cache the whole table in memory in Q1.
    
    With the current implementation, even any one of the data scientists is only interested in one out of three locations, the queries they submit to Spark cluster is still reading 1TB data completely.
    
    The reason behind the extra reading operation is that we implement CachedBatch as
    
    case class CachedBatch(numRows: Int, buffers: Array[Array[Byte]], stats: InternalRow)
    where the stats is a part of every CachedBatch, so we can only filter batches for output of InMemoryTableExec operator by reading all data in in-memory table as input. The extra reading would be even more unacceptable when some of the table's data is evicted to disks.
    
    We propose to introduce a new type of block, metadata block, for the partitions of RDD representing data in the cached table. Every metadata block contains stats info for all columns in a partition and is saved to BlockManager when executing compute() method for the partition. To minimize the number of bytes to read,
    
    ## How was this patch tested?
    
    (TBD, post it soon)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/CodingCat/spark partition_level_pruning_2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19810.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19810
    
----
commit c3f1a9bb4b53ceef8ad3fb31b3164665549f8bc9
Author: CodingCat <zh...@gmail.com>
Date:   2016-03-07T14:37:37Z

    improve the doc for "spark.memory.offHeap.size"

commit 6a2b3ca56d26f5fb03d165466e9b6edadeb0adac
Author: CodingCat <zh...@gmail.com>
Date:   2016-03-07T19:00:16Z

    fix

commit 6e37fa2d06107c175cede4ff1b1a65dd14de7d5a
Author: CodingCat <zh...@gmail.com>
Date:   2017-10-27T17:36:08Z

    add configuration for partition_metadata

commit aa7066038e23cea7c5eb720aa70cd1a84d6f751f
Author: CodingCat <zh...@gmail.com>
Date:   2017-10-27T19:49:33Z

    framework of CachedColumnarRDD

commit d1380821d4a9cbb45f5b28c775939e0d130f0922
Author: CodingCat <zh...@gmail.com>
Date:   2017-10-27T22:53:38Z

    code framework

commit a72d7798b17000250eda0bcc8ae726b7dce7aa0c
Author: CodingCat <zh...@gmail.com>
Date:   2017-10-30T20:41:18Z

    remove cachedcolumnarbatchRDD

commit 0fe35f82fcd9175b409429c03c9f7b33712df8ae
Author: CodingCat <zh...@gmail.com>
Date:   2017-10-30T21:12:24Z

    fix styly error

commit 9e342432a82cbe325f338bd787be6427002981e1
Author: CodingCat <zh...@gmail.com>
Date:   2017-10-31T00:01:53Z

    temp

commit 677ca81709fa34b3cad13f5843b8f408e401476b
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-01T23:01:49Z

    'CachedColumnarRDD'

commit df1d79620f7c8073b6bc1a119a245c3d5413ec71
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-01T23:11:52Z

    change types

commit 08fd0857024a192307e66b6c3cffb19ae879000b
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-01T23:16:58Z

    fix compilation error

commit d4fc2b772b60161a24892133bdae2ff28233250a
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T17:13:15Z

    update

commit 97a63d6d1c1cd81bb97f0b998e716b13f5bd92d9
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T17:21:23Z

    fix storage level

commit a24b7bbfa6c393974d553ba703777934bed85ec1
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T17:42:25Z

    fix getOrCompute

commit 0e8e6395df97b4adeabb5a32ad441b96e5c19eb9
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T17:55:36Z

    evaluate with partition metadata

commit b89d58b26650ce9b63713a8a2371e280986720bb
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T18:21:19Z

    fix getOrCompute

commit 3f2eae73cefd5676c799a6fd1e384556ee6c33a8
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T19:06:49Z

    add logging

commit 507c1a22da7de8b5bd24c066ec61a1c5f1c604dd
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T19:12:57Z

    add logging for skipped partition

commit 40d441cc6985aa995e5c7fe1387744381425e6ef
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T20:15:29Z

    try to print stats

commit 520e5aab3a06b9c1599af3984d4d2d028ced2ad6
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T20:20:43Z

    add logging for skipped partition

commit 885808fc0c0f78d0ce7bbb5a5d06222bb06bf2cb
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T20:25:18Z

    add logging for skipped partition

commit 37b5971524c4bb7f42b23a2ea013564ce3f6015c
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-02T21:14:57Z

    add logging for skipped partition

commit 4dbfe37d6d719e7755bb6c9160262201908d36f4
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T21:23:33Z

    refactor the code

commit 6165838371bbe345f6bc9d462671ffb3f32cbb94
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T21:30:57Z

    fix compilation issue

commit 05f226772e26ab81b764632a5d979eca4e4deffd
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T21:33:51Z

    refactor the code

commit bcafe822a879bf48982f6d9f3255cbc7a5e8236c
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T21:49:14Z

    test

commit 5b888d303a2b68fd36c293286e8bf6e67c06802c
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T21:50:11Z

    fix compilation issue

commit 977b93f4b67ff5e57f1d0081b5aead70a90ac13f
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T22:11:04Z

    add missing filtering

commit 9c9bcadf6f7f2ae90393f1ab26db595a0a836c99
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T22:16:43Z

    test

commit 56a430722fc7b76b3f2365eee4d94e6ab62380a6
Author: CodingCat <zh...@gmail.com>
Date:   2017-11-09T22:22:23Z

    test

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    If the author is inactive, its ok to take this over by someone. But, we should first discuss more to get consensus for this feature. I'm not sure we could get enough performance gains at the expense of the current simple cache logic.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    @cloud-fan for this case, if the data has been dumped to disk or some non-local tasks are started, I/O is involved in addition to the overhead to start extra tasks. If all data is in-memory, only the overhead related to tasks' launching is there
    
    
    > It sounds like something can be done better in 3rd party data sources, or we need to change the Spark core just for a better table cache, which seems risky.
    
    Yes, some work can be done in 3rd party data sources, e.g. to avoid parsing overhead in parquet, 
    
    Regarding the risk, in the current implementation, I directly modify the core part to add a new type of block and make it recognizable by BlockManager. The new RDD and dependency implementation is in SQL module. An alternative way to do that is implementing this new type block in SQL as well (but it needs some small refactoring to make BlockManager open to anything outside of Spark Core)
    
    I personally think it's a good feature to add without that much threat to existing code


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84178/testReport)** for PR 19810 at commit [`a853ce6`](https://github.com/apache/spark/commit/a853ce6ffa8c8b09271441473bcf7c1f9d463a9d).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r218714493
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/CachedColumnarRDD.scala ---
    @@ -0,0 +1,99 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.columnar
    +
    +import java.util.concurrent.ConcurrentHashMap
    +
    +import scala.collection.JavaConverters._
    +import scala.collection.mutable
    +
    +import org.apache.spark._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.storage.{RDDPartitionMetadataBlockId, StorageLevel}
    +
    +private[columnar] class CachedColumnarRDD(
    +    @transient private var _sc: SparkContext,
    +    private var dataRDD: RDD[CachedBatch],
    +    private[columnar] val containsPartitionMetadata: Boolean,
    +    expectedStorageLevel: StorageLevel)
    +  extends RDD[CachedBatch](_sc, Seq(new OneToOneDependency(dataRDD))) {
    +
    +  override def compute(split: Partition, context: TaskContext): Iterator[CachedBatch] = {
    +    firstParent.iterator(split, context)
    +  }
    +
    +  override def unpersist(blocking: Boolean = true): this.type = {
    +    CachedColumnarRDD.allMetadataFetched.remove(id)
    +    CachedColumnarRDD.rddIdToMetadata.remove(id)
    +    super.unpersist(blocking)
    +  }
    +
    +  override protected def getPartitions: Array[Partition] = dataRDD.partitions
    +
    +  override private[spark] def getOrCompute(split: Partition, context: TaskContext):
    +      Iterator[CachedBatch] = {
    +    val metadataBlockId = RDDPartitionMetadataBlockId(id, split.index)
    +    val superGetOrCompute: (Partition, TaskContext) => Iterator[CachedBatch] = super.getOrCompute
    +    SparkEnv.get.blockManager.getSingle[InternalRow](metadataBlockId).map(_ =>
    +      superGetOrCompute(split, context)
    +    ).getOrElse {
    +      val batchIter = superGetOrCompute(split, context)
    +      if (containsPartitionMetadata && getStorageLevel != StorageLevel.NONE && batchIter.hasNext) {
    +        val cachedBatch = batchIter.next()
    +        SparkEnv.get.blockManager.putSingle(metadataBlockId, cachedBatch.stats,
    +          expectedStorageLevel)
    +        new InterruptibleIterator[CachedBatch](context, Iterator(cachedBatch))
    +      } else {
    +        batchIter
    +      }
    +    }
    +  }
    +}
    +
    +private[columnar] object CachedColumnarRDD {
    +
    +  private val rddIdToMetadata = new ConcurrentHashMap[Int, mutable.ArraySeq[Option[InternalRow]]]()
    --- End diff --
    
    could these be moved to become a member of the RDD class? seems like a map of this->some.property, in this case can be made an instance member.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by sadikovi <gi...@git.apache.org>.
Github user sadikovi commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r154814573
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ---
    @@ -52,6 +52,68 @@ object InMemoryRelation {
     private[columnar]
     case class CachedBatch(numRows: Int, buffers: Array[Array[Byte]], stats: InternalRow)
     
    +private[columnar] class CachedBatchIterator(
    +    rowIterator: Iterator[InternalRow],
    +    output: Seq[Attribute],
    +    batchSize: Int,
    +    useCompression: Boolean,
    +    batchStats: LongAccumulator,
    +    singleBatchPerPartition: Boolean) extends Iterator[CachedBatch] {
    +
    +  def next(): CachedBatch = {
    +    val columnBuilders = output.map { attribute =>
    +      ColumnBuilder(attribute.dataType, batchSize, attribute.name, useCompression)
    +    }.toArray
    +
    +    var rowCount = 0
    +    var totalSize = 0L
    +
    +    val terminateLoop = (singleBatch: Boolean, rowIter: Iterator[InternalRow],
    --- End diff --
    
    @CodingCat Could you explain me what `singleBatch` means here? I cannot get my head around it:) Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    @cloud-fan  would you mind continuing the review?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r154844579
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ---
    @@ -52,6 +52,68 @@ object InMemoryRelation {
     private[columnar]
     case class CachedBatch(numRows: Int, buffers: Array[Array[Byte]], stats: InternalRow)
     
    +private[columnar] class CachedBatchIterator(
    +    rowIterator: Iterator[InternalRow],
    +    output: Seq[Attribute],
    +    batchSize: Int,
    +    useCompression: Boolean,
    +    batchStats: LongAccumulator,
    +    singleBatchPerPartition: Boolean) extends Iterator[CachedBatch] {
    +
    +  def next(): CachedBatch = {
    +    val columnBuilders = output.map { attribute =>
    +      ColumnBuilder(attribute.dataType, batchSize, attribute.name, useCompression)
    +    }.toArray
    +
    +    var rowCount = 0
    +    var totalSize = 0L
    +
    +    val terminateLoop = (singleBatch: Boolean, rowIter: Iterator[InternalRow],
    --- End diff --
    
    to make getting partition stats easier, we construct only one CatchedBatch for each partition when enabling the functionality proposed in this PR. `singleBatch` distinguishes the scenarios which enables/disables the functionality by introducing different while loop termination conditions, making the other code reusable


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84153 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84153/testReport)** for PR 19810 at commit [`accd549`](https://github.com/apache/spark/commit/accd5493464d5bd5f7401285defbd8ca2628ae68).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    @CodingCat Is this still active?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    When all data is in memory, what do you mean by reading less data? Starting less tasks makes sense.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Hi, @cloud-fan, this PR is not only for the case where the data size is larger than the memory size, even when all data is in-memory, I observed up to 10-40% speedup  because the implementation here
    
    (1) read less data
    
    (2) started less tasks
    
    you can understand this PR as it implement the functionality of Parquet's footer for the in-memory table
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84151 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84151/testReport)** for PR 19810 at commit [`b4f51ed`](https://github.com/apache/spark/commit/b4f51ed0c0774cfea5d26fffcf6abe6f07775a92).
     * This patch **fails due to an unknown error code, -9**.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #97714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97714/testReport)** for PR 19810 at commit [`9d450ad`](https://github.com/apache/spark/commit/9d450addd095e2ac78770bdd03c3bf77817379e8).
     * This patch **fails to generate documentation**.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84240/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97714/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r218716111
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/CachedColumnarRDD.scala ---
    @@ -0,0 +1,99 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.columnar
    +
    +import java.util.concurrent.ConcurrentHashMap
    +
    +import scala.collection.JavaConverters._
    +import scala.collection.mutable
    +
    +import org.apache.spark._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.storage.{RDDPartitionMetadataBlockId, StorageLevel}
    +
    +private[columnar] class CachedColumnarRDD(
    +    @transient private var _sc: SparkContext,
    +    private var dataRDD: RDD[CachedBatch],
    +    private[columnar] val containsPartitionMetadata: Boolean,
    +    expectedStorageLevel: StorageLevel)
    +  extends RDD[CachedBatch](_sc, Seq(new OneToOneDependency(dataRDD))) {
    +
    +  override def compute(split: Partition, context: TaskContext): Iterator[CachedBatch] = {
    +    firstParent.iterator(split, context)
    +  }
    +
    +  override def unpersist(blocking: Boolean = true): this.type = {
    +    CachedColumnarRDD.allMetadataFetched.remove(id)
    +    CachedColumnarRDD.rddIdToMetadata.remove(id)
    +    super.unpersist(blocking)
    +  }
    +
    +  override protected def getPartitions: Array[Partition] = dataRDD.partitions
    +
    +  override private[spark] def getOrCompute(split: Partition, context: TaskContext):
    +      Iterator[CachedBatch] = {
    --- End diff --
    
    can this be avoided by maintaining two (zipped) RDDs? one of `CachedBatch`s and the other holding only the stats?
    can this approach avoid the need for a specialized block type for managing metadata? 
    correct me if i'm wrong, but first time the stats are accessed, your approach performs a full scan to extract the stats (happens in the sql code), so having a second RDD which is something like : `batches.map(_.stats).persist` should give the same behavior, right?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    reading less data is a observation from the input metrics in Spark UI which includes both of local/remote read in BlockManagers, and also the overhead in BlockManager layer itself (especially when the user chooses to cache with serialized format)
    
    but I didn't count how much it contributes to the speedup (and a small portion of data is in disk in my perf test)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r154845669
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -193,38 +195,68 @@ case class InMemoryTableScanExec(
     
       private val inMemoryPartitionPruningEnabled = sqlContext.conf.inMemoryPartitionPruning
     
    +  private def doFilterCachedBatches(
    +      rdd: RDD[CachedBatch],
    +      partitionStatsSchema: Seq[AttributeReference]): RDD[CachedBatch] = {
    +    val schemaIndex = partitionStatsSchema.zipWithIndex
    +    rdd.mapPartitionsWithIndex {
    +      case (partitionIndex, cachedBatches) =>
    +        if (inMemoryPartitionPruningEnabled) {
    +          cachedBatches.filter { cachedBatch =>
    +            val partitionFilter = newPredicate(
    +              partitionFilters.reduceOption(And).getOrElse(Literal(true)),
    +              partitionStatsSchema)
    +            partitionFilter.initialize(partitionIndex)
    +            if (!partitionFilter.eval(cachedBatch.stats)) {
    --- End diff --
    
    I might not understand your proposal well...are you trying to simplify the logic in https://github.com/apache/spark/pull/19810/files#diff-5fc188468d3066580ea9a766114b8f1dR74? it would make the code simpler but degrade pruning effect here, 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84153/testReport)** for PR 19810 at commit [`accd549`](https://github.com/apache/spark/commit/accd5493464d5bd5f7401285defbd8ca2628ae68).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by sadikovi <gi...@git.apache.org>.
Github user sadikovi commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r154848470
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -193,38 +195,68 @@ case class InMemoryTableScanExec(
     
       private val inMemoryPartitionPruningEnabled = sqlContext.conf.inMemoryPartitionPruning
     
    +  private def doFilterCachedBatches(
    +      rdd: RDD[CachedBatch],
    +      partitionStatsSchema: Seq[AttributeReference]): RDD[CachedBatch] = {
    +    val schemaIndex = partitionStatsSchema.zipWithIndex
    +    rdd.mapPartitionsWithIndex {
    +      case (partitionIndex, cachedBatches) =>
    +        if (inMemoryPartitionPruningEnabled) {
    +          cachedBatches.filter { cachedBatch =>
    +            val partitionFilter = newPredicate(
    +              partitionFilters.reduceOption(And).getOrElse(Literal(true)),
    +              partitionStatsSchema)
    +            partitionFilter.initialize(partitionIndex)
    +            if (!partitionFilter.eval(cachedBatch.stats)) {
    --- End diff --
    
    All good, no need to change. I was trying to understand the code, so my question would be referring to statistics collection overall, not changes in this PR. Link points to a condition (exists before this PR) that could potentially result in exiting iterator before exhausting all records in it, so statistics would be partially collected, which might affect any filtering that uses such statistics - though it is quite possibly handled later, or a theoretical use case.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84178/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84181/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4244/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84152/testReport)** for PR 19810 at commit [`accd549`](https://github.com/apache/spark/commit/accd5493464d5bd5f7401285defbd8ca2628ae68).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84181/testReport)** for PR 19810 at commit [`9d450ad`](https://github.com/apache/spark/commit/9d450addd095e2ac78770bdd03c3bf77817379e8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84240/testReport)** for PR 19810 at commit [`9d450ad`](https://github.com/apache/spark/commit/9d450addd095e2ac78770bdd03c3bf77817379e8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    > store_sales is a 1TB-sized table in cloud storage, which is partitioned by 'location'. The first query, Q1, wants to output several metrics A, B, C for all stores in all locations. After that, a small team of 3 data scientists wants to do some causal analysis for the sales in different locations. To avoid unnecessary I/O and parquet/orc parsing overhead, they want to cache the whole table in memory in Q1.
    
    Reading your use case, it sounds like you are trying to optimize the case that data is too large to fit in memory. For that case, if a partition is on the disk, Spark needs to load the entire partition to memory before filtering blocks.
    
    It sounds like something can be done better in 3rd party data sources, or we need to change the Spark core just for a better table cache, which seems risky.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84181/testReport)** for PR 19810 at commit [`9d450ad`](https://github.com/apache/spark/commit/9d450addd095e2ac78770bdd03c3bf77817379e8).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r155392065
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -193,38 +195,68 @@ case class InMemoryTableScanExec(
     
       private val inMemoryPartitionPruningEnabled = sqlContext.conf.inMemoryPartitionPruning
     
    +  private def doFilterCachedBatches(
    +      rdd: RDD[CachedBatch],
    +      partitionStatsSchema: Seq[AttributeReference]): RDD[CachedBatch] = {
    +    val schemaIndex = partitionStatsSchema.zipWithIndex
    +    rdd.mapPartitionsWithIndex {
    +      case (partitionIndex, cachedBatches) =>
    +        if (inMemoryPartitionPruningEnabled) {
    +          cachedBatches.filter { cachedBatch =>
    +            val partitionFilter = newPredicate(
    +              partitionFilters.reduceOption(And).getOrElse(Literal(true)),
    +              partitionStatsSchema)
    +            partitionFilter.initialize(partitionIndex)
    +            if (!partitionFilter.eval(cachedBatch.stats)) {
    --- End diff --
    
    @sadikovi this while loop is building CatchedBatch, it just decides what's the right time to seal the building window of a CatchedBatch and start the next one....so, in any way, you need to go through every all records in the partition, 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4255/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84152/testReport)** for PR 19810 at commit [`accd549`](https://github.com/apache/spark/commit/accd5493464d5bd5f7401285defbd8ca2628ae68).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84153/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    jenkins, retest it please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r218743585
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/CachedColumnarRDD.scala ---
    @@ -0,0 +1,99 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.columnar
    +
    +import java.util.concurrent.ConcurrentHashMap
    +
    +import scala.collection.JavaConverters._
    +import scala.collection.mutable
    +
    +import org.apache.spark._
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.storage.{RDDPartitionMetadataBlockId, StorageLevel}
    +
    +private[columnar] class CachedColumnarRDD(
    +    @transient private var _sc: SparkContext,
    +    private var dataRDD: RDD[CachedBatch],
    +    private[columnar] val containsPartitionMetadata: Boolean,
    +    expectedStorageLevel: StorageLevel)
    +  extends RDD[CachedBatch](_sc, Seq(new OneToOneDependency(dataRDD))) {
    +
    +  override def compute(split: Partition, context: TaskContext): Iterator[CachedBatch] = {
    +    firstParent.iterator(split, context)
    +  }
    +
    +  override def unpersist(blocking: Boolean = true): this.type = {
    +    CachedColumnarRDD.allMetadataFetched.remove(id)
    +    CachedColumnarRDD.rddIdToMetadata.remove(id)
    +    super.unpersist(blocking)
    +  }
    +
    +  override protected def getPartitions: Array[Partition] = dataRDD.partitions
    +
    +  override private[spark] def getOrCompute(split: Partition, context: TaskContext):
    +      Iterator[CachedBatch] = {
    +    val metadataBlockId = RDDPartitionMetadataBlockId(id, split.index)
    +    val superGetOrCompute: (Partition, TaskContext) => Iterator[CachedBatch] = super.getOrCompute
    +    SparkEnv.get.blockManager.getSingle[InternalRow](metadataBlockId).map(_ =>
    +      superGetOrCompute(split, context)
    +    ).getOrElse {
    +      val batchIter = superGetOrCompute(split, context)
    +      if (containsPartitionMetadata && getStorageLevel != StorageLevel.NONE && batchIter.hasNext) {
    +        val cachedBatch = batchIter.next()
    --- End diff --
    
    assert `!batchIter.hasNext`, you expect this partition to contain a single batch


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SQL][SPARK-22599] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84151/testReport)** for PR 19810 at commit [`b4f51ed`](https://github.com/apache/spark/commit/b4f51ed0c0774cfea5d26fffcf6abe6f07775a92).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #97714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97714/testReport)** for PR 19810 at commit [`9d450ad`](https://github.com/apache/spark/commit/9d450addd095e2ac78770bdd03c3bf77817379e8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    are you trying to optimize the case that data is too large to fit in memory? Spark RDD cache doesn't work well for this case.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84151/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84155/testReport)** for PR 19810 at commit [`a853ce6`](https://github.com/apache/spark/commit/a853ce6ffa8c8b09271441473bcf7c1f9d463a9d).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    When I contributed it back, the community is as looking at something else, so I didn’t spend too much time to convince the people to review....but if the interests are raised again now, I am happy to pick it up again, 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    @maropu , this looks rather cold :sunglasses: , but extremely interesting and relevant.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84178/testReport)** for PR 19810 at commit [`a853ce6`](https://github.com/apache/spark/commit/a853ce6ffa8c8b09271441473bcf7c1f9d463a9d).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84155/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    @sadikovi thanks for the review, I replied in comments


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r218704390
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ---
    @@ -52,6 +52,68 @@ object InMemoryRelation {
     private[columnar]
     case class CachedBatch(numRows: Int, buffers: Array[Array[Byte]], stats: InternalRow)
     
    +private[columnar] class CachedBatchIterator(
    +    rowIterator: Iterator[InternalRow],
    +    output: Seq[Attribute],
    +    batchSize: Int,
    +    useCompression: Boolean,
    +    batchStats: LongAccumulator,
    +    singleBatchPerPartition: Boolean) extends Iterator[CachedBatch] {
    +
    +  def next(): CachedBatch = {
    +    val columnBuilders = output.map { attribute =>
    +      ColumnBuilder(attribute.dataType, batchSize, attribute.name, useCompression)
    +    }.toArray
    +
    +    var rowCount = 0
    +    var totalSize = 0L
    +
    +    val terminateLoop = (singleBatch: Boolean, rowIter: Iterator[InternalRow],
    +      rowCount: Int, size: Long) => {
    +      if (!singleBatch) {
    +        rowIter.hasNext && rowCount < batchSize && totalSize < ColumnBuilder.MAX_BATCH_SIZE_IN_BYTE
    +      } else {
    +        rowIter.hasNext
    --- End diff --
    
    doesn't this run the risk of OOM for large partitions?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r218705219
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -193,38 +195,68 @@ case class InMemoryTableScanExec(
     
       private val inMemoryPartitionPruningEnabled = sqlContext.conf.inMemoryPartitionPruning
     
    +  private def doFilterCachedBatches(
    +      rdd: RDD[CachedBatch],
    +      partitionStatsSchema: Seq[AttributeReference]): RDD[CachedBatch] = {
    +    val schemaIndex = partitionStatsSchema.zipWithIndex
    +    rdd.mapPartitionsWithIndex {
    +      case (partitionIndex, cachedBatches) =>
    +        if (inMemoryPartitionPruningEnabled) {
    +          cachedBatches.filter { cachedBatch =>
    +            val partitionFilter = newPredicate(
    --- End diff --
    
    can this be pulled up out of `usePartitionLevelMetadata` ? seems like you're constructing the predicate per record


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84240/testReport)** for PR 19810 at commit [`9d450ad`](https://github.com/apache/spark/commit/9d450addd095e2ac78770bdd03c3bf77817379e8).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

Posted by sadikovi <gi...@git.apache.org>.
Github user sadikovi commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19810#discussion_r154815012
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ---
    @@ -193,38 +195,68 @@ case class InMemoryTableScanExec(
     
       private val inMemoryPartitionPruningEnabled = sqlContext.conf.inMemoryPartitionPruning
     
    +  private def doFilterCachedBatches(
    +      rdd: RDD[CachedBatch],
    +      partitionStatsSchema: Seq[AttributeReference]): RDD[CachedBatch] = {
    +    val schemaIndex = partitionStatsSchema.zipWithIndex
    +    rdd.mapPartitionsWithIndex {
    +      case (partitionIndex, cachedBatches) =>
    +        if (inMemoryPartitionPruningEnabled) {
    +          cachedBatches.filter { cachedBatch =>
    +            val partitionFilter = newPredicate(
    +              partitionFilters.reduceOption(And).getOrElse(Literal(true)),
    +              partitionStatsSchema)
    +            partitionFilter.initialize(partitionIndex)
    +            if (!partitionFilter.eval(cachedBatch.stats)) {
    --- End diff --
    
    Are there any issues with discarding a partition based on statistics that could be partially computed (e.g. when total size in bytes of a partition iterator is larger than configurable batch size) as per https://github.com/apache/spark/pull/19810/files#diff-5fc188468d3066580ea9a766114b8f1dR74? 
    
    Would be it be beneficial to record such situation by logging it, and still include such partition when statistics are partially computed and filters are evaluated to false, or discard all statistics when some of the partitions hit this situation? Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    **[Test build #84155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84155/testReport)** for PR 19810 at commit [`a853ce6`](https://github.com/apache/spark/commit/a853ce6ffa8c8b09271441473bcf7c1f9d463a9d).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84152/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Retest this please.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    ping @cloud-fan @viirya @gatorsmile @felixcheung @hvanhovell @HyukjinKwon


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19810: [SPARK-22599][SQL] In-Memory Table Pruning without Extra...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19810
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4260/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org