You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by marmbrus <gi...@git.apache.org> on 2016/03/11 05:46:28 UTC

[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

GitHub user marmbrus opened a pull request:

    https://github.com/apache/spark/pull/11646

    [SPARK-XXXX][SQL] Add a strategy for planning partitioned and bucketed scans of files

    **I'll add the correct JIRA when JIRA comes back up...**
    
    This PR adds a new strategy, `FileSourceStrategy`, that can be used for planning scans of collections of files that might be partitioned or bucketed.
    
    Compared with the existing planning logic in `DataSourceStrategy` this version has the following desirable properties:
     - It removes the need to have `RDD`, `broadcastedHadoopConf` and other distributed concerns  in the public API of `org.apache.spark.sql.sources.FileFormat`
     - Partition column appending is delegated to the format to avoid an extra copy / devectorization when appending partition columns
     - It minimizes the amount of data that is shipped to each executor (i.e. it does not send the whole list of files to every worker in the form of a hadoop conf)
     - it natively supports bucketing files into partitions, and thus does not require coalescing / creating a `UnionRDD` with the correct partitioning.
     - Small files are automatically coalesced into fewer tasks using an approximate bin-packing algorithm.
    
    Currently only a testing source is planned / tested using this strategy.  In follow-up PRs we will port the existing formats to this API.
    
    A stub for `FileScanRDD` is also added, but most methods remain unimplemented.
    
    Other minor cleanups:
     - partition pruning is pushed into `FileCatalog` so both the new and old code paths can use this logic.  This will also allow future implementations to use indexes or other tricks (i.e. a MySQL metastore)
     - The partitions from the `FileCatalog` now propagate information about file sizes all the way up to the planner so we can intelligently spread files out.
     - `Array` -> `Seq` in some internal APIs to avoid unnecessary `toArray` calls
     - Rename `Partition` to `PartitionDirectory` to differentiate partitions used earlier in pruning from those where we have already enumerated the files and their sizes.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/marmbrus/spark fileStrategy

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11646
    
----
commit 94cede673339bddc083b77a1d8e95e31015dcae3
Author: Michael Armbrust <mi...@databricks.com>
Date:   2016-03-09T18:28:42Z

    WIP

commit 09df4c8f4ad9f9e83fbd9524afd8d49007ba12da
Author: Michael Armbrust <mi...@databricks.com>
Date:   2016-03-11T03:29:51Z

    WIP

commit ef6ddaf1a8e0d44e10f3a04d8357f945639075cd
Author: Michael Armbrust <mi...@databricks.com>
Date:   2016-03-11T03:51:16Z

    Merge remote-tracking branch 'apache/master' into fileStrategy
    
    Conflicts:
    	sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

commit 9990d41a59cbe23b93a592b6248aed259513aac5
Author: Michael Armbrust <mi...@databricks.com>
Date:   2016-03-11T03:52:27Z

    cleanup

commit 4f298458fee9e0ebe59b4b569921bc32d3e1d4b1
Author: Michael Armbrust <mi...@databricks.com>
Date:   2016-03-11T04:28:07Z

    more cleanup

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196574813
  
    **[Test build #53123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53123/consoleFull)** for PR 11646 at commit [`4744b97`](https://github.com/apache/spark/commit/4744b9736adaed89a2c47199338626bf57cd56f8).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `abstract class NativeDDLCommand(val sql: String) extends RunnableCommand `
      * `case class CreateDatabase(`
      * `case class CreateFunction(`
      * `case class AlterTableRename(`
      * `case class AlterTableSetProperties(`
      * `case class AlterTableUnsetProperties(`
      * `case class AlterTableSerDeProperties(`
      * `case class AlterTableStorageProperties(`
      * `case class AlterTableNotClustered(`
      * `case class AlterTableNotSorted(`
      * `case class AlterTableSkewed(`
      * `case class AlterTableNotSkewed(`
      * `case class AlterTableNotStoredAsDirs(`
      * `case class AlterTableSkewedLocation(`
      * `case class AlterTableAddPartition(`
      * `case class AlterTableRenamePartition(`
      * `case class AlterTableExchangePartition(`
      * `case class AlterTableDropPartition(`
      * `case class AlterTableArchivePartition(`
      * `case class AlterTableUnarchivePartition(`
      * `case class AlterTableSetFileFormat(`
      * `case class AlterTableSetLocation(`
      * `case class AlterTableTouch(`
      * `case class AlterTableCompact(`
      * `case class AlterTableMerge(`
      * `case class AlterTableChangeCol(`
      * `case class AlterTableAddCol(`
      * `case class AlterTableReplaceCol(`
      * `case class In(attribute: String, values: Array[Any]) extends Filter `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r55795675
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
    @@ -454,19 +454,51 @@ trait FileFormat {
           requiredColumns: Array[String],
           filters: Array[Filter],
           bucketSet: Option[BitSet],
    -      inputFiles: Array[FileStatus],
    +      inputFiles: Seq[FileStatus],
           broadcastedConf: Broadcast[SerializableConfiguration],
           options: Map[String, String]): RDD[InternalRow]
    +
    +  /**
    +   * Returns a function that can be used to read a single file in as an Iterator of InternalRow.
    +   *
    +   * @param partitionSchema The schema of the partition column row that will be present in each
    +   *                        PartitionedFile.  These columns should be prepended to the rows that
    +   *                        are produced by the iterator.
    +   * @param dataSchema The schema of the data that should be output for each row.  This may be a
    +   *                   subset of the columns that are present in the file if  column pruning has
    +   *                   occurred.
    +   * @param filters A set of filters than can optionally be used to reduce the number of rows output
    +   * @param options A set of string -> string configuration options.
    +   * @return
    +   */
    +  def buildReader(
    +      sqlContext: SQLContext,
    +      partitionSchema: StructType,
    +      dataSchema: StructType,
    +      filters: Seq[Filter],
    +      options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = {
    +    // TODO: Remove this default implementation when the other formats have been ported
    +    // Until then we guard in [[FileSourceStrategy]] to only call this method on supported formats.
    +    throw new UnsupportedOperationException(s"buildReader is not supported for $this")
    +  }
     }
     
     /**
    + * A collection of data files from a partitioned relation, along with the partition values in the
    + * form of an [[InternalRow]].
    + */
    +case class Partition(values: InternalRow, files: Seq[FileStatus])
    +
    +/**
      * An interface for objects capable of enumerating the files that comprise a relation as well
      * as the partitioning characteristics of those files.
      */
     trait FileCatalog {
       def paths: Seq[Path]
     
    -  def partitionSpec(schema: Option[StructType]): PartitionSpec
    +  def partitionSpec(): PartitionSpec
    +
    +  def listFiles(filters: Seq[Expression]): Seq[Partition]
    --- End diff --
    
    Is this used to do partition pruning?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/11646


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57447363
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    --- End diff --
    
    An [example snippet][1] is shown in PR #11936 to illustrate the problem.
    
    One apparent naive "fix" for this is to include at least one whole file in each partition. But this doesn't work well with large input files.
    
    [1]: https://github.com/apache/spark/pull/11936#issuecomment-201300143


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195522580
  
    **[Test build #52939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52939/consoleFull)** for PR 11646 at commit [`65596df`](https://github.com/apache/spark/commit/65596dff42ab5dba7869827c73a9d8bf801b0941).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195222056
  
    **[Test build #52900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52900/consoleFull)** for PR 11646 at commit [`7ad3119`](https://github.com/apache/spark/commit/7ad311903bc2a71c52132757adc4e28a2b83eafe).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196618401
  
    Thanks!  Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57938987
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    --- End diff --
    
    I don't think thats how it works.  In [`PhysicalOperation`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala#L41), when there are no projections, we use the output of the child as the list of projections (i.e. all columns).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57441074
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    --- End diff --
    
    Is it safe to split file ourselves? If it's a text file, how can we guarantee we don't break lines while splitting the file?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195530198
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195222466
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195530072
  
    **[Test build #52939 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52939/consoleFull)** for PR 11646 at commit [`65596df`](https://github.com/apache/spark/commit/65596dff42ab5dba7869827c73a9d8bf801b0941).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r55873151
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala ---
    @@ -0,0 +1,56 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import org.apache.spark.{Partition, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.sql.SQLContext
    +import org.apache.spark.sql.catalyst.InternalRow
    +
    +/**
    + * A single file that should be read, along with partition column values that
    + * need to be prepended to each row.  The reading should start at the first
    + * valid record found after `offset`.
    + */
    +case class PartitionedFile(
    +    partitionValues: InternalRow,
    +    filePath: String,
    +    start: Long,
    +    length: Long)
    --- End diff --
    
    Good question.  We should add locality information here, but it probably belongs in `FilePartition` since that is the actual unit of work distribution.
    
    I'll add a comment that this strategy does not yet take locality into account.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56083932
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
    @@ -454,19 +454,60 @@ trait FileFormat {
           requiredColumns: Array[String],
           filters: Array[Filter],
           bucketSet: Option[BitSet],
    -      inputFiles: Array[FileStatus],
    +      inputFiles: Seq[FileStatus],
           broadcastedConf: Broadcast[SerializableConfiguration],
           options: Map[String, String]): RDD[InternalRow]
    +
    +  /**
    +   * Returns a function that can be used to read a single file in as an Iterator of InternalRow.
    +   *
    +   * @param partitionSchema The schema of the partition column row that will be present in each
    +   *                        PartitionedFile.  These columns should be prepended to the rows that
    +   *                        are produced by the iterator.
    +   * @param dataSchema The schema of the data that should be output for each row.  This may be a
    +   *                   subset of the columns that are present in the file if  column pruning has
    +   *                   occurred.
    +   * @param filters A set of filters than can optionally be used to reduce the number of rows output
    +   * @param options A set of string -> string configuration options.
    +   * @return
    +   */
    +  def buildReader(
    +      sqlContext: SQLContext,
    +      partitionSchema: StructType,
    +      dataSchema: StructType,
    +      filters: Seq[Filter],
    +      options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = {
    --- End diff --
    
    The same way as before (relying on erasure).  Before we make this public we should decide if we want to make all external sources return batches or have two interfaces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56595063
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala ---
    @@ -0,0 +1,57 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import org.apache.spark.{Partition, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.sql.SQLContext
    +import org.apache.spark.sql.catalyst.InternalRow
    +
    +/**
    + * A single file that should be read, along with partition column values that
    + * need to be prepended to each row.  The reading should start at the first
    + * valid record found after `offset`.
    --- End diff --
    
    offset -> start


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196541666
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56108277
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    --- End diff --
    
    Currently only the test format, but in a follow up PR we'll add Parquet and eventually remove this check and the old code path.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196625543
  
    Sorry for the late review, LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196545237
  
    Don't forget to create a JIRA for this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195547936
  
    **[Test build #52946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52946/consoleFull)** for PR 11646 at commit [`35be8d5`](https://github.com/apache/spark/commit/35be8d5400f7be28e9c9d5783cbe3c5adfba6965).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195217172
  
    **[Test build #52899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52899/consoleFull)** for PR 11646 at commit [`4f29845`](https://github.com/apache/spark/commit/4f298458fee9e0ebe59b4b569921bc32d3e1d4b1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `case class PartitionedFile(`
      * `case class FilePartition(val index: Int, files: Seq[PartitionedFile]) extends Partition`
      * `class FileScanRDD(`
      * `case class Partition(values: InternalRow, files: Seq[FileStatus])`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195591798
  
    **[Test build #52946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52946/consoleFull)** for PR 11646 at commit [`35be8d5`](https://github.com/apache/spark/commit/35be8d5400f7be28e9c9d5783cbe3c5adfba6965).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195181831
  
    /cc @cloud-fan @nongli @davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195222470
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52900/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56082915
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{PhysicalRDD, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    +                val remaining = file.getLen - offset
    +                val size = if (remaining > maxSplitBytes) maxSplitBytes else remaining
    +                PartitionedFile(partition.values, file.getPath.toUri.toString, offset, size)
    +              }
    +            }
    +          }.toArray.sortBy(_.length)(implicitly[Ordering[Long]].reverse)
    +
    +          val partitions = new ArrayBuffer[FilePartition]
    +          val currentFiles = new ArrayBuffer[PartitionedFile]
    +          var currentSize = 0L
    +
    +          /** Add the given file to the current partition. */
    +          def addFile(file: PartitionedFile): Unit = {
    +            currentSize += file.length
    +            currentFiles.append(file)
    +          }
    +
    +          /** Close the current partition and move to the next. */
    +          def closePartition(): Unit = {
    +            if (currentFiles.nonEmpty) {
    +              val newPartition =
    +                FilePartition(
    +                  partitions.size,
    +                  currentFiles.toArray.toSeq) // Copy to a new Array.
    +              partitions.append(newPartition)
    +            }
    +            currentFiles.clear()
    +            currentSize = 0
    +          }
    +
    +          // Assign files to partitions using "First Fit Decreasing" (FFD)
    +          // TODO: consider adding a slop factor here?
    +          splitFiles.foreach { file =>
    +            if (currentSize + file.length > maxSplitBytes) {
    +              closePartition()
    +              addFile(file)
    +            } else {
    +              addFile(file)
    +            }
    +          }
    +          closePartition()
    +          partitions
    +      }
    +
    +      val scan =
    +        PhysicalRDD(
    +          l.output,
    +          new FileScanRDD(
    --- End diff --
    
    Should we move this part (creating FileScanRDD) into DataSourceScan? We could have a new `FileScan` for this rule.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195217358
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52899/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56083071
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
    @@ -454,19 +454,60 @@ trait FileFormat {
           requiredColumns: Array[String],
           filters: Array[Filter],
           bucketSet: Option[BitSet],
    -      inputFiles: Array[FileStatus],
    +      inputFiles: Seq[FileStatus],
           broadcastedConf: Broadcast[SerializableConfiguration],
           options: Map[String, String]): RDD[InternalRow]
    +
    +  /**
    +   * Returns a function that can be used to read a single file in as an Iterator of InternalRow.
    +   *
    +   * @param partitionSchema The schema of the partition column row that will be present in each
    +   *                        PartitionedFile.  These columns should be prepended to the rows that
    +   *                        are produced by the iterator.
    +   * @param dataSchema The schema of the data that should be output for each row.  This may be a
    +   *                   subset of the columns that are present in the file if  column pruning has
    +   *                   occurred.
    +   * @param filters A set of filters than can optionally be used to reduce the number of rows output
    +   * @param options A set of string -> string configuration options.
    +   * @return
    +   */
    +  def buildReader(
    +      sqlContext: SQLContext,
    +      partitionSchema: StructType,
    +      dataSchema: StructType,
    +      filters: Seq[Filter],
    +      options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = {
    --- End diff --
    
    How can this API support row batch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r55862748
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala ---
    @@ -0,0 +1,56 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import org.apache.spark.{Partition, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.sql.SQLContext
    +import org.apache.spark.sql.catalyst.InternalRow
    +
    +/**
    + * A single file that should be read, along with partition column values that
    + * need to be prepended to each row.  The reading should start at the first
    + * valid record found after `offset`.
    + */
    +case class PartitionedFile(
    +    partitionValues: InternalRow,
    +    filePath: String,
    +    start: Long,
    +    length: Long)
    --- End diff --
    
    Should this has something about locations?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195181900
  
    **[Test build #52899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52899/consoleFull)** for PR 11646 at commit [`4f29845`](https://github.com/apache/spark/commit/4f298458fee9e0ebe59b4b569921bc32d3e1d4b1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195592211
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52946/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195592207
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57844323
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    --- End diff --
    
    What if there is no `Project` or `Filter` above the relation? We should read all columns but here we treat it as no column is required.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r55873402
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
    @@ -454,19 +454,51 @@ trait FileFormat {
           requiredColumns: Array[String],
           filters: Array[Filter],
           bucketSet: Option[BitSet],
    -      inputFiles: Array[FileStatus],
    +      inputFiles: Seq[FileStatus],
           broadcastedConf: Broadcast[SerializableConfiguration],
           options: Map[String, String]): RDD[InternalRow]
    +
    +  /**
    +   * Returns a function that can be used to read a single file in as an Iterator of InternalRow.
    +   *
    +   * @param partitionSchema The schema of the partition column row that will be present in each
    +   *                        PartitionedFile.  These columns should be prepended to the rows that
    +   *                        are produced by the iterator.
    +   * @param dataSchema The schema of the data that should be output for each row.  This may be a
    +   *                   subset of the columns that are present in the file if  column pruning has
    +   *                   occurred.
    +   * @param filters A set of filters than can optionally be used to reduce the number of rows output
    +   * @param options A set of string -> string configuration options.
    +   * @return
    +   */
    +  def buildReader(
    +      sqlContext: SQLContext,
    +      partitionSchema: StructType,
    +      dataSchema: StructType,
    +      filters: Seq[Filter],
    +      options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = {
    +    // TODO: Remove this default implementation when the other formats have been ported
    +    // Until then we guard in [[FileSourceStrategy]] to only call this method on supported formats.
    +    throw new UnsupportedOperationException(s"buildReader is not supported for $this")
    +  }
     }
     
     /**
    + * A collection of data files from a partitioned relation, along with the partition values in the
    + * form of an [[InternalRow]].
    + */
    +case class Partition(values: InternalRow, files: Seq[FileStatus])
    +
    +/**
      * An interface for objects capable of enumerating the files that comprise a relation as well
      * as the partitioning characteristics of those files.
      */
     trait FileCatalog {
       def paths: Seq[Path]
     
    -  def partitionSpec(schema: Option[StructType]): PartitionSpec
    +  def partitionSpec(): PartitionSpec
    +
    +  def listFiles(filters: Seq[Expression]): Seq[Partition]
    --- End diff --
    
    I'll add some comments about the `filters`.  I called it `listFiles` because it is actually enumerating all of the files for each partition internally (unlike `prunePartitions` which is only selecting the partition directories).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195217356
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195183208
  
    **[Test build #52900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52900/consoleFull)** for PR 11646 at commit [`7ad3119`](https://github.com/apache/spark/commit/7ad311903bc2a71c52132757adc4e28a2b83eafe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r55795906
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
    @@ -454,19 +454,51 @@ trait FileFormat {
           requiredColumns: Array[String],
           filters: Array[Filter],
           bucketSet: Option[BitSet],
    -      inputFiles: Array[FileStatus],
    +      inputFiles: Seq[FileStatus],
           broadcastedConf: Broadcast[SerializableConfiguration],
           options: Map[String, String]): RDD[InternalRow]
    +
    +  /**
    +   * Returns a function that can be used to read a single file in as an Iterator of InternalRow.
    +   *
    +   * @param partitionSchema The schema of the partition column row that will be present in each
    +   *                        PartitionedFile.  These columns should be prepended to the rows that
    +   *                        are produced by the iterator.
    +   * @param dataSchema The schema of the data that should be output for each row.  This may be a
    +   *                   subset of the columns that are present in the file if  column pruning has
    +   *                   occurred.
    +   * @param filters A set of filters than can optionally be used to reduce the number of rows output
    +   * @param options A set of string -> string configuration options.
    +   * @return
    +   */
    +  def buildReader(
    +      sqlContext: SQLContext,
    +      partitionSchema: StructType,
    +      dataSchema: StructType,
    +      filters: Seq[Filter],
    +      options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = {
    +    // TODO: Remove this default implementation when the other formats have been ported
    +    // Until then we guard in [[FileSourceStrategy]] to only call this method on supported formats.
    +    throw new UnsupportedOperationException(s"buildReader is not supported for $this")
    +  }
     }
     
     /**
    + * A collection of data files from a partitioned relation, along with the partition values in the
    + * form of an [[InternalRow]].
    + */
    +case class Partition(values: InternalRow, files: Seq[FileStatus])
    +
    +/**
      * An interface for objects capable of enumerating the files that comprise a relation as well
      * as the partitioning characteristics of those files.
      */
     trait FileCatalog {
       def paths: Seq[Path]
     
    -  def partitionSpec(schema: Option[StructType]): PartitionSpec
    +  def partitionSpec(): PartitionSpec
    +
    +  def listFiles(filters: Seq[Expression]): Seq[Partition]
    --- End diff --
    
    The name looks confusing, it's called `listFiles` but return a `Seq[Partition]`. Maybe we should call it `listPartitions`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196575075
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57454374
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    --- End diff --
    
    False alarm... See [here][1].
    
    [1]: https://github.com/apache/spark/pull/11936#issuecomment-201332718


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196542181
  
    **[Test build #53123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53123/consoleFull)** for PR 11646 at commit [`4744b97`](https://github.com/apache/spark/commit/4744b9736adaed89a2c47199338626bf57cd56f8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57445207
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    --- End diff --
    
    We found that at least `ParquetRecordReader` and `LineRecordReader` are designed to support splits that are cut at arbitrary positions. But is this a mandatory conversion that applies to all Hadoop record readers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r55881083
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
    @@ -454,19 +454,60 @@ trait FileFormat {
           requiredColumns: Array[String],
           filters: Array[Filter],
           bucketSet: Option[BitSet],
    -      inputFiles: Array[FileStatus],
    +      inputFiles: Seq[FileStatus],
           broadcastedConf: Broadcast[SerializableConfiguration],
           options: Map[String, String]): RDD[InternalRow]
    +
    +  /**
    +   * Returns a function that can be used to read a single file in as an Iterator of InternalRow.
    +   *
    +   * @param partitionSchema The schema of the partition column row that will be present in each
    +   *                        PartitionedFile.  These columns should be prepended to the rows that
    +   *                        are produced by the iterator.
    +   * @param dataSchema The schema of the data that should be output for each row.  This may be a
    +   *                   subset of the columns that are present in the file if  column pruning has
    +   *                   occurred.
    +   * @param filters A set of filters than can optionally be used to reduce the number of rows output
    +   * @param options A set of string -> string configuration options.
    +   * @return
    +   */
    +  def buildReader(
    +      sqlContext: SQLContext,
    +      partitionSchema: StructType,
    +      dataSchema: StructType,
    +      filters: Seq[Filter],
    +      options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = {
    --- End diff --
    
    @rxin @nongli 
    
    Should we make this our own iterator (i.e. w/ `close()` w/o `hasNext())?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56107919
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    --- End diff --
    
    what does this match for? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-196575077
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53123/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11646#issuecomment-195530204
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52939/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-XXXX][SQL] Add a strategy for planning ...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r56084083
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{PhysicalRDD, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    +                val remaining = file.getLen - offset
    +                val size = if (remaining > maxSplitBytes) maxSplitBytes else remaining
    +                PartitionedFile(partition.values, file.getPath.toUri.toString, offset, size)
    +              }
    +            }
    +          }.toArray.sortBy(_.length)(implicitly[Ordering[Long]].reverse)
    +
    +          val partitions = new ArrayBuffer[FilePartition]
    +          val currentFiles = new ArrayBuffer[PartitionedFile]
    +          var currentSize = 0L
    +
    +          /** Add the given file to the current partition. */
    +          def addFile(file: PartitionedFile): Unit = {
    +            currentSize += file.length
    +            currentFiles.append(file)
    +          }
    +
    +          /** Close the current partition and move to the next. */
    +          def closePartition(): Unit = {
    +            if (currentFiles.nonEmpty) {
    +              val newPartition =
    +                FilePartition(
    +                  partitions.size,
    +                  currentFiles.toArray.toSeq) // Copy to a new Array.
    +              partitions.append(newPartition)
    +            }
    +            currentFiles.clear()
    +            currentSize = 0
    +          }
    +
    +          // Assign files to partitions using "First Fit Decreasing" (FFD)
    +          // TODO: consider adding a slop factor here?
    +          splitFiles.foreach { file =>
    +            if (currentSize + file.length > maxSplitBytes) {
    +              closePartition()
    +              addFile(file)
    +            } else {
    +              addFile(file)
    +            }
    +          }
    +          closePartition()
    +          partitions
    +      }
    +
    +      val scan =
    +        PhysicalRDD(
    +          l.output,
    +          new FileScanRDD(
    --- End diff --
    
    Good idea, that would be cleaner.  We can't do that until we get rid of the old code path though (since there are two different types are RDDs we want to create).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13664][SQL] Add a strategy for planning...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11646#discussion_r57446993
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ---
    @@ -0,0 +1,202 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.datasources
    +
    +import scala.collection.mutable.ArrayBuffer
    +
    +import org.apache.hadoop.fs.Path
    +
    +import org.apache.spark.Logging
    +import org.apache.spark.sql._
    +import org.apache.spark.sql.catalyst.expressions
    +import org.apache.spark.sql.catalyst.expressions._
    +import org.apache.spark.sql.catalyst.planning.PhysicalOperation
    +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.execution.{DataSourceScan, SparkPlan}
    +import org.apache.spark.sql.sources._
    +import org.apache.spark.sql.types._
    +
    +/**
    + * A strategy for planning scans over collections of files that might be partitioned or bucketed
    + * by user specified columns.
    + *
    + * At a high level planning occurs in several phases:
    + *  - Split filters by when they need to be evaluated.
    + *  - Prune the schema of the data requested based on any projections present. Today this pruning
    + *    is only done on top level columns, but formats should support pruning of nested columns as
    + *    well.
    + *  - Construct a reader function by passing filters and the schema into the FileFormat.
    + *  - Using an partition pruning predicates, enumerate the list of files that should be read.
    + *  - Split the files into tasks and construct a FileScanRDD.
    + *  - Add any projection or filters that must be evaluated after the scan.
    + *
    + * Files are assigned into tasks using the following algorithm:
    + *  - If the table is bucketed, group files by bucket id into the correct number of partitions.
    + *  - If the table is not bucketed or bucketing is turned off:
    + *   - If any file is larger than the threshold, split it into pieces based on that threshold
    + *   - Sort the files by decreasing file size.
    + *   - Assign the ordered files to buckets using the following algorithm.  If the current partition
    + *     is under the threshold with the addition of the next file, add it.  If not, open a new bucket
    + *     and add it.  Proceed to the next file.
    + */
    +private[sql] object FileSourceStrategy extends Strategy with Logging {
    +  def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
    +    case PhysicalOperation(projects, filters, l@LogicalRelation(files: HadoopFsRelation, _, _))
    +      if files.fileFormat.toString == "TestFileFormat" =>
    +      // Filters on this relation fall into four categories based on where we can use them to avoid
    +      // reading unneeded data:
    +      //  - partition keys only - used to prune directories to read
    +      //  - bucket keys only - optionally used to prune files to read
    +      //  - keys stored in the data only - optionally used to skip groups of data in files
    +      //  - filters that need to be evaluated again after the scan
    +      val filterSet = ExpressionSet(filters)
    +
    +      val partitionColumns =
    +        AttributeSet(l.resolve(files.partitionSchema, files.sqlContext.analyzer.resolver))
    +      val partitionKeyFilters =
    +        ExpressionSet(filters.filter(_.references.subsetOf(partitionColumns)))
    +      logInfo(s"Pruning directories with: ${partitionKeyFilters.mkString(",")}")
    +
    +      val bucketColumns =
    +        AttributeSet(
    +          files.bucketSpec
    +              .map(_.bucketColumnNames)
    +              .getOrElse(Nil)
    +              .map(l.resolveQuoted(_, files.sqlContext.conf.resolver)
    +                  .getOrElse(sys.error(""))))
    +
    +      // Partition keys are not available in the statistics of the files.
    +      val dataFilters = filters.filter(_.references.intersect(partitionColumns).isEmpty)
    +
    +      // Predicates with both partition keys and attributes need to be evaluated after the scan.
    +      val afterScanFilters = filterSet -- partitionKeyFilters
    +      logInfo(s"Post-Scan Filters: ${afterScanFilters.mkString(",")}")
    +
    +      val selectedPartitions = files.location.listFiles(partitionKeyFilters.toSeq)
    +
    +      val filterAttributes = AttributeSet(afterScanFilters)
    +      val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects
    +      val requiredAttributes = AttributeSet(requiredExpressions).map(_.name).toSet
    +
    +      val prunedDataSchema =
    +        StructType(
    +          files.dataSchema.filter(f => requiredAttributes.contains(f.name)))
    +      logInfo(s"Pruned Data Schema: ${prunedDataSchema.simpleString(5)}")
    +
    +      val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.translateFilter)
    +      logInfo(s"Pushed Filters: ${pushedDownFilters.mkString(",")}")
    +
    +      val readFile = files.fileFormat.buildReader(
    +        sqlContext = files.sqlContext,
    +        partitionSchema = files.partitionSchema,
    +        dataSchema = prunedDataSchema,
    +        filters = pushedDownFilters,
    +        options = files.options)
    +
    +      val plannedPartitions = files.bucketSpec match {
    +        case Some(bucketing) if files.sqlContext.conf.bucketingEnabled =>
    +          logInfo(s"Planning with ${bucketing.numBuckets} buckets")
    +          val bucketed =
    +            selectedPartitions
    +                .flatMap { p =>
    +                  p.files.map(f => PartitionedFile(p.values, f.getPath.toUri.toString, 0, f.getLen))
    +                }.groupBy { f =>
    +              BucketingUtils
    +                  .getBucketId(new Path(f.filePath).getName)
    +                  .getOrElse(sys.error(s"Invalid bucket file ${f.filePath}"))
    +            }
    +
    +          (0 until bucketing.numBuckets).map { bucketId =>
    +            FilePartition(bucketId, bucketed.getOrElse(bucketId, Nil))
    +          }
    +
    +        case _ =>
    +          val maxSplitBytes = files.sqlContext.conf.filesMaxPartitionBytes
    +          logInfo(s"Planning scan with bin packing, max size: $maxSplitBytes bytes")
    +
    +          val splitFiles = selectedPartitions.flatMap { partition =>
    +            partition.files.flatMap { file =>
    +              assert(file.getLen != 0)
    +              (0L to file.getLen by maxSplitBytes).map { offset =>
    --- End diff --
    
    Aww... Unfortunately ORC doesn't obey this convention, and it breaks ORC version of `buildReader()` implemented in PR #11936. Seems that we have to have `FileFormat` to tell how to generate file splits.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org