You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liancheng <gi...@git.apache.org> on 2015/04/15 15:53:04 UTC
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/5526
[SQL] [WIP] Partitioning support for the data sources API
**NOTE:** Currently this is only a tentative PR discussing API design.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark partitioning-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5526.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5526
----
commit 9b58be607a23c1456287365c2e91d7365f259dca
Author: Cheng Lian <li...@databricks.com>
Date: 2015-04-15T13:49:43Z
Adds new interfaces and stub methods for data sources API partitioning support
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93448767
[Test build #30342 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30342/consoleFull) for PR 5526 at commit [`9b58be6`](https://github.com/apache/spark/commit/9b58be607a23c1456287365c2e91d7365f259dca).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait PartitionedSchemaRelationProvider `
* `trait OutputWriter `
* `trait FSBasedPrunedFilteredScan extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97545857
[Test build #31294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31294/consoleFull) for PR 5526 at commit [`afc7f1f`](https://github.com/apache/spark/commit/afc7f1fd935347e0f576820ffa5a23acde6ab150).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29354273
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
+ import caseInsensitiveContext._
+ import caseInsensitiveContext.implicits._
+
+ var basePath: Path = _
+
+ var fs: FileSystem = _
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("b", StringType, nullable = false)))
+
+ val testDF = (for {
+ i <- 1 to 3
+ p <- 1 to 2
+ } yield (i, s"val_$i", p)).toDF("a", "b", "p1")
+
+ before {
+ basePath = new Path(Utils.createTempDir().getCanonicalPath)
+ fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
+ TestResult.reset()
+ }
+
+ ignore("load() - partitioned table - partition column not included in data files") {
+ fs.mkdirs(new Path(basePath, "p1=1/p2=hello"))
+ fs.mkdirs(new Path(basePath, "p1=2/p2=world"))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("load() - partitioned table - partition column included in data files") {
+ val data = sparkContext.parallelize(Seq.empty[String])
+ data.saveAsTextFile(new Path(basePath, "p1=1/p2=hello").toString)
+ data.saveAsTextFile(new Path(basePath, "p1=2/p2=world").toString)
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false)))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("save() - partitioned table - Overwrite") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Append") {
--- End diff --
Thank you very much for the extra test cases, they did help me catching more bugs :)
As for the last issue, currently all `FSBasedRelation` instances do partition discovery while being instantiated. Later we can add `PARTITIONED BY` clause to the data sources DDL, and probably a corresponding overloaded version of `SQLContext.load()` with user defined partition columns schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93477429
**[Test build #30344 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30344/consoleFull)** for PR 5526 at commit [`f973010`](https://github.com/apache/spark/commit/f973010cede214d9ac9550847c10bf324c40ef31) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97994977
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97873223
[Test build #31430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31430/consoleFull) for PR 5526 at commit [`1eb0414`](https://github.com/apache/spark/commit/1eb0414490cde6f0752dd8923b709fdcec3c7f52).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29477039
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala ---
@@ -63,6 +101,122 @@ private[sql] object DataSourceStrategy extends Strategy {
case _ => Nil
}
+ private def buildPartitionedTableScan(
+ logicalRelation: LogicalRelation,
+ projections: Seq[NamedExpression],
+ filters: Seq[Expression],
+ partitionColumns: StructType,
+ partitions: Array[Partition]) = {
+ val output = projections.map(_.toAttribute)
+ val relation = logicalRelation.relation.asInstanceOf[FSBasedRelation]
+ val dataSchema = relation.dataSchema
+
+ // Builds RDD[Row]s for each selected partition.
+ val perPartitionRows = partitions.map { case Partition(partitionValues, dir) =>
+ // Paths to all data files within this partition
+ val dataFilePaths = {
+ val dirPath = new Path(dir)
+ val fs = dirPath.getFileSystem(SparkHadoopUtil.get.conf)
+ fs.listStatus(dirPath)
+ .map(_.getPath)
+ .filter { path =>
+ val name = path.getName
+ name.startsWith("_") || name.startsWith(".")
+ }
+ .map(fs.makeQualified(_).toString)
+ }
+
+ // The table scan operator (PhysicalRDD) which retrieves required columns from data files.
+ // Notice that the schema of data files, represented by `relation.dataSchema`, may contain
+ // some partition column(s). Those partition columns that are only encoded in partition
+ // directory paths are not covered by this table scan operator.
+ val scan =
+ pruneFilterProject(
+ logicalRelation,
+ projections,
+ filters,
+ (requiredColumns, filters) => {
+ // Only columns appear in actual data, which possibly include some partition column(s)
+ relation.buildScan(
+ requiredColumns.filter(dataSchema.fieldNames.contains),
--- End diff --
No logic error, but may be optimizable. If a column appears in both data file and partition, here the value will be retrieved from file. How do you think if we make assumption that the partition value is consistent with the column value in file so that such reading can be skipped and later on it is filled by mergePartitionValues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95636852
@marmbrus @yhuai @rxin Previous comments are addressed, Also added tests (`ignore`d for now) in `FSBasedRelationSuite`. Going to implement all the interface.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97514780
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31290/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95674413
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30851/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28977394
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
--- End diff --
This issue has been fixed. Three variants are provided. Developers can override any one of them to implement the read path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97952767
[Test build #31437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31437/consoleFull) for PR 5526 at commit [`e98fec1`](https://github.com/apache/spark/commit/e98fec11cdb3e0f8d8230616065b0d12851ba0f7).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch **adds the following new dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
* This patch **removes the following dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97907407
[Test build #31439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31439/consoleFull) for PR 5526 at commit [`c43cca4`](https://github.com/apache/spark/commit/c43cca4af0b2e143ad0e98cbcaf5162faac4304a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97522480
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31295/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28452924
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
--- End diff --
this should be an abstract class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97504049
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97522442
[Test build #31295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31295/consoleFull) for PR 5526 at commit [`9a6a7f5`](https://github.com/apache/spark/commit/9a6a7f5b7cc1d6fcb8cffdaf9641e96acd3ad5cc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-96026476
@liancheng can you fill in the JIRA number here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28455676
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
--- End diff --
We were going to provide 3 variants of buildScan in this method, weren't we?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97906831
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97522474
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97517766
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95614524
[Test build #30846 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30846/consoleFull) for PR 5526 at commit [`4e93e9b`](https://github.com/apache/spark/commit/4e93e9b04790b24d0df57d8c62c9447abea0f74f).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* `abstract class FSBasedRelation extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97505417
[Test build #31295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31295/consoleFull) for PR 5526 at commit [`9a6a7f5`](https://github.com/apache/spark/commit/9a6a7f5b7cc1d6fcb8cffdaf9641e96acd3ad5cc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95640536
[Test build #30850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30850/consoleFull) for PR 5526 at commit [`b63f813`](https://github.com/apache/spark/commit/b63f81375e3e3cdea884c2e7c3f1925294c61c21).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* `abstract class FSBasedRelation extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97458994
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29077429
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
+ import caseInsensitiveContext._
+ import caseInsensitiveContext.implicits._
+
+ var basePath: Path = _
+
+ var fs: FileSystem = _
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("b", StringType, nullable = false)))
+
+ val testDF = (for {
+ i <- 1 to 3
+ p <- 1 to 2
+ } yield (i, s"val_$i", p)).toDF("a", "b", "p1")
+
+ before {
+ basePath = new Path(Utils.createTempDir().getCanonicalPath)
+ fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
+ TestResult.reset()
+ }
+
+ ignore("load() - partitioned table - partition column not included in data files") {
+ fs.mkdirs(new Path(basePath, "p1=1/p2=hello"))
+ fs.mkdirs(new Path(basePath, "p1=2/p2=world"))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("load() - partitioned table - partition column included in data files") {
+ val data = sparkContext.parallelize(Seq.empty[String])
+ data.saveAsTextFile(new Path(basePath, "p1=1/p2=hello").toString)
+ data.saveAsTextFile(new Path(basePath, "p1=2/p2=world").toString)
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false)))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("save() - partitioned table - Overwrite") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Append") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Append,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 4) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - ErrorIfExists") {
+ fs.delete(basePath, true)
+
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ assert(TestResult.writtenRows.sameElements(testDF.collect()))
+
+ intercept[IOException] {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+ }
+ }
+
+ ignore("save() - partitioned table - Ignore") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ assert(TestResult.writtenRows.isEmpty)
+ }
+
+ ignore("saveAsTable() - partitioned table - Overwrite") {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - Append") {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Append,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 4) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - ErrorIfExists") {
+ fs.delete(basePath, true)
+
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ assert(TestResult.writtenRows.sameElements(testDF.collect()))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ intercept[IOException] {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - Ignore") {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
--- End diff --
Ignore?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29015695
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
+ import caseInsensitiveContext._
+ import caseInsensitiveContext.implicits._
+
+ var basePath: Path = _
+
+ var fs: FileSystem = _
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("b", StringType, nullable = false)))
+
+ val testDF = (for {
+ i <- 1 to 3
+ p <- 1 to 2
+ } yield (i, s"val_$i", p)).toDF("a", "b", "p1")
+
+ before {
+ basePath = new Path(Utils.createTempDir().getCanonicalPath)
+ fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
+ TestResult.reset()
+ }
+
+ ignore("load() - partitioned table - partition column not included in data files") {
+ fs.mkdirs(new Path(basePath, "p1=1/p2=hello"))
+ fs.mkdirs(new Path(basePath, "p1=2/p2=world"))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("load() - partitioned table - partition column included in data files") {
+ val data = sparkContext.parallelize(Seq.empty[String])
+ data.saveAsTextFile(new Path(basePath, "p1=1/p2=hello").toString)
+ data.saveAsTextFile(new Path(basePath, "p1=2/p2=world").toString)
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false)))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("save() - partitioned table - Overwrite") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Append") {
--- End diff --
Seems if we want to append and data exist, we need to discover the partitioning structures and make sure we can safely write data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97956619
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28425682
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row]
+
+ /**
+ * When writing rows to this relation, this method is invoked on the driver side before the actual
+ * write job is issued. It provides an opportunity to configure the write job to be performed.
+ */
+ def prepareForWrite(conf: Configuration): Unit
--- End diff --
@rxin I decided to keep this method. For data sources like Hive, Parquet, and ORC, although the driver side preparation work done before issuing the write job can be somehow converted to executor side, we still need another similar hook to do the work. Constructor of `OutputWriter` instances is not a proper place to do the executor side preparation. Because in case of dynamic partitioning, a single task may create multiple `OutputWriter`, while the preparation should be done only once. Another reason is that, for traditional Hadoop users, driver side setup code can be pretty conventional and familiar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28455069
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
--- End diff --
what's in this row -- does it include the partitioning column(s)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28482809
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
--- End diff --
Since we only want to support dynamic partitioning in this version, all partition column(s) must be included.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97956621
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31439/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28424139
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -78,6 +80,40 @@ trait SchemaRelationProvider {
schema: StructType): BaseRelation
}
+/**
+ * ::DeveloperApi::
+ * Implemented by objects that produce relations for a specific kind of data source
+ * with a given schema and partitioned columns. When Spark SQL is given a DDL operation with a
+ * USING clause specified (to specify the implemented SchemaRelationProvider), a user defined
+ * schema, and an optional list of partition columns, this interface is used to pass in the
+ * parameters specified by a user.
+ *
+ * Users may specify the fully qualified class name of a given data source. When that class is
+ * not found Spark SQL will append the class name `DefaultSource` to the path, allowing for
+ * less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the
+ * data source 'org.apache.spark.sql.json.DefaultSource'
+ *
+ * A new instance of this class with be instantiated each time a DDL call is made.
+ *
+ * The difference between a [[RelationProvider]] and a [[PartitionedSchemaRelationProvider]] is
+ * that users need to provide a schema and a (possibly empty) list of partition columns when
+ * using a SchemaRelationProvider. A relation provider can inherits both [[RelationProvider]],
+ * and [[PartitionedSchemaRelationProvider]] if it can support schema inference, user-specified
+ * schemas, and accessing partitioned relations.
+ */
+trait PartitionedSchemaRelationProvider {
+ /**
+ * Returns a new base relation with the given parameters, a user defined schema, and a list of
+ * partition columns. Note: the parameters' keywords are case insensitive and this insensitivity
+ * is enforced by the Map that is passed to the function.
+ */
+ def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String],
+ schema: StructType,
+ partitionColumns: StructType): BaseRelation
--- End diff --
Just realized that, to be truly Java API friendly, we also need to add another overridden `createRelation` method where `parameters` is passed in as a `java.util.Map[String, String]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97994994
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97902209
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93424047
[Test build #30347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30347/consoleFull) for PR 5526 at commit [`e209ab0`](https://github.com/apache/spark/commit/e209ab0f6641bc0a68af12621cf9733f60085f42).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95674382
[Test build #30851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30851/consoleFull) for PR 5526 at commit [`3b22c32`](https://github.com/apache/spark/commit/3b22c3291cd920eb02a345dace2556dc23d57efb).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* `abstract class FSBasedRelation extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28975672
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row]
+
+ /**
+ * When writing rows to this relation, this method is invoked on the driver side before the actual
+ * write job is issued. It provides an opportunity to configure the write job to be performed.
+ */
+ def prepareForWrite(conf: Configuration): Unit
+
+ /**
+ * This method is responsible for producing a new [[OutputWriter]] for each newly opened output
+ * file on the executor side.
+ */
+ def newOutputWriter(path: String): OutputWriter
--- End diff --
One issue here is about passing driver side Hadoop configuration to OutputWriters on executor side. Users may set properties to Hadoop configurations on driver side (e.g. `mapreduce.fileoutputcommitter.marksuccessfuljobs`), and we should inherit these settings on executor side when writing data. zero-arg constructor plus `init(...)` is a good way to avoid forcing `BaseRelation` to be serializable, but I guess we have to put `Configuration` as an argument of `OutputWriter.init(...)`. This makes the data sources API coupled with Hadoop API via `Configuration`, but I guess this should be more acceptable comparing to forcing `BaseRelation` subclasses to be serializable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29015520
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
+ import caseInsensitiveContext._
+ import caseInsensitiveContext.implicits._
+
+ var basePath: Path = _
+
+ var fs: FileSystem = _
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("b", StringType, nullable = false)))
+
+ val testDF = (for {
+ i <- 1 to 3
+ p <- 1 to 2
+ } yield (i, s"val_$i", p)).toDF("a", "b", "p1")
+
+ before {
+ basePath = new Path(Utils.createTempDir().getCanonicalPath)
+ fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
+ TestResult.reset()
+ }
+
+ ignore("load() - partitioned table - partition column not included in data files") {
+ fs.mkdirs(new Path(basePath, "p1=1/p2=hello"))
+ fs.mkdirs(new Path(basePath, "p1=2/p2=world"))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("load() - partitioned table - partition column included in data files") {
+ val data = sparkContext.parallelize(Seq.empty[String])
+ data.saveAsTextFile(new Path(basePath, "p1=1/p2=hello").toString)
+ data.saveAsTextFile(new Path(basePath, "p1=2/p2=world").toString)
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false)))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("save() - partitioned table - Overwrite") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Append") {
--- End diff --
Also, when we append to existing dataset, what will happen if the partitioning columns provided in `save` does not match those encoded by the dir structure. For example, we say `partitionColumns = Seq("p1")`, but the existing dir struct is like `/p1=.../p2=...`.
Also, what will happen if the ordering of partitioning columns does not match? For example, we say `partitionColumns = Seq("p1", "p2")`, but the existing dir struct is like `/p2=.../p1=...`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28482788
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -78,6 +80,40 @@ trait SchemaRelationProvider {
schema: StructType): BaseRelation
}
+/**
+ * ::DeveloperApi::
+ * Implemented by objects that produce relations for a specific kind of data source
+ * with a given schema and partitioned columns. When Spark SQL is given a DDL operation with a
+ * USING clause specified (to specify the implemented SchemaRelationProvider), a user defined
+ * schema, and an optional list of partition columns, this interface is used to pass in the
+ * parameters specified by a user.
+ *
+ * Users may specify the fully qualified class name of a given data source. When that class is
+ * not found Spark SQL will append the class name `DefaultSource` to the path, allowing for
+ * less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the
+ * data source 'org.apache.spark.sql.json.DefaultSource'
+ *
+ * A new instance of this class with be instantiated each time a DDL call is made.
+ *
+ * The difference between a [[RelationProvider]] and a [[PartitionedSchemaRelationProvider]] is
+ * that users need to provide a schema and a (possibly empty) list of partition columns when
+ * using a SchemaRelationProvider. A relation provider can inherits both [[RelationProvider]],
+ * and [[PartitionedSchemaRelationProvider]] if it can support schema inference, user-specified
+ * schemas, and accessing partitioned relations.
+ */
+trait PartitionedSchemaRelationProvider {
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97873234
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28452935
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
--- End diff --
this should be an abstract class also
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29077578
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
--- End diff --
This looks pretty good. I would also add a test case that involves a self join (as those are quite error prone). I'm not sure how hard this would be, but it would also be awesome if we could use this harness (or something similar) on concrete implementations like json and parquet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29479873
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala ---
@@ -63,6 +101,122 @@ private[sql] object DataSourceStrategy extends Strategy {
case _ => Nil
}
+ private def buildPartitionedTableScan(
+ logicalRelation: LogicalRelation,
+ projections: Seq[NamedExpression],
+ filters: Seq[Expression],
+ partitionColumns: StructType,
+ partitions: Array[Partition]) = {
+ val output = projections.map(_.toAttribute)
+ val relation = logicalRelation.relation.asInstanceOf[FSBasedRelation]
+ val dataSchema = relation.dataSchema
+
+ // Builds RDD[Row]s for each selected partition.
+ val perPartitionRows = partitions.map { case Partition(partitionValues, dir) =>
+ // Paths to all data files within this partition
+ val dataFilePaths = {
+ val dirPath = new Path(dir)
+ val fs = dirPath.getFileSystem(SparkHadoopUtil.get.conf)
+ fs.listStatus(dirPath)
+ .map(_.getPath)
+ .filter { path =>
+ val name = path.getName
+ name.startsWith("_") || name.startsWith(".")
+ }
+ .map(fs.makeQualified(_).toString)
+ }
+
+ // The table scan operator (PhysicalRDD) which retrieves required columns from data files.
+ // Notice that the schema of data files, represented by `relation.dataSchema`, may contain
+ // some partition column(s). Those partition columns that are only encoded in partition
+ // directory paths are not covered by this table scan operator.
+ val scan =
+ pruneFilterProject(
+ logicalRelation,
+ projections,
+ filters,
+ (requiredColumns, filters) => {
+ // Only columns appear in actual data, which possibly include some partition column(s)
+ relation.buildScan(
+ requiredColumns.filter(dataSchema.fieldNames.contains),
--- End diff --
Good catch! Simply changing the filter condition would be enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97458968
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97902775
[Test build #31437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31437/consoleFull) for PR 5526 at commit [`e98fec1`](https://github.com/apache/spark/commit/e98fec11cdb3e0f8d8230616065b0d12851ba0f7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29092105
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
+ import caseInsensitiveContext._
+ import caseInsensitiveContext.implicits._
+
+ var basePath: Path = _
+
+ var fs: FileSystem = _
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("b", StringType, nullable = false)))
+
+ val testDF = (for {
+ i <- 1 to 3
+ p <- 1 to 2
+ } yield (i, s"val_$i", p)).toDF("a", "b", "p1")
+
+ before {
+ basePath = new Path(Utils.createTempDir().getCanonicalPath)
+ fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
+ TestResult.reset()
+ }
+
+ ignore("load() - partitioned table - partition column not included in data files") {
+ fs.mkdirs(new Path(basePath, "p1=1/p2=hello"))
+ fs.mkdirs(new Path(basePath, "p1=2/p2=world"))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("load() - partitioned table - partition column included in data files") {
+ val data = sparkContext.parallelize(Seq.empty[String])
+ data.saveAsTextFile(new Path(basePath, "p1=1/p2=hello").toString)
+ data.saveAsTextFile(new Path(basePath, "p1=2/p2=world").toString)
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false)))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("save() - partitioned table - Overwrite") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Append") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Append,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 4) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - ErrorIfExists") {
+ fs.delete(basePath, true)
+
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ assert(TestResult.writtenRows.sameElements(testDF.collect()))
+
+ intercept[IOException] {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+ }
+ }
+
+ ignore("save() - partitioned table - Ignore") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ assert(TestResult.writtenRows.isEmpty)
+ }
+
+ ignore("saveAsTable() - partitioned table - Overwrite") {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - Append") {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Append,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 4) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - ErrorIfExists") {
+ fs.delete(basePath, true)
+
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ assert(TestResult.writtenRows.sameElements(testDF.collect()))
+
+ assertResult(table("t").schema) {
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true)))
+ }
+
+ intercept[IOException] {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+ }
+
+ sql("DROP TABLE t")
+ }
+
+ ignore("saveAsTable() - partitioned table - Ignore") {
+ testDF.saveAsTable(
+ tableName = "t",
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
--- End diff --
Oops, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93417227
[Test build #30344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30344/consoleFull) for PR 5526 at commit [`f973010`](https://github.com/apache/spark/commit/f973010cede214d9ac9550847c10bf324c40ef31).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97902257
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-98000643
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97956605
[Test build #31439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31439/consoleFull) for PR 5526 at commit [`c43cca4`](https://github.com/apache/spark/commit/c43cca4af0b2e143ad0e98cbcaf5162faac4304a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` "public class " + className + extendsText + " implements java.io.Serializable `
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97505100
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97470924
[Test build #31290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31290/consoleFull) for PR 5526 at commit [`03bf5eb`](https://github.com/apache/spark/commit/03bf5eb4b25ba4ac57bcd11f58b2f1a6f08deb8e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29015390
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
+ import caseInsensitiveContext._
+ import caseInsensitiveContext.implicits._
+
+ var basePath: Path = _
+
+ var fs: FileSystem = _
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("b", StringType, nullable = false)))
+
+ val testDF = (for {
+ i <- 1 to 3
+ p <- 1 to 2
+ } yield (i, s"val_$i", p)).toDF("a", "b", "p1")
+
+ before {
+ basePath = new Path(Utils.createTempDir().getCanonicalPath)
+ fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
+ TestResult.reset()
+ }
+
+ ignore("load() - partitioned table - partition column not included in data files") {
+ fs.mkdirs(new Path(basePath, "p1=1/p2=hello"))
+ fs.mkdirs(new Path(basePath, "p1=2/p2=world"))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ dataSchema ++ Seq(
+ StructField("p1", IntegerType, nullable = true),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("load() - partitioned table - partition column included in data files") {
+ val data = sparkContext.parallelize(Seq.empty[String])
+ data.saveAsTextFile(new Path(basePath, "p1=1/p2=hello").toString)
+ data.saveAsTextFile(new Path(basePath, "p1=2/p2=world").toString)
+
+ val dataSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false)))
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.queryExecution.analyzed.collect {
+ case LogicalRelation(relation: SimpleFSBasedRelation) =>
+ assert(relation.dataSchema === dataSchema)
+ case _ =>
+ fail("Couldn't find expected SimpleFSBasedRelation instance")
+ }
+
+ val expectedSchema =
+ StructType(
+ Seq(
+ StructField("a", IntegerType, nullable = false),
+ StructField("p1", IntegerType, nullable = true),
+ StructField("b", StringType, nullable = false),
+ StructField("p2", StringType, nullable = true)))
+
+ assert(df.schema === expectedSchema)
+
+ df.select("b").where($"a" > 0 && $"p1" === 1).collect()
+
+ // Check for column pruning, filter push-down, and partition pruning
+ assert(TestResult.requiredColumns.toSet === Set("a", "b"))
+ assert(TestResult.filters === Seq(GreaterThan("a", 0)))
+ assert(TestResult.inputPaths === Seq(new Path(basePath, "p1=1").toString))
+ }
+
+ ignore("save() - partitioned table - Overwrite") {
+ testDF.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Overwrite - select and overwrite the same table") {
+ TestResult.rowsToRead = testDF.rdd
+
+ val df = load(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json))
+
+ df.save(
+ source = classOf[SimpleFSBasedSource].getCanonicalName,
+ mode = SaveMode.Overwrite,
+ options = Map(
+ "path" -> basePath.toString,
+ "schema" -> dataSchema.json),
+ partitionColumns = Seq("p1"))
+
+ // Written rows shouldn't contain dynamic partition column
+ val expectedRows = for (i <- 1 to 3; _ <- 1 to 2) yield Row(i, s"val_$i")
+ assert(TestResult.writtenRows.sameElements(expectedRows))
+ }
+
+ ignore("save() - partitioned table - Append") {
--- End diff --
For append, we can have a little bit more tests. Here are two in my mind:
* having a df that only will be written to a subset of partitions;
* having a df that contains new partitioning value (we need to create new dirs).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93448783
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30342/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97475089
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97545884
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97872197
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-98000645
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31470/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97545886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31294/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28451657
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row]
+
+ /**
+ * When writing rows to this relation, this method is invoked on the driver side before the actual
+ * write job is issued. It provides an opportunity to configure the write job to be performed.
+ */
+ def prepareForWrite(conf: Configuration): Unit
+
+ /**
+ * This method is responsible for producing a new [[OutputWriter]] for each newly opened output
+ * file on the executor side.
+ */
+ def newOutputWriter(path: String): OutputWriter
--- End diff --
does this mean now BaseRelation needs to be serializable? I don't think this was the case before, was it?
I think it'd make more sense to have OutputWriter to have a zero-arg ctor, and the relation simply returns the class or the class name, and then we create the OutputWriter on each executor using reflection, and then call init(path: String, schema: StructType, options: Map[...]) on those.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93482097
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30349/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97872721
[Test build #31430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31430/consoleFull) for PR 5526 at commit [`1eb0414`](https://github.com/apache/spark/commit/1eb0414490cde6f0752dd8923b709fdcec3c7f52).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28455332
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -78,6 +80,40 @@ trait SchemaRelationProvider {
schema: StructType): BaseRelation
}
+/**
+ * ::DeveloperApi::
+ * Implemented by objects that produce relations for a specific kind of data source
+ * with a given schema and partitioned columns. When Spark SQL is given a DDL operation with a
+ * USING clause specified (to specify the implemented SchemaRelationProvider), a user defined
+ * schema, and an optional list of partition columns, this interface is used to pass in the
+ * parameters specified by a user.
+ *
+ * Users may specify the fully qualified class name of a given data source. When that class is
+ * not found Spark SQL will append the class name `DefaultSource` to the path, allowing for
+ * less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the
+ * data source 'org.apache.spark.sql.json.DefaultSource'
+ *
+ * A new instance of this class with be instantiated each time a DDL call is made.
+ *
+ * The difference between a [[RelationProvider]] and a [[PartitionedSchemaRelationProvider]] is
+ * that users need to provide a schema and a (possibly empty) list of partition columns when
+ * using a SchemaRelationProvider. A relation provider can inherits both [[RelationProvider]],
+ * and [[PartitionedSchemaRelationProvider]] if it can support schema inference, user-specified
+ * schemas, and accessing partitioned relations.
+ */
+trait PartitionedSchemaRelationProvider {
--- End diff --
shouldn't this be named FSBasedRelationProvider?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93480235
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30348/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97473639
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93482080
[Test build #30349 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30349/consoleFull) for PR 5526 at commit [`b46ee49`](https://github.com/apache/spark/commit/b46ee49c8e8915a745704713f1f15ceaf3961a79).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait PartitionedSchemaRelationProvider `
* `trait OutputWriter `
* `trait FSBasedRelation extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95614464
[Test build #30846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30846/consoleFull) for PR 5526 at commit [`4e93e9b`](https://github.com/apache/spark/commit/4e93e9b04790b24d0df57d8c62c9447abea0f74f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97906787
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97504183
[Test build #31294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31294/consoleFull) for PR 5526 at commit [`afc7f1f`](https://github.com/apache/spark/commit/afc7f1fd935347e0f576820ffa5a23acde6ab150).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97470524
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93427546
[Test build #30348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30348/consoleFull) for PR 5526 at commit [`d432248`](https://github.com/apache/spark/commit/d4322488c7c9b0f549b94fceb0f2bd2c6e86c43c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93411632
[Test build #30342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30342/consoleFull) for PR 5526 at commit [`9b58be6`](https://github.com/apache/spark/commit/9b58be607a23c1456287365c2e91d7365f259dca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97470500
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97517769
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31291/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28482882
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
--- End diff --
What do you mean? Supporting dynamic partitioning doesn't imply partition columns should be included (or not included).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97872223
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28482813
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
--- End diff --
OK, makes sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97459143
[Test build #31289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31289/consoleFull) for PR 5526 at commit [`43078dc`](https://github.com/apache/spark/commit/43078dc8d1c6a788916a3508bc747f47e5f4bb8e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95646404
[Test build #30851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30851/consoleFull) for PR 5526 at commit [`3b22c32`](https://github.com/apache/spark/commit/3b22c3291cd920eb02a345dace2556dc23d57efb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93424090
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30347/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93480206
[Test build #30348 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30348/consoleFull) for PR 5526 at commit [`d432248`](https://github.com/apache/spark/commit/d4322488c7c9b0f549b94fceb0f2bd2c6e86c43c).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait PartitionedSchemaRelationProvider `
* `trait OutputWriter `
* `trait FSBasedRelation extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97873235
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31430/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97473676
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97504014
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29402960
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala ---
@@ -53,6 +53,25 @@ private[sql] object DataSourceStrategy extends Strategy {
filters,
(a, _) => t.buildScan(a)) :: Nil
+ case PhysicalOperation(projectList, filters, l @ LogicalRelation(t: FSBasedRelation)) =>
+ val selectedPartitions = prunePartitions(filters, t.partitionSpec)
+ val inputPaths = selectedPartitions.map(_.path).toArray
+
+ // Don't push down predicates that reference partition columns
+ val pushedFilters = {
+ val partitionColumnNames = t.partitionSpec.partitionColumns.map(_.name).toSet
+ filters.filter { f =>
+ val referencedColumnNames = f.references.map(_.name).toSet
+ referencedColumnNames.intersect(partitionColumnNames).isEmpty
+ }
+ }
+
+ pruneFilterProject(
+ l,
+ projectList,
+ pushedFilters,
+ (a, f) => t.buildScan(a, f, inputPaths)) :: Nil
--- End diff --
Sorry, my bad, and thanks for pointing this out! Also realized this when migrating Parquet data source. I should populate partition values here. `t.buildScan` should only accept data file paths within a single selected partition. So for a table scan operation with `N` selected partitions:
1. `t.buildScan` is called `N` times, resulting `N` RDDs
2. Populate partition values accordingly to each RDD
3. Union all RDDs to form the final result
This is similar to what we've done in `HiveTableScan`.
I'm updating this part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97505075
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29361708
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -341,9 +341,13 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
case c: CreateTableUsing if c.temporary && c.allowExisting =>
sys.error("allowExisting should be set to false when creating a temporary table.")
- case CreateTableUsingAsSelect(tableName, provider, true, mode, opts, query) =>
- val cmd =
- CreateTempTableUsingAsSelect(tableName, provider, mode, opts, query)
+ case CreateTableUsingAsSelect(tableName, provider, true, partitionsCols, mode, opts, query)
+ if partitionsCols.nonEmpty =>
+ sys.error("Cannot create temporary partitioned table.")
--- End diff --
Or maybe "partitioned temporary table"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97952780
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31437/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29385127
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala ---
@@ -53,6 +53,25 @@ private[sql] object DataSourceStrategy extends Strategy {
filters,
(a, _) => t.buildScan(a)) :: Nil
+ case PhysicalOperation(projectList, filters, l @ LogicalRelation(t: FSBasedRelation)) =>
+ val selectedPartitions = prunePartitions(filters, t.partitionSpec)
+ val inputPaths = selectedPartitions.map(_.path).toArray
+
+ // Don't push down predicates that reference partition columns
+ val pushedFilters = {
+ val partitionColumnNames = t.partitionSpec.partitionColumns.map(_.name).toSet
+ filters.filter { f =>
+ val referencedColumnNames = f.references.map(_.name).toSet
+ referencedColumnNames.intersect(partitionColumnNames).isEmpty
+ }
+ }
+
+ pruneFilterProject(
+ l,
+ projectList,
+ pushedFilters,
+ (a, f) => t.buildScan(a, f, inputPaths)) :: Nil
--- End diff --
Trying to understand the logic. How does the value in partition get populated to each row?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28481733
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
--- End diff --
We didn't decide whether to make `FSBaseRelation` a trait or an abstract class during the design review. I left it as a trait here so that it's more consistent with other existing traits. Will make it an abstract class and add the other 3 variants as discussed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95640578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30850/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93424080
[Test build #30347 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30347/consoleFull) for PR 5526 at commit [`e209ab0`](https://github.com/apache/spark/commit/e209ab0f6641bc0a68af12621cf9733f60085f42).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait PartitionedSchemaRelationProvider `
* `trait OutputWriter `
* `trait FSBasedRelation extends BaseRelation `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95614545
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30846/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r29080102
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/FSBasedRelationSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.io.IOException
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SQLContext, SaveMode}
+import org.apache.spark.util.Utils
+
+class SimpleFSBasedSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleFSBasedRelation(parameters)(sqlContext)
+ }
+}
+
+case class SimpleFSBasedRelation
+ (parameter: Map[String, String])
+ (val sqlContext: SQLContext)
+ extends FSBasedRelation {
+
+ class SimpleOutputWriter extends OutputWriter {
+ override def write(row: Row): Unit = TestResult.writtenRows += row
+ }
+
+ override val path = parameter("path")
+
+ override def dataSchema: StructType =
+ DataType.fromJson(parameter("schema")).asInstanceOf[StructType]
+
+ override def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row] = {
+ val basePath = new Path(path)
+ val fs = basePath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
+
+ // Used to test queries like "INSERT OVERWRITE tbl SELECT * FROM tbl". Source directory
+ // shouldn't be removed before scanning the data.
+ assert(inputPaths.map(new Path(_)).forall(fs.exists))
+
+ TestResult.requiredColumns = requiredColumns
+ TestResult.filters = filters
+ TestResult.inputPaths = inputPaths
+
+ Option(TestResult.rowsToRead).getOrElse(sqlContext.emptyResult)
+ }
+
+ override def outputWriterClass: Class[_ <: OutputWriter] = classOf[SimpleOutputWriter]
+}
+
+object TestResult {
+ var requiredColumns: Array[String] = _
+ var filters: Array[Filter] = _
+ var inputPaths: Array[String] = _
+ var rowsToRead: RDD[Row] = _
+ var writtenRows: ArrayBuffer[Row] = ArrayBuffer.empty[Row]
+
+ def reset(): Unit = {
+ requiredColumns = null
+ filters = null
+ inputPaths = null
+ rowsToRead = null
+ writtenRows.clear()
+ }
+}
+
+class FSBasedRelationSuite extends DataSourceTest {
--- End diff --
Please also add error message tests for when the functions are called on a datasource that does not support partitioning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28470478
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row]
+
+ /**
+ * When writing rows to this relation, this method is invoked on the driver side before the actual
+ * write job is issued. It provides an opportunity to configure the write job to be performed.
+ */
+ def prepareForWrite(conf: Configuration): Unit
+
+ /**
+ * This method is responsible for producing a new [[OutputWriter]] for each newly opened output
+ * file on the executor side.
+ */
+ def newOutputWriter(path: String): OutputWriter
--- End diff --
It did have to be serializable before as it is hooked into a query plan.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93432036
[Test build #30349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30349/consoleFull) for PR 5526 at commit [`b46ee49`](https://github.com/apache/spark/commit/b46ee49c8e8915a745704713f1f15ceaf3961a79).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28540438
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
--- End diff --
Summary of our offline discussion:
- For dynamic partitioning, partition column values must be retrieved from given rows. However, when writing to a partition directory, we can drop dynamic columns. So the `row` argument of `write(row: Row): Unit` needn't to contain partition columns.
- Dropping dynamic columns is compatible with Hive
- Keeping dynamic columns can be more convenient in the sense that the data files can be accessed independently without extracting partition columns from partition directory paths. However
For this version, we drop all dynamic partition columns for Hive compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97473976
[Test build #31291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31291/consoleFull) for PR 5526 at commit [`64c9021`](https://github.com/apache/spark/commit/64c9021a9f313d450bda09826be4115d715c4fad).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28453018
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -78,6 +80,40 @@ trait SchemaRelationProvider {
schema: StructType): BaseRelation
}
+/**
+ * ::DeveloperApi::
+ * Implemented by objects that produce relations for a specific kind of data source
+ * with a given schema and partitioned columns. When Spark SQL is given a DDL operation with a
+ * USING clause specified (to specify the implemented SchemaRelationProvider), a user defined
+ * schema, and an optional list of partition columns, this interface is used to pass in the
+ * parameters specified by a user.
+ *
+ * Users may specify the fully qualified class name of a given data source. When that class is
+ * not found Spark SQL will append the class name `DefaultSource` to the path, allowing for
+ * less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the
+ * data source 'org.apache.spark.sql.json.DefaultSource'
+ *
+ * A new instance of this class with be instantiated each time a DDL call is made.
+ *
+ * The difference between a [[RelationProvider]] and a [[PartitionedSchemaRelationProvider]] is
+ * that users need to provide a schema and a (possibly empty) list of partition columns when
+ * using a SchemaRelationProvider. A relation provider can inherits both [[RelationProvider]],
+ * and [[PartitionedSchemaRelationProvider]] if it can support schema inference, user-specified
+ * schemas, and accessing partitioned relations.
+ */
+trait PartitionedSchemaRelationProvider {
+ /**
+ * Returns a new base relation with the given parameters, a user defined schema, and a list of
+ * partition columns. Note: the parameters' keywords are case insensitive and this insensitivity
+ * is enforced by the Map that is passed to the function.
+ */
+ def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String],
+ schema: StructType,
+ partitionColumns: StructType): BaseRelation
--- End diff --
It seems to me Scala users can use Java maps pretty easily -- but that's maybe a broader debate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97514775
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-95635721
[Test build #30850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30850/consoleFull) for PR 5526 at commit [`b63f813`](https://github.com/apache/spark/commit/b63f81375e3e3cdea884c2e7c3f1925294c61c21).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97475092
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31289/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-93477462
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30344/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97995031
[Test build #31470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31470/consoleFull) for PR 5526 at commit [`f39cbd4`](https://github.com/apache/spark/commit/f39cbd4a7f1d50b04b5fc83411e016e0ee873506).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-98000640
[Test build #31470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31470/consoleFull) for PR 5526 at commit [`f39cbd4`](https://github.com/apache/spark/commit/f39cbd4a7f1d50b04b5fc83411e016e0ee873506).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* ` public static final class UNSAFE `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] [WIP] Partitioning support for the data ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5526#discussion_r28482359
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -197,3 +233,69 @@ trait InsertableRelation {
trait CatalystScan {
def buildScan(requiredColumns: Seq[Attribute], filters: Seq[Expression]): RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * [[OutputWriter]] is used together with [[FSBasedRelation]] for persisting rows to the
+ * underlying file system. An [[OutputWriter]] instance is created when a new output file is
+ * opened. This instance is used to persist rows to this single output file.
+ */
+@Experimental
+trait OutputWriter {
+ /**
+ * Persists a single row. Invoked on the executor side.
+ */
+ def write(row: Row): Unit
+
+ /**
+ * Closes the [[OutputWriter]]. Invoked on the executor side after all rows are persisted, before
+ * the task output is committed.
+ */
+ def close(): Unit
+}
+
+/**
+ * ::Experimental::
+ * A [[BaseRelation]] that abstracts file system based data sources.
+ *
+ * For the read path, similar to [[PrunedFilteredScan]], it can eliminate unneeded columns and
+ * filter using selected predicates before producing an RDD containing all matching tuples as
+ * [[Row]] objects.
+ *
+ * In addition, when reading from Hive style partitioned tables stored in file systems, it's able to
+ * discover partitioning information from the paths of input directories, and perform partition
+ * pruning before start reading the data.
+ *
+ * For the write path, it provides the ability to write to both non-partitioned and partitioned
+ * tables. Directory layout of the partitioned tables is compatible with Hive.
+ */
+@Experimental
+trait FSBasedRelation extends BaseRelation {
+ /**
+ * Builds an `RDD[Row]` containing all rows within this relation.
+ *
+ * @param requiredColumns Required columns.
+ * @param filters Candidate filters to be pushed down. The actual filter should be the conjunction
+ * of all `filters`. The pushed down filters are currently purely an optimization as they
+ * will all be evaluated again. This means it is safe to use them with methods that produce
+ * false positives such as filtering partitions based on a bloom filter.
+ * @param inputPaths Data files to be read. If the underlying relation is partitioned, only data
+ * files within required partition directories are included.
+ */
+ def buildScan(
+ requiredColumns: Array[String],
+ filters: Array[Filter],
+ inputPaths: Array[String]): RDD[Row]
+
+ /**
+ * When writing rows to this relation, this method is invoked on the driver side before the actual
+ * write job is issued. It provides an opportunity to configure the write job to be performed.
+ */
+ def prepareForWrite(conf: Configuration): Unit
+
+ /**
+ * This method is responsible for producing a new [[OutputWriter]] for each newly opened output
+ * file on the executor side.
+ */
+ def newOutputWriter(path: String): OutputWriter
--- End diff --
That was never part of the contract, was there? Also up until this point, it could've been a transient variable, because we only need it on the driver side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97517725
[Test build #31291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31291/consoleFull) for PR 5526 at commit [`64c9021`](https://github.com/apache/spark/commit/64c9021a9f313d450bda09826be4115d715c4fad).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97475079
[Test build #31289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31289/consoleFull) for PR 5526 at commit [`43078dc`](https://github.com/apache/spark/commit/43078dc8d1c6a788916a3508bc747f47e5f4bb8e).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch **adds the following new dependencies:**
* `commons-math3-3.1.1.jar`
* `snappy-java-1.1.1.6.jar`
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
* This patch **removes the following dependencies:**
* `commons-math3-3.4.1.jar`
* `snappy-java-1.1.1.7.jar`
* `tachyon-0.6.4.jar`
* `tachyon-client-0.6.4.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] [WIP] Partitioning support ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97514742
[Test build #31290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31290/consoleFull) for PR 5526 at commit [`03bf5eb`](https://github.com/apache/spark/commit/03bf5eb4b25ba4ac57bcd11f58b2f1a6f08deb8e).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `trait FSBasedRelationProvider `
* `abstract class OutputWriter `
* This patch **adds the following new dependencies:**
* `commons-math3-3.1.1.jar`
* `snappy-java-1.1.1.6.jar`
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
* This patch **removes the following dependencies:**
* `commons-math3-3.4.1.jar`
* `snappy-java-1.1.1.7.jar`
* `spark-unsafe_2.10-1.4.0-SNAPSHOT.jar`
* `tachyon-0.6.4.jar`
* `tachyon-client-0.6.4.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5182] [SQL] Partitioning support for th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5526#issuecomment-97952778
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org