You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mallman <gi...@git.apache.org> on 2017/01/13 18:31:00 UTC
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
GitHub user mallman opened a pull request:
https://github.com/apache/spark/pull/16578
[SPARK-4502][SQL] Parquet nested column pruning
(Link to Jira: https://issues.apache.org/jira/browse/SPARK-4502)
## What changes were proposed in this pull request?
One of the hallmarks of a column-oriented data storage format is the ability to read data from a subset of columns, efficiently skipping reads from other columns. Spark has long had support for pruning unneeded top-level schema fields from the scan of a parquet file. For example, consider a table, `contacts`, backed by parquet with the following Spark SQL schema:
```
root
|-- name: struct
| |-- first: string
| |-- last: string
|-- address: string
```
Parquet stores this table's data in three physical columns: `name.first`, `name.last` and `address`. To answer the query
```SQL
select address from contacts
```
Spark will read only from the `address` column of parquet data. However, to answer the query
```SQL
select name.first from contacts
```
Spark will read `name.first` and `name.last` from parquet.
This PR modifies Spark SQL to support a finer-grain of schema pruning. With this patch, Spark reads only the `name.first` column to answer the previous query.
### Implementation
There are three main components of this patch. First, there is a `ParquetSchemaPruning` optimizer rule for gathering the required schema fields of a `PhysicalOperation` over a parquet file, constructing a new schema based on those required fields and rewriting the plan in terms of that pruned schema. The pruned schema fields are pushed down to the parquet requested read schema. `ParquetSchemaPruning` uses a new `ProjectionOverSchema` extractor for rewriting a catalyst expression in terms of a pruned schema.
Second, the `ParquetRowConverter` has been patched to ensure the ordinals of the parquet columns read are correct for the pruned schema. `ParquetReadSupport` has been patched to address a compatibility mismatch between Spark's built in vectorized reader and the parquet-mr library's reader.
Third, we introduce two new catalyst query transformations, `AggregateFieldExtractionPushdown` and `JoinFieldExtractionPushdown`, to support schema pruning in aggregation and join query plans. These rules extract field references in aggregations and joins respectively, push down aliases to those references and replace them with references to the pushed down aliases. They use a new `SelectedField` extractor that transforms a catalyst complex type extractor (the "selected field") into a corresponding `StructField`.
### Performance
The performance difference in executing queries with this patch compared to master is related to the depth of the table schema and the query itself. At VideoAmp, one of our biggest tables stores OpenRTB bid requests we receive from our exchange partners. Our bid request table's schema closely follows the OpenRTB bid request object schema. Additionally, when we bid we save our response along with the request in the same table. We store these two objects as two top-level fields in our table. Therefore, all bid request and response data are contained within nested fields.
For the purposes of measuring the performance impact of this patch, we ran some queries on our bid request table with the un-patched and patched master. We measured query execution time and the amount of data read from the underlying parquet files. I'll focus on a couple of benchmarks. (All benchmarks were run on an AWS EC2 cluster with four c3.8xl workers.) The first query I'll highlight is
```SQL
select count(request.device.ip) from event.bid_request where ds=20161128 and h=0
```
(Hopefully it's obvious what this query means.) On the un-patched master, this query ran in 2.7 minutes and read 34.3 GB of data. On the patched master, this query ran in 4 seconds and read 987.3 MB of data.
We also ran a reporting-oriented query benchmark. I won't reproduce the query here, but it reads a larger subset of the bid request fields and joins against another table with a deeply nested schema. In addition to a join, we perform several aggregations in this query. On the un-patched master, this query ran in 3.4 minutes and read 34.6 GB of data. On the patched master, this query ran in 59 seconds and read 2.6 GB of data.
### Limitation
Among the complex Spark SQL data types, this patch supports parquet column pruning of nested sequences of struct fields only.
## How was this patch tested?
Care has been taken to ensure correctness and prevent regressions. This patch introduces over two dozen new unit tests and has been running on a production Spark 1.5 cluster at VideoAmp for about a year. In that time, one bug was found and fixed early on, and we added a regression test for that bug.
We forward-ported this patch to Spark master in June 2016 and have been running this patch against Spark 2.0 and 2.1 branches on ad-hoc clusters since then.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/VideoAmp/spark-public spark-4502-parquet_column_pruning
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16578.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16578
----
commit 89253722bde65b02467d05f2b64c4fdbef3b885d
Author: Michael Allman <mi...@videoamp.com>
Date: 2016-06-24T17:21:24Z
[SPARK-4502][SQL] Parquet nested column pruning
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75181/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85662/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #87859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87859/testReport)** for PR 16578 at commit [`27737a0`](https://github.com/apache/spark/commit/27737a07ea39e953add1fab74d877c6543206a29).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@nfx I'll need help finishing this. Can you review the current patchset?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148452868
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a nested field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
+ val dataSchemaFieldNames = dataSchema.fieldNames.toSet
+ val prunedDataSchema =
+ StructType(prunedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))
+
+ // If the data schema is different from the pruned data schema, continue. Otherwise,
+ // return [[op]]. We effect this comparison by counting the number of "leaf" fields in
+ // each schemata, assuming the fields in [[prunedDataSchema]] are a subset of the fields
+ // in [[dataSchema]].
+ if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
+ val prunedParquetRelation =
+ hadoopFsRelation.copy(dataSchema = prunedDataSchema)(hadoopFsRelation.sparkSession)
+
+ // We need to replace the expression ids of the pruned relation output attributes
+ // with the expression ids of the original relation output attributes so that
+ // references to the original relation's output are not broken
+ val outputIdMap = l.output.map(att => (att.name, att.exprId)).toMap
+ val prunedRelationOutput =
--- End diff --
Will we change the top-level attributes during pruning and make a different output for the logical relation? I think only the data types of the nested columns are changed?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
I cannot do it. @marmbrus can you help with this pull request ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #83128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83128/testReport)** for PR 16578 at commit [`7af925b`](https://github.com/apache/spark/commit/7af925bcf44861119ec305ab9631a3511d8e8bbb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82047/testReport)** for PR 16578 at commit [`38cec5f`](https://github.com/apache/spark/commit/38cec5f05066f75d868b64d360b04edb18dcebeb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #77342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77342/testReport)** for PR 16578 at commit [`464b18d`](https://github.com/apache/spark/commit/464b18d61b578ee5621b7a22f41955dd0f8b60a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #87859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87859/testReport)** for PR 16578 at commit [`27737a0`](https://github.com/apache/spark/commit/27737a07ea39e953add1fab74d877c6543206a29).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AggregateFieldExtractionPushdownSuite extends SchemaPruningTest `
* `class JoinFieldExtractionPushdownSuite extends SchemaPruningTest `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Swat123 <gi...@git.apache.org>.
Github user Swat123 commented on the issue:
https://github.com/apache/spark/pull/16578
@viirya can we close this before we get another set of merge conflicts ? Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
>Test build #82051 has finished for PR 16578 at commit 00ab80c.
>
> * This patch fails Spark unit tests.
> * This patch merges cleanly.
> * This patch adds no public classes.
Oh crap. I know what happened here. I've been updating some of the unit tests locally, and I've been running changes against these modified tests. I'll fix this and push a commit.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #75181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75181/testReport)** for PR 16578 at commit [`30bc7e9`](https://github.com/apache/spark/commit/30bc7e9e29b639c65e3839171aa7b527c17e756b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82008/testReport)** for PR 16578 at commit [`a267701`](https://github.com/apache/spark/commit/a26770189fb2ba07d76e4e9d10c3fd47b26a9d8d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83389/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
Looks like this build was `kill -9`ed by something:
```
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/build/sbt -Phadoop-2.6 -Phive-thriftserver -Phive -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest hive-thriftserver/test mllib/test hive/test examples/test sql/test sql-kafka-0-10/test catalyst/test ; process was terminated by signal 9
Build step 'Execute shell' marked build as failure
```
Can we get a retest, puhleeze?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148437549
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/JoinFieldExtractionPushdown.scala ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in a projection over a join
+ * and its join condition. The original [[expressions.GetStructField]] expressions are replaced
+ * with references to the pushed down aliases.
+ */
+object JoinFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, Seq(),
+ join @ Join(left, right, joinType, Some(joinCondition))) =>
+ val fieldExtractors = (projects :+ joinCondition).flatMap(getFieldExtractors).distinct
+
+ if (fieldExtractors.nonEmpty) {
+ val (aliases, substituteAttributes) = constructAliasesAndSubstitutions(fieldExtractors)
--- End diff --
We can return original plan if `aliases` is empty.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139331198
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
+ ParquetReadSupport.intersectParquetGroups(clippedParquetSchema, context.getFileSchema)
+ .map(intersectionGroup =>
+ new MessageType(intersectionGroup.getName, intersectionGroup.getFields))
+ .getOrElse(ParquetSchemaConverter.EMPTY_MESSAGE)
+ } else {
+ // Spark's built-in Parquet reader will throw an exception in some cases if the requested
+ // schema is not the same as the clipped schema
--- End diff --
I think the built-in Parquet reader means vectorized reader. But I think we don't use `ParquetReadSupport` in vectorized reader?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148720677
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdown.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in an aggregate's grouping and
+ * aggregate expressions into a projection over its children. The original
+ * [[expressions.GetStructField]] expressions are replaced with references to the pushed down
+ * aliases.
+ */
+object AggregateFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case agg @ Aggregate(groupingExpressions, aggregateExpressions, child) =>
+ val expressions = groupingExpressions ++ aggregateExpressions
+ val attributes = AttributeSet(expressions.collect { case att: Attribute => att })
+ val childAttributes = AttributeSet(child.expressions)
+ val fieldExtractors0 =
+ expressions
+ .flatMap(getFieldExtractors)
+ .distinct
+ val fieldExtractors1 =
+ fieldExtractors0
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(attributes.contains).isEmpty)
+ val fieldExtractors =
+ fieldExtractors1
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(childAttributes.contains).nonEmpty)
+
+ if (fieldExtractors.nonEmpty) {
+ val (aliases, substituteAttributes) = constructAliasesAndSubstitutions(fieldExtractors)
+
+ // Construct the new grouping and aggregate expressions by substituting
+ // each GetStructField expression with a reference to its alias
+ val newAggregateExpressions =
+ aggregateExpressions.map(substituteAttributes)
+ .collect { case named: NamedExpression => named }
--- End diff --
I meant it might be like: `aggregateExpressions.map(substituteAttributes).asInstanceOf[Seq[NamedExpression]]`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> Yeah, I think with a config for this optimization is good.
I added a config switch, `spark.sql.nestedSchemaPruning.enabled`, which disables the optimizations if set to `false`. By default it's `true`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16578
To ensure the PR and review quality, we normally avoid doing everything in a single huge PR. It would be much better if you can cut it to a few smaller PRs. Both @cloud-fan and I think separating the optimizer rules makes sense. WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98809770
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
Can we have a better name for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16578
Please do the review @gengliangwang @jiangxb1987 . We should support this feature in Spark 2.4.0
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16578
thanks! ping/add @rxin @hvanhovell @gatorsmile @cloud-fan @liancheng @joseph-torres
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@viirya I've rebased to resolve conflicts. All tests are passing. Can you take another look and sign off?
Cheers.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
Let me clean up `SelectedField.scala` before we proceed further.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #84820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84820/testReport)** for PR 16578 at commit [`1936c9b`](https://github.com/apache/spark/commit/1936c9b2e4cf4008e5ee7282c6371fc0ca0535bb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AggregateFieldExtractionPushdownSuite extends SchemaPruningTest `
* `class JoinFieldExtractionPushdownSuite extends SchemaPruningTest `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98844304
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => field }
--- End diff --
Those two `map` can be done one `map`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
New year, new review? ;)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
I can't tell what's causing the build to fail:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83390/console
Any ideas?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
I've been on vacation and traveling since the 26th, hence my sluggishness in responding. I will take some time to work on this tonight. I'll start by rebasing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #83532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83532/testReport)** for PR 16578 at commit [`48a509e`](https://github.com/apache/spark/commit/48a509e8602ed44a4a0fd5268d91d917bb8e0748).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148725325
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/FileSchemaPruningTest.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.scalactic.Equality
+import org.scalatest.Assertions
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.types.StructType
+
+private[sql] trait FileSchemaPruningTest {
+ _: Assertions =>
+
+ private val schemaEquality = new Equality[StructType] {
+ override def areEqual(a: StructType, b: Any) =
+ b match {
+ case otherType: StructType => a sameType otherType
+ case _ => false
+ }
+ }
+
+ protected def checkScanSchemata(df: DataFrame, expectedSchemaCatalogStrings: String*): Unit = {
--- End diff --
I used the plural form here because `expectedSchemaCatalogStrings` is a varargs type.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139346525
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
--- End diff --
Is the exception only thrown when we read a superset of fields for a nested column? E.g., there is a struct of two fields "a" and "b", and we try to read "c".
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> can I do something to help this pull request?
Hi @Gauravshah. Thanks for asking. Right now I need to fix a broken piece of the code, or reimplement it. At the moment this is something I'm dedicating myself to. Once I have that completed, a code review would be very helpful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@snir It's hard to pin down exactly where the problem is and how to fix it. We removed the `NullIntolerant` trait from the implementations of the `ExtractValue` trait in `complexTypeExtractors.scala` and that fixed the problem. I've been reluctant to push a commit with this change, because I don't know that this is correct. The problem you raise is an open issue with this PR and a blocker for its merge.
Of course, getting a review of this PR is a blocker, too. 😉
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16578
Yeah, I think with a config for this optimization is good.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148456230
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/FileSchemaPruningTest.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.scalactic.Equality
+import org.scalatest.Assertions
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.types.StructType
+
+private[sql] trait FileSchemaPruningTest {
+ _: Assertions =>
+
+ private val schemaEquality = new Equality[StructType] {
+ override def areEqual(a: StructType, b: Any) =
+ b match {
+ case otherType: StructType => a sameType otherType
+ case _ => false
+ }
+ }
+
+ protected def checkScanSchemata(df: DataFrame, expectedSchemaCatalogStrings: String*): Unit = {
+ val fileSourceScanSchemata =
--- End diff --
`fileSourceScanSchemata` -> `fileSourceScanSchema`? and also below.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148433577
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/ProjectionOverSchema.scala ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.types._
+
+/**
+ * A Scala extractor that projects an expression over a given schema. Data types,
+ * field indexes and array lengths of complex type extractors and attributes
--- End diff --
Have we changed array lengths? I just saw adjusting to data type, field index.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75178/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98819150
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
What do you mean by combining it with the existing case class extractor?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82381/testReport)** for PR 16578 at commit [`88786d3`](https://github.com/apache/spark/commit/88786d3eaf9d3a3d2c80c2235c5014074ade3dc1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82045/testReport)** for PR 16578 at commit [`2524a48`](https://github.com/apache/spark/commit/2524a4847968e8a814009e74842bbf7261cc5da6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139312181
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
--- End diff --
Can the merging order of the fields affect the comparison later in line 57?
For example, depending on the merging order, if we require `a.field1` and `a.field2`, is it possibly we may have `StructType(StructField("a", StructType(StructField("field1") :: StructField("field2") :: Nil)) :: Nil)` or `StructType(StructField("a", StructType(StructField("field2") :: StructField("field1") :: Nil)) :: Nil)` as merged schema?
Once we have the later, and the data schema is the first, at line 57 we may think they are different?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #89794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89794/testReport)** for PR 16578 at commit [`dd4f2d8`](https://github.com/apache/spark/commit/dd4f2d8829335b9d9e71fead6d0d056d48a9d7e6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #84812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84812/testReport)** for PR 16578 at commit [`a3d5d9b`](https://github.com/apache/spark/commit/a3d5d9b1c9fa5746016ffc7d2e88eda921503f4f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AggregateFieldExtractionPushdownSuite extends SchemaPruningTest `
* `class JoinFieldExtractionPushdownSuite extends SchemaPruningTest `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman closed the pull request at:
https://github.com/apache/spark/pull/16578
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman Thanks for let me know. I'll try your patch and check #14957 take over or not.
I also think we need getting feedback from @liancheng , from our last discussion, liancheng may do some work based on the old PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82009/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #84041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84041/testReport)** for PR 16578 at commit [`75971ae`](https://github.com/apache/spark/commit/75971aed0cec9aa2e0e26b593eb9c01164303f1f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AggregateFieldExtractionPushdownSuite extends SchemaPruningTest `
* `class JoinFieldExtractionPushdownSuite extends SchemaPruningTest `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16578
We are still merging changes to the 2.3 branch :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148731266
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/planning/SelectedFieldSuite.scala ---
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.scalatest.BeforeAndAfterAll
+import org.scalatest.exceptions.TestFailedException
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.NamedExpression
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.types._
+
+// scalastyle:off line.size.limit
+class SelectedFieldSuite extends SparkFunSuite with BeforeAndAfterAll {
+ // The test schema as a tree string, i.e. `schema.treeString`
+ // root
+ // |-- col1: string (nullable = false)
+ // |-- col2: struct (nullable = true)
+ // | |-- field1: integer (nullable = true)
+ // | |-- field2: array (nullable = true)
+ // | | |-- element: integer (containsNull = false)
+ // | |-- field3: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | | |-- subfield3: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = true)
+ // | |-- field4: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = false)
+ // | |-- field5: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: struct (nullable = false)
+ // | | | | |-- subsubfield1: integer (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | | | |-- subfield2: struct (nullable = true)
+ // | | | | |-- subsubfield1: struct (nullable = true)
+ // | | | | | |-- subsubsubfield1: string (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field6: struct (nullable = true)
+ // | | |-- subfield1: string (nullable = false)
+ // | | |-- subfield2: string (nullable = true)
+ // | |-- field7: struct (nullable = true)
+ // | | |-- subfield1: struct (nullable = true)
+ // | | | |-- subsubfield1: integer (nullable = true)
+ // | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field8: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: array (valueContainsNull = false)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: array (nullable = true)
+ // | | | | | |-- element: integer (containsNull = false)
+ // | |-- field9: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: integer (valueContainsNull = false)
+ // |-- col3: array (nullable = false)
+ // | |-- element: struct (containsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col4: map (nullable = false)
+ // | |-- key: string
+ // | |-- value: struct (valueContainsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col5: array (nullable = true)
+ // | |-- element: map (containsNull = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col6: map (nullable = true)
+ // | |-- key: string
+ // | |-- value: array (valueContainsNull = true)
+ // | | |-- element: struct (containsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col7: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: integer (nullable = false)
+ // | | |-- field2: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | |-- field3: array (nullable = true)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = false)
+ // |-- col8: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: array (nullable = false)
+ // | | | |-- element: integer (containsNull = false)
+ private val schema =
+ StructType(
+ StructField("col1", StringType, nullable = false) ::
+ StructField("col2", StructType(
+ StructField("field1", IntegerType) ::
+ StructField("field2", ArrayType(IntegerType, containsNull = false)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) ::
+ StructField("subfield3", ArrayType(IntegerType)) :: Nil)), nullable = false) ::
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil), valueContainsNull = false)) ::
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil), nullable = false) ::
+ StructField("subfield2", StructType(
+ StructField("subsubfield1", StructType(
+ StructField("subsubsubfield1", StringType) :: Nil)) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)), nullable = false) ::
+ StructField("field6", StructType(
+ StructField("subfield1", StringType, nullable = false) ::
+ StructField("subfield2", StringType) :: Nil)) ::
+ StructField("field7", StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)) ::
+ StructField("field8", MapType(StringType, ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil)), valueContainsNull = false)) ::
+ StructField("field9", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil)) ::
+ StructField("col3", ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), containsNull = false), nullable = false) ::
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), valueContainsNull = false), nullable = false) ::
+ StructField("col5", ArrayType(MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), valueContainsNull = false))) ::
+ StructField("col6", MapType(StringType, ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), containsNull = false))) ::
+ StructField("col7", ArrayType(StructType(
+ StructField("field1", IntegerType, nullable = false) ::
+ StructField("field2", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil))) :: Nil))) ::
+ StructField("col8", ArrayType(StructType(
+ StructField("field1", ArrayType(IntegerType, containsNull = false), nullable = false) :: Nil))) :: Nil)
+
+ private val testRelation = LocalRelation(schema.toAttributes)
+
+ override def beforeAll() {
+ val indentedSchema = schema.treeString.split('\n').map(x => " " + x).mkString("\n") + "\n"
+ // Print the test schema at the beginning of the test
+ // scalastyle:off println
+ println("Testing SelectedField extractor using the following schema:\n" + indentedSchema)
+ // scalastyle:on println
+ }
+
+ test("should not match an attribute reference") {
+ assertResult(None)(unapplySelect("col1"))
+ assertResult(None)(unapplySelect("col1 as foo"))
+ assertResult(None)(unapplySelect("col2"))
+ }
+
+ test("col2.field2, col2.field2[0] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field2", ArrayType(IntegerType, containsNull = false)) :: Nil))
+ testSelect("col2.field2", expected)
+ testSelect("col2.field2[0] as foo", expected)
+ }
+
+ test("col2.field9, col2.field9['foo'] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field9", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil))
+ testSelect("col2.field9", expected)
+ testSelect("col2.field9['foo'] as foo", expected)
+ }
+
+ test("col2.field3.subfield3, col2.field3[0].subfield3 as foo, col2.field3.subfield3[0] as foo, col2.field3[0].subfield3[0] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield3", ArrayType(IntegerType)) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field3.subfield3", expected)
+ testSelect("col2.field3[0].subfield3 as foo", expected)
+ testSelect("col2.field3.subfield3[0] as foo", expected)
+ testSelect("col2.field3[0].subfield3[0] as foo", expected)
+ }
+
+ test("col2.field3.subfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field3.subfield1", expected)
+ }
+
+ test("col2.field5.subfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil), nullable = false) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field5.subfield1", expected)
+ }
+
+ test("col3.field1.subfield1") {
+ val expected =
+ StructField("col3", ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) :: Nil), containsNull = false), nullable = false)
+ testSelect("col3.field1.subfield1", expected)
+ }
+
+ test("col3.field2['foo'] as foo") {
+ val expected =
+ StructField("col3", ArrayType(StructType(
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), containsNull = false), nullable = false)
+ testSelect("col3.field2['foo'] as foo", expected)
+ }
+
+ test("col4['foo'].field1.subfield1 as foo") {
+ val expected =
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) :: Nil), valueContainsNull = false), nullable = false)
+ testSelect("col4['foo'].field1.subfield1 as foo", expected)
+ }
+
+ test("col4['foo'].field2['bar'] as foo") {
+ val expected =
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), valueContainsNull = false), nullable = false)
+ testSelect("col4['foo'].field2['bar'] as foo", expected)
+ }
+
+ test("col5[0]['foo'].field1.subfield1 as foo") {
+ val expected =
+ StructField("col5", ArrayType(MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) :: Nil)) :: Nil), valueContainsNull = false)))
+ testSelect("col5[0]['foo'].field1.subfield1 as foo", expected)
+ }
+
+ test("col6['foo'][0].field1.subfield1 as foo") {
+ val expected =
+ StructField("col6", MapType(StringType, ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) :: Nil)) :: Nil), containsNull = false)))
+ testSelect("col6['foo'][0].field1.subfield1 as foo", expected)
+ }
+
+ test("col2.field5.subfield1.subsubfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) :: Nil), nullable = false) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field5.subfield1.subsubfield1", expected)
+ }
+
+ test("col2.field5.subfield2.subsubfield1.subsubsubfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield2", StructType(
+ StructField("subsubfield1", StructType(
+ StructField("subsubsubfield1", StringType) :: Nil)) :: Nil)) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field5.subfield2.subsubfield1.subsubsubfield1", expected)
+ }
+
+ test("col2.field4['foo'].subfield1 as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield1", IntegerType) :: Nil), valueContainsNull = false)) :: Nil))
+ testSelect("col2.field4['foo'].subfield1 as foo", expected)
+ }
+
+ test("col2.field4['foo'].subfield2 as foo, col2.field4['foo'].subfield2[0] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil), valueContainsNull = false)) :: Nil))
--- End diff --
It's something I'll look into.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148450632
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
--- End diff --
Can you give an example it would fail? We didn't change `clipParquetSchema`, so I think even when pruning happens, why we read a super set of the file's schema and cause the exception, according to the comment? We won't add new fields but remove existing from the file's schema, right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@saulshanabrook I've pushed my latest work (rebased off the latest master). I'm not sure it's quite complete, but feel free to review and comment. FYI, the one piece of code that's given me the most trouble is the `SelectedField` extractor. It's passing all of the test cases I've thrown at it, but I can't quite convince myself that it's complete/correct. If you have some ideas for how to simplify it to make it easier to understand I welcome them.
If you'd like to submit actual changes to the code in this PR, please submit a PR against our https://github.com/VideoAmp/spark-public/tree/spark-4502-parquet_column_pruning branch, and I will review it there.
Thanks for your help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #79815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79815/testReport)** for PR 16578 at commit [`b0c67c2`](https://github.com/apache/spark/commit/b0c67c2efb9020a57bcb6955d4d2641a7cbd24d1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
Also, I'm confused about something—who has jenkins retest privileges? And can I get them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77794/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #71343 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71343/testReport)** for PR 16578 at commit [`efc4d76`](https://github.com/apache/spark/commit/efc4d7662c98a5ee8f719b5839ca389a2a987d97).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148436498
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdown.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in an aggregate's grouping and
+ * aggregate expressions into a projection over its children. The original
+ * [[expressions.GetStructField]] expressions are replaced with references to the pushed down
+ * aliases.
+ */
+object AggregateFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case agg @ Aggregate(groupingExpressions, aggregateExpressions, child) =>
+ val expressions = groupingExpressions ++ aggregateExpressions
+ val attributes = AttributeSet(expressions.collect { case att: Attribute => att })
+ val childAttributes = AttributeSet(child.expressions)
+ val fieldExtractors0 =
+ expressions
+ .flatMap(getFieldExtractors)
+ .distinct
+ val fieldExtractors1 =
+ fieldExtractors0
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(attributes.contains).isEmpty)
+ val fieldExtractors =
+ fieldExtractors1
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(childAttributes.contains).nonEmpty)
+
+ if (fieldExtractors.nonEmpty) {
+ val (aliases, substituteAttributes) = constructAliasesAndSubstitutions(fieldExtractors)
+
+ // Construct the new grouping and aggregate expressions by substituting
+ // each GetStructField expression with a reference to its alias
+ val newAggregateExpressions =
+ aggregateExpressions.map(substituteAttributes)
+ .collect { case named: NamedExpression => named }
--- End diff --
`asInstanceOf[Seq[NamedExpression]]`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
I'd just suggest trying it. Since this PR is a patch for master, please
message me personally at msa@allman.ms to discuss progress and questions
on a backport to 2.2. If we get it working, we can post back here with a
link to a fork.
Thanks for taking this on!
Michael
On Mon, 8 Jan 2018, Gaurav M Shah wrote:
>
> @mallman do you foresee any issues ? planning to backport it to spark 2.2 on
> personal fork. will probably make jitpack release
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the
> thread.[AAy4nfrO2mfWXiVObJZERmlMm1J9RH0Qks5tIoHYgaJpZM4LjK0N.gif]
>
>
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84820/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148720590
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FieldExtractionPushdown.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, GetStructField}
+import org.apache.spark.sql.catalyst.planning.SelectedField
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+
+abstract class FieldExtractionPushdown extends Rule[LogicalPlan] {
--- End diff --
Oh. I just thought `FieldExtractionPushdown` seems not to be necessarily a `Rule` and being just a simple trait. `AggregateFieldExtractionPushdown` and `JoinFieldExtractionPushdown` extends `Rule` and `FieldExtractionPushdown` then. Not a big deal.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83532/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82045/testReport)** for PR 16578 at commit [`2524a48`](https://github.com/apache/spark/commit/2524a4847968e8a814009e74842bbf7261cc5da6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148450744
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala ---
@@ -127,8 +127,8 @@ private[parquet] class ParquetRowConverter(
extends ParquetGroupConverter(updater) with Logging {
assert(
- parquetType.getFieldCount == catalystType.length,
- s"""Field counts of the Parquet schema and the Catalyst schema don't match:
+ parquetType.getFieldCount <= catalystType.length,
--- End diff --
Why it can be less than catalyst type length now?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by vkhristenko <gi...@git.apache.org>.
Github user vkhristenko commented on the issue:
https://github.com/apache/spark/pull/16578
Hi,
My name is Viktor and I'm working at CERN on ROOT I/O DataSource for JVM and an interface for Spark. ROOT I/O is the format that is used for CERN's LHC data.
ROOT data format (a columnar data format) is similar to Parquet format and is subject to pruning nested fields as well. I'm new to contributing to Apache Spark and this is why I'm writing this all explicitly.
- I found that this PR is more general than just Parquet!
- By using my source with this PR, buildReader function, https://github.com/diana-hep/spark-root/blob/master/src/main/scala/org/dianahep/sparkroot/experimental/package.scala#L86 , receives only the schema that is required by the df.select("") statement
- There is a minor change needed though, "parquetFormat: ParquetFileFormat" should be replaced by "fileFormat: FileFormat" as there is no dependency on the actual ParquetFileFormat class defined in parquet package
https://github.com/apache/spark/pull/16578/files?diff=unified#diff-3bad814b3336a83f360d7395bd740759R38
- And may be renaming this ParquetSchemaPruning and taking it outside of the parquet package as it is quite more general than just for parquet, otherwise I have to add a special Rule here, https://github.com/apache/spark/pull/16578/files?diff=unified#diff-2370d8ed85930c93ef8e5ce67abca53fR35 ???
Thanks!
VK
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82047/testReport)** for PR 16578 at commit [`38cec5f`](https://github.com/apache/spark/commit/38cec5f05066f75d868b64d360b04edb18dcebeb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16578
Yeah. That is fine. Will try to review the relevant PRs ASAP. Please ping me. Thanks again!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r99174674
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.types._
+
+/**
+ * A Scala extractor that builds a [[StructField]] from a Catalyst complex type
+ * extractor. This is like the opposite of [[ExtractValue#apply]].
+ */
+object SelectedField {
+ def unapply(expr: Expression): Option[StructField] = {
+ // If this expression is an alias, work on its child instead
+ val unaliased = expr match {
+ case Alias(child, _) => child
+ case expr => expr
+ }
+ selectField(unaliased, None)
+ }
+
+ /**
+ * Converts some chain of complex type extractors into a [[StructField]].
+ *
+ * @param expr the top-level complex type extractor
+ * @param fieldOpt the subfield of [[expr]], where relevent
+ */
+ private def selectField(expr: Expression, fieldOpt: Option[StructField]): Option[StructField] =
+ expr match {
+ case AttributeReference(name, _, nullable, _) =>
+ fieldOpt.map(field => StructField(name, StructType(Array(field)), nullable))
+ case GetArrayItem(GetStructField2(child, field @ StructField(name,
+ ArrayType(_, arrayNullable), fieldNullable, _)), _) =>
+ val childField = fieldOpt.map(field => StructField(name, ArrayType(
+ StructType(Array(field)), arrayNullable), fieldNullable)).getOrElse(field)
+ selectField(child, Some(childField))
+ case GetArrayStructFields(child,
--- End diff --
I believe I have a fix for this, but I probably won't be able to post a new commit until early next week\u2014I'm working on a proposal for the Spark Summit RFP.
Cheers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #79816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79816/testReport)** for PR 16578 at commit [`91368cd`](https://github.com/apache/spark/commit/91368cdaa306055e36bf13c2b3941852751131f4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16578
Thanks @CodingCat
+1 on config switch. I think that would be a good idea.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82028/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #84041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84041/testReport)** for PR 16578 at commit [`75971ae`](https://github.com/apache/spark/commit/75971aed0cec9aa2e0e26b593eb9c01164303f1f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r96048060
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdownSuite.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types._
+
+class AggregateFieldExtractionPushdownSuite extends PlanTest {
+ private val testRelation =
+ LocalRelation(
+ StructField("a", StructType(
+ StructField("a1", IntegerType) :: Nil)),
+ StructField("b", IntegerType),
+ StructField("c", StructType(
+ StructField("c1", IntegerType) :: Nil)))
+
+ object Optimizer extends RuleExecutor[LogicalPlan] {
+ val batches =
+ Batch("Aggregate Field Extraction Pushdown", Once,
+ AggregateFieldExtractionPushdown) :: Nil
+ }
+
+ test("basic aggregate field extraction pushdown") {
+ val originalQuery =
+ testRelation
+ .select('a)
+ .groupBy('a getField "a1")('a getField "a1" as 'a1, Count('*))
--- End diff --
replace it by `Count("*")`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #75178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75178/testReport)** for PR 16578 at commit [`b20da5c`](https://github.com/apache/spark/commit/b20da5cc25a169d8285daca51119f302812004c6).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r142140412
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
--- End diff --
I've updated the logic for comparing the original schema to the pruned schema:
https://github.com/apache/spark/blob/52fddc181f32726cea1dd12a23ebf7201986be01/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala#L53-L57
The following test validates that the selection order is irrelevant in comparing the pruned schema to the original schema:
https://github.com/apache/spark/blob/52fddc181f32726cea1dd12a23ebf7201986be01/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala#L82-L107
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
Time in seconds
| Test | w Patch | w/o Patch |
| ------------------------- | ------------- | ------------- |
| count(*) | ~20 | ~20 |
| 2 level nested field filter | 13 | 525 |
| 5 nested field filter | 11 | 550 |
| 10 nested field filter | 13 | 524 |
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman how about adding comment explaining why such workaround was done + bug number in parquet-mr ? So in future once that bug is fixed, code can be cleaned.
Also maybe it's time to remove "DO NOT MERGE" from title? As I understand most of comments were addressed :)
Thank you very much for work on this feature. I must admit that we are looking forward to have this merged. For us this will be most important improvement in Spark 2.3.0 (I hope it will be part of 2.3.0 :) )
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83387/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #71339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71339/testReport)** for PR 16578 at commit [`8925372`](https://github.com/apache/spark/commit/89253722bde65b02467d05f2b64c4fdbef3b885d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151344821
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
How about making it default true for tests?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
just in case someone wants to try:
```
resolvers += "jitpack" at "https://jitpack.io"
libraryDependencies += "com.github.VideoAmp" % "spark-public" % "spark-4502-parquet_column_pruning-SNAPSHOT"
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by nfx <gi...@git.apache.org>.
Github user nfx commented on the issue:
https://github.com/apache/spark/pull/16578
quite interested in this functionality
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77342/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148725482
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/FileSchemaPruningTest.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.scalactic.Equality
+import org.scalatest.Assertions
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.types.StructType
+
+private[sql] trait FileSchemaPruningTest {
+ _: Assertions =>
+
+ private val schemaEquality = new Equality[StructType] {
+ override def areEqual(a: StructType, b: Any) =
+ b match {
+ case otherType: StructType => a sameType otherType
+ case _ => false
+ }
+ }
+
+ protected def checkScanSchemata(df: DataFrame, expectedSchemaCatalogStrings: String*): Unit = {
+ val fileSourceScanSchemata =
--- End diff --
`fileSourceScanSchemata` is a `Seq[StructType]`, so I made it plural.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82382/testReport)** for PR 16578 at commit [`be1a257`](https://github.com/apache/spark/commit/be1a257576df76803524a944ff0e26a873d80b30).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151485646
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
Nope :( maybe @viirya can give input about it?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by sathchand <gi...@git.apache.org>.
Github user sathchand commented on the issue:
https://github.com/apache/spark/pull/16578
Hi Mallman,
I am looking for a similar fix for using it against spark 2.1. Do you have a fix for spark 2.1, I am willing to help you here..
regards
Sath
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148717122
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/JoinFieldExtractionPushdown.scala ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in a projection over a join
+ * and its join condition. The original [[expressions.GetStructField]] expressions are replaced
+ * with references to the pushed down aliases.
+ */
+object JoinFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, Seq(),
+ join @ Join(left, right, joinType, Some(joinCondition))) =>
+ val fieldExtractors = (projects :+ joinCondition).flatMap(getFieldExtractors).distinct
+
+ if (fieldExtractors.nonEmpty) {
+ val (aliases, substituteAttributes) = constructAliasesAndSubstitutions(fieldExtractors)
+
+ // Construct the new projections and join condition by substituting each GetStructField
+ // expression with a reference to its alias
+ val newProjects =
+ projects.map(substituteAttributes).collect { case named: NamedExpression => named }
--- End diff --
Please see my reply to the same question for `AggregateFieldExtractionPushdown`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148446149
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/JoinFieldExtractionPushdown.scala ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in a projection over a join
--- End diff --
Please describe this is specialized for pushing down pruned nested column.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16578
I will review this huge PR. : )
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
Hi Guys,
Can someone with super-Jenkins-powers please kick off a retest?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71343/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16578
Does this take over https://github.com/apache/spark/pull/14957? If so, we might need `Closes #14957` in the PR description for the merge script to close that one or let the author know this takes over that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1209/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84812/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r181575614
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ---
@@ -151,6 +151,9 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
// The following batch should be executed after batch "Join Reorder" and "LocalRelation".
Batch("Check Cartesian Products", Once,
CheckCartesianProducts) :+
+ Batch("Field Extraction Pushdown", fixedPoint,
+ AggregateFieldExtractionPushdown,
+ JoinFieldExtractionPushdown) :+
--- End diff --
Hi @gatorsmile.
Given the scope of your request, can I ask you to provide a reason for it? What you ask would invalidate some of the existing conversation and review of this PR. It would also substantially restrict the practical usability of this patch.
I believe I've written this patch with a logical separation of concerns along the lines you've requested. As a compromise, would you consider an incremental review starting with the basic projection/filter functionality and proceeding to the optimizer rules following them?
BTW I'm traveling for a few weeks, and I'm spending most of my time away from work. If I'm delayed in responding, that's the reason. I'll still keep up, but at a slower pace.
Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151474342
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
It needs to be set `true` for the tests. This can be done.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by vkhristenko <gi...@git.apache.org>.
Github user vkhristenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139312613
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
--- End diff --
@viirya @mallman
If I may add here, given a schema:
```
root
| - a: StructType
| | - f1: Int
| | - f2: Int
```
and a selection `df.select("a.f1", "a.f2")` vs `df.select("a.f2", "a.f1")`
will produce different requiredSchema fed into buildReader upon some action.
In the first case it will be `StructType( StructField( "a", StructType(StructField("f1") :: StructField("f2") :: Nil)) :: Nil)` and in the second `StructType( StructField( "a", StructType(StructField("f2") :: StructField("f1") :: Nil)) :: Nil)`
which means that the original schema is different from the one that is required for the second case. But as long as fields f1 and f2 are splitted (can be read without reading the other), it's remappable on the data source level.
VK
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148452275
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a nested field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
+ val dataSchemaFieldNames = dataSchema.fieldNames.toSet
+ val prunedDataSchema =
+ StructType(prunedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))
+
+ // If the data schema is different from the pruned data schema, continue. Otherwise,
+ // return [[op]]. We effect this comparison by counting the number of "leaf" fields in
+ // each schemata, assuming the fields in [[prunedDataSchema]] are a subset of the fields
+ // in [[dataSchema]].
+ if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
+ val prunedParquetRelation =
+ hadoopFsRelation.copy(dataSchema = prunedDataSchema)(hadoopFsRelation.sparkSession)
+
+ // We need to replace the expression ids of the pruned relation output attributes
+ // with the expression ids of the original relation output attributes so that
+ // references to the original relation's output are not broken
+ val outputIdMap = l.output.map(att => (att.name, att.exprId)).toMap
+ val prunedRelationOutput =
+ prunedParquetRelation
+ .schema
+ .toAttributes
+ .map {
+ case att if outputIdMap.contains(att.name) =>
+ att.withExprId(outputIdMap(att.name))
+ case att => att
+ }
+ val prunedRelation =
+ l.copy(relation = prunedParquetRelation, output = prunedRelationOutput,
+ catalogTable = None)
--- End diff --
Why remove original `catalogTable` if any?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/16578
made a simple test in a single-node spark environment
I used a synthetic dataset which is generated as: (that’s 20M)
```scala
import spark.implicits._
import org.apache.spark.{SparkContext, TaskContext}
case class Job(title: String, department: String)
case class Person(id: Int, name: String, job: Job)
(0 until 20000000).map(id => Person(id, id.toString, Job(id.toString, id.toString))).toDF.write.mode(SaveMode.Overwrite).parquet("/home/zhunan/parquet_test")
```
And then I read the directory and write to another place by
```scala
val df = spark.read.parquet("/home/zhunan/parquet_test")
df.select("job.title").write.mode(SaveMode.Overwrite).parquet("/home/zhunan/parquet_out")
```
without patch, it reads 169 MB, with patch, it will read around 86 MB.
Basically it proves that the PR is working
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139311391
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.types._
+
+/**
+ * A Scala extractor that builds a [[org.apache.spark.sql.types.StructField]] from a Catalyst
+ * complex type extractor. For example, consider a relation with the following schema:
+ *
+ * {{{
+ * root
+ * |-- name: struct (nullable = true)
+ * | |-- first: string (nullable = true)
+ * | |-- last: string (nullable = true)
+ * }}}
+ *
+ * Further, suppose we take the select expression `name.first`. This will parse into an
+ * `Alias(child, "first")`. Ignoring the alias, `child` matches the following pattern:
+ *
+ * {{{
+ * GetStructFieldObject(
+ * AttributeReference("name", StructType(_), _, _),
+ * StructField("first", StringType, _, _))
+ * }}}
+ *
+ * [[SelectedField]] converts that expression into
+ *
+ * {{{
+ * StructField("name", StructType(Array(StructField("first", StringType))))
+ * }}}
+ *
+ * by mapping each complex type extractor to a [[org.apache.spark.sql.types.StructField]] with the
+ * same name as its child (or "parent" going right to left in the select expression) and a data
+ * type appropriate to the complex type extractor. In our example, the name of the child expression
+ * is "name" and its data type is a [[org.apache.spark.sql.types.StructType]] with a single string
+ * field named "first".
+ *
+ * @param expr the top-level complex type extractor
+ */
+object SelectedField {
+ def unapply(expr: Expression): Option[StructField] = {
+ // If this expression is an alias, work on its child instead
+ val unaliased = expr match {
+ case Alias(child, _) => child
+ case expr => expr
+ }
+ selectField(unaliased, None)
+ }
+
+ private def selectField(expr: Expression, fieldOpt: Option[StructField]): Option[StructField] = {
+ expr match {
+ // No children. Returns a StructField with the attribute name or None if fieldOpt is None.
+ case AttributeReference(name, dataType, nullable, metadata) =>
+ fieldOpt.map(field =>
+ StructField(name, wrapStructType(dataType, field), nullable, metadata))
+ // Handles case "expr0.field[n]", where "expr0" is of struct type and "expr0.field" is of
+ // array type.
+ case GetArrayItem(x @ GetStructFieldObject(child, field @ StructField(name,
+ dataType, nullable, metadata)), _) =>
+ val childField = fieldOpt.map(field => StructField(name,
+ wrapStructType(dataType, field), nullable, metadata)).getOrElse(field)
+ selectField(child, Some(childField))
+ // Handles case "expr0.field[n]", where "expr0.field" is of array type.
+ case GetArrayItem(child, _) =>
+ selectField(child, fieldOpt)
+ // Handles case "expr0.field.subfield", where "expr0" and "expr0.field" are of array type.
+ case GetArrayStructFields(child: GetArrayStructFields,
+ field @ StructField(name, dataType, nullable, metadata), _, _, _) =>
+ val childField = fieldOpt.map(field => StructField(name,
+ wrapStructType(dataType, field),
+ nullable, metadata)).getOrElse(field)
+ selectField(child, Some(childField))
+ // Handles case "expr0.field", where "expr0" is of array type.
+ case GetArrayStructFields(child,
+ field @ StructField(name, dataType, nullable, metadata), _, _, containsNull) =>
+ val childField =
+ fieldOpt.map(field => StructField(name,
+ wrapStructType(dataType, field),
+ nullable, metadata)).getOrElse(field)
+ selectField(child, Some(childField)).map {
--- End diff --
line 97-107 can be simplified as `selectField(child, Some(childField))`, if line 121-126 becomes:
case GetStructFieldObject(child,
- field @ StructField(name, _, nullable, metadata)) =>
+ field @ StructField(name, dataType, nullable, metadata)) =>
val childField = fieldOpt.map(field => StructField(name,
- StructType(Array(field)),
+ wrapStructType(dataType, field),
nullable, metadata)).getOrElse(field)
selectField(child, Some(childField))
All tests in `SelectedFieldSuite` are passed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139337054
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdown.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in an aggregate's grouping and
+ * aggregate expressions into a projection over its children. The original
+ * [[expressions.GetStructField]] expressions are replaced with references to the pushed down
+ * aliases.
+ */
+object AggregateFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case agg @ Aggregate(groupingExpressions, aggregateExpressions, child) =>
+ val expressions = groupingExpressions ++ aggregateExpressions
+ val attributes = AttributeSet(expressions.collect { case att: Attribute => att })
+ val childAttributes = AttributeSet(child.expressions)
+ val fieldExtractors0 =
+ expressions
+ .flatMap(getFieldExtractors)
+ .distinct
+ val fieldExtractors1 =
+ fieldExtractors0
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(attributes.contains).isEmpty)
--- End diff --
Why do we need to trim the extractors which contain attributes referred from `groupingExpressions ++ aggregateExpressions`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148866084
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
--- End diff --
On the other hand, if we fix `parquetMrCompatibility` to `true`, then a couple of other tests fail. Namely, these tests are
"Filter applied on merged Parquet schema with new column should work" in `ParquetFilterSuite.scala`
"read partitioned table - merging compatible schemas" in `ParquetPartitionDiscoverySuite.scala`
In both cases, the failures involve queries over multiple files with compatible but different schema.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84404/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148435955
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdown.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in an aggregate's grouping and
+ * aggregate expressions into a projection over its children. The original
+ * [[expressions.GetStructField]] expressions are replaced with references to the pushed down
+ * aliases.
+ */
+object AggregateFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case agg @ Aggregate(groupingExpressions, aggregateExpressions, child) =>
+ val expressions = groupingExpressions ++ aggregateExpressions
+ val attributes = AttributeSet(expressions.collect { case att: Attribute => att })
+ val childAttributes = AttributeSet(child.expressions)
+ val fieldExtractors0 =
+ expressions
+ .flatMap(getFieldExtractors)
+ .distinct
+ val fieldExtractors1 =
+ fieldExtractors0
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(attributes.contains).isEmpty)
--- End diff --
If the query is:
```sql
select a.b, count(1) from r1 group by a.b
```
`fieldExtractors0` gets `GetStructField` `a.b`. Won't `fieldExtractors1` filter it out, because the attribute `a` is contained in the attribute set of all expressions? But we don't need all fields of `a` now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82381/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman I will try to go through this again. Do you think this can be generalize to data source v2 API?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> NPE is a problem though. any luck isolating that further?
I can push a commit that prevents it, I just can't say for certain it's a proper fix versus a mere workaround.
I'd like to push a unit test to trigger this bug, too, but I haven't written that yet. This is something that occurs only when codegen is turned on, so I need to figure out how to write a test that runs with and without codegen turned on. There's probably a test harness of sorts for this kind of test already.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #77814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77814/testReport)** for PR 16578 at commit [`6d748cd`](https://github.com/apache/spark/commit/6d748cd102f74c8d0affd62a0d27f8e8cf83ed35).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by saulshanabrook <gi...@git.apache.org>.
Github user saulshanabrook commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman Would it be helpful for me to rebase this off the head? I am interested in helping with this too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16578
I agree that this PR needs to be allocated more review bandwidth, and it is unfortunate that it has been blocked on that. However, I am -1 on merging a change this large after branch cut.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@saulshanabrook I'm sorry this effort has stalled. I actually have a lot of new code for this PR in my own clone, including *a lot* of new tests, but I haven't convinced myself that the code is actually correct and complete. I actually wrote a lot of new code to satisfy failing test cases that I found. I don't know that there aren't more test cases that would fail that I simply haven't uncovered.
I will take some time right now to re-examine my current code-to-date. If I push something tonight, it's likely to be something I have not established full confidence in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82383/testReport)** for PR 16578 at commit [`52fddc1`](https://github.com/apache/spark/commit/52fddc181f32726cea1dd12a23ebf7201986be01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@kamalptw Thanks for pinging Sean. I was going to do that myself soon. He's a helpful guy.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82097/testReport)** for PR 16578 at commit [`9fac482`](https://github.com/apache/spark/commit/9fac4826f6289e9e8c02f3799912463bbc8ea046).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #77814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77814/testReport)** for PR 16578 at commit [`6d748cd`](https://github.com/apache/spark/commit/6d748cd102f74c8d0affd62a0d27f8e8cf83ed35).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman code looks good, I can test out things with my dataset. Have deeply nested data, 20-30 nestings and millions of rows in a partition to test performance. Will do it Monday
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by abhaynahar <gi...@git.apache.org>.
Github user abhaynahar commented on the issue:
https://github.com/apache/spark/pull/16578
@viirya can you please help tag people you think should review ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87859/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71339/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
I'm still working actively on this PR (as I have time), but I wanted to share that I will be away and unavailable from tonight, March 24th until Tuesday, April 11th. If you post a comment to this PR during that timeframe I'll respond as soon as I can after I return.
Cheers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139333125
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
--- End diff --
We should add related tests for such cases.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98845200
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => field }
+ .map(field => StructType(Array(field)))
+ .reduceLeft(_ merge _)
+ val parquetDataColumnNames = dataSchema.fieldNames
+ val prunedDataSchema =
+ StructType(prunedSchema.filter(f => parquetDataColumnNames.contains(f.name)))
+ val parquetDataFields = dataSchema.fields.toSet
+ val prunedDataFields = prunedDataSchema.fields.toSet
+
+ // If the original Parquet relation data fields are different from the
+ // pruned data fields, continue. Otherwise, return [[op]]
+ if (parquetDataFields != prunedDataFields) {
+ val dataSchemaFieldNames = hadoopFsRelation.dataSchema.fieldNames
+ val newDataSchema =
+ StructType(prunedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))
--- End diff --
Is `newDataSchema` actually `prunedDataSchema`, if `dataSchemaFieldNames` is `parquetDataColumnNames`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148446321
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FieldExtractionPushdown.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, GetStructField}
+import org.apache.spark.sql.catalyst.planning.SelectedField
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+
+abstract class FieldExtractionPushdown extends Rule[LogicalPlan] {
--- End diff --
nit: Does this need to be a `Rule`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
So if it's not going to be included in `2.3.0` maybe we could change `spark.sql.nestedSchemaPruning.enabled` to default `true` ? I hope that this time this PR could be finalized at the early stage of `2.4.0` so there would be plenty of time to fix any unforseen problems?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by abhaynahar <gi...@git.apache.org>.
Github user abhaynahar commented on the issue:
https://github.com/apache/spark/pull/16578
sorry for spamming, but @rxin @marmbrus @ericl @cloud-fan @liancheng can you please help taking this forward ? @viirya has reviewed it closely and is looking for someone else to review this as well. This patch has a lot of speed improvements and we are hoping it can make it in 2.3 release.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> To ensure the PR and review quality, we normally avoid doing everything in a single huge PR. It would be much better if you can cut it to a few smaller PRs.
I'll have a go at it. Of course this will rewrite most of the commits, but I assume you don't mind that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by zaycev <gi...@git.apache.org>.
Github user zaycev commented on the issue:
https://github.com/apache/spark/pull/16578
I observed about 5x better performance in reading a small subset of fields of a highly nested parquet table:
master:
<img width="1121" alt="screen shot 2018-03-02 at 1 59 39 pm" src="https://user-images.githubusercontent.com/283938/36928047-e07e5b52-1e36-11e8-98e4-a614ad7589b6.png">
<img width="403" alt="screen shot 2018-03-02 at 1 59 19 pm" src="https://user-images.githubusercontent.com/283938/36928033-c9a21022-1e36-11e8-81bf-7008e1f40d6f.png">
master with @mallman patch:
<img width="1033" alt="screen shot 2018-03-02 at 2 58 42 pm" src="https://user-images.githubusercontent.com/283938/36928037-cdc9ec10-1e36-11e8-8830-5e77c074e4ab.png">
<img width="388" alt="screen shot 2018-03-02 at 2 59 09 pm" src="https://user-images.githubusercontent.com/283938/36928048-e3e15a88-1e36-11e8-8dda-9b384c4a04c8.png">
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
BTW I’ve been and am currently traveling with a busy itinerary. I haven’t started work on this and probably won’t get to work on it until Monday at the very earliest.
> On May 5, 2018, at 8:32 AM, Xiao Li <no...@github.com> wrote:
>
> Yeah. That is fine. Will try to review the relevant PRs ASAP. Please ping me. Thanks again!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2636/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77326/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82008 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82008/testReport)** for PR 16578 at commit [`a267701`](https://github.com/apache/spark/commit/a26770189fb2ba07d76e4e9d10c3fd47b26a9d8d).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82028/testReport)** for PR 16578 at commit [`3fe0e53`](https://github.com/apache/spark/commit/3fe0e5335c6cf2a7ef767b3e2971e1dc91f6860b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r100360523
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
Let's go with `GetStructFieldObject`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148863673
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
--- End diff --
As for the problem of requesting a read of a superset of a file's fields, if we disable the `parquetMrCompatibility` code, then the "partial schema intersection - select missing subfield" test in `ParquetSchemaPruningSuite.scala` fails with the following stack trace:
```
[info] Cause: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:/Volumes/VideoAmpCS/msa/workspace/spark-public/target/tmp/spark-a0bda193-9d3f-4cd1-885c-9e8b5b0fc1ed/contacts/p=2/part-00001-4a8671f1-afb2-482f-8c4d-4f6f4df896bc-c000.snappy.parquet
[info] at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:223)
[info] at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:215)
[info] at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
[info] at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:106)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:182)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:106)
[info] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
[info] at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[info] at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$7$$anon$1.hasNext(WholeStageCodegenExec.scala:432)
[info] at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
[info] at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
[info] at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1820)
[info] at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1160)
[info] at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1160)
[info] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
[info] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
[info] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
[info] at org.apache.spark.scheduler.Task.run(Task.scala:108)
[info] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] at java.lang.Thread.run(Thread.java:748)
[info] Cause: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
[info] at java.util.ArrayList.rangeCheck(ArrayList.java:653)
[info] at java.util.ArrayList.get(ArrayList.java:429)
[info] at org.apache.parquet.io.GroupColumnIO.getFirst(GroupColumnIO.java:103)
[info] at org.apache.parquet.io.GroupColumnIO.getFirst(GroupColumnIO.java:103)
[info] at org.apache.parquet.io.PrimitiveColumnIO.getFirst(PrimitiveColumnIO.java:102)
[info] at org.apache.parquet.io.PrimitiveColumnIO.isFirst(PrimitiveColumnIO.java:97)
[info] at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:278)
[info] at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:141)
[info] at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:107)
[info] at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:155)
[info] at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:107)
[info] at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:136)
[info] at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:194)
[info] at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:215)
[info] at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
[info] at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:106)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:182)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:106)
[info] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
[info] at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[info] at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$7$$anon$1.hasNext(WholeStageCodegenExec.scala:432)
[info] at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
[info] at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
[info] at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1820)
[info] at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1160)
[info] at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1160)
[info] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
[info] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
[info] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
[info] at org.apache.spark.scheduler.Task.run(Task.scala:108)
[info] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] at java.lang.Thread.run(Thread.java:748)
```
As you can see, the problem comes when attempting the read. The problem is deep within the logic of the Parquet record reader code and is not something I want to dig into.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #84404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84404/testReport)** for PR 16578 at commit [`981e53b`](https://github.com/apache/spark/commit/981e53bf6b4b23f790ff0bbd457f54f308441076).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79815/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by sriramrajendiran <gi...@git.apache.org>.
Github user sriramrajendiran commented on the issue:
https://github.com/apache/spark/pull/16578
@felixcheung can you help ? we are hoping to see it in 2.3 release. Feature underneath a default disabled flag looks safe option.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151597063
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
As I know, seems we don't have a config setting specified for all tests previously. We have set a config for a whole test suite, but not for all tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148434529
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/ProjectionOverSchema.scala ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.types._
+
+/**
+ * A Scala extractor that projects an expression over a given schema. Data types,
+ * field indexes and array lengths of complex type extractors and attributes
+ * are adjusted to fit the schema. All other expressions are left as-is.
+ */
+case class ProjectionOverSchema(schema: StructType) {
--- End diff --
Can you describe clearly in comment that this is used in nested column pruning?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #72836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72836/testReport)** for PR 16578 at commit [`948b584`](https://github.com/apache/spark/commit/948b5848526960495f8f71ebbd348f8b8ba3c197).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #77794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77794/testReport)** for PR 16578 at commit [`464b18d`](https://github.com/apache/spark/commit/464b18d61b578ee5621b7a22f41955dd0f8b60a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by snir <gi...@git.apache.org>.
Github user snir commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman Yes, the issue is only reproducible with codegen on. However the performance improvement by codegen is too significant to disable. I managed to fix it with some changes to the codegen (Added a null check in `GetArrayStructFields.doGenCode`) but it feels like a workaround. Do you have an idea of what could cause it in the patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82051/testReport)** for PR 16578 at commit [`00ab80c`](https://github.com/apache/spark/commit/00ab80c9b78c45c1a8f8c202c5bab04a62cda2ef).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #79534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79534/testReport)** for PR 16578 at commit [`9066fb3`](https://github.com/apache/spark/commit/9066fb35cd6c153dcf4ed5529b404ffb73a0deed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by amankothari04 <gi...@git.apache.org>.
Github user amankothari04 commented on the issue:
https://github.com/apache/spark/pull/16578
@viirya did you get a chance to review this ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82051/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> @mallman (just pure curiosity :) ) how is it possible that this NPE was not found by automated tests? There's no single test case in entire spark suite which verifies scenarios like that described by @snir ?
A good question. This probably calls for a new unit test...
Presently I'm not feeling too motivated to invest more time in this PR unless and until a reviewer shows some interest in it. The last time I received any feedback from a reviewer was seven months ago. Not a word since. I'm surprised, as my past experience submitting PRs has been nothing like this.
Let's try @hvanhovell. He also works on Spark SQL. Herman, any position on this? Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by bitcot <gi...@git.apache.org>.
Github user bitcot commented on the issue:
https://github.com/apache/spark/pull/16578
Thanks @mallman this is very helpful.
@felixcheung @rxin can you please help to take this forward ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148456140
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/FileSchemaPruningTest.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.scalactic.Equality
+import org.scalatest.Assertions
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.types.StructType
+
+private[sql] trait FileSchemaPruningTest {
+ _: Assertions =>
+
+ private val schemaEquality = new Equality[StructType] {
+ override def areEqual(a: StructType, b: Any) =
+ b match {
+ case otherType: StructType => a sameType otherType
+ case _ => false
+ }
+ }
+
+ protected def checkScanSchemata(df: DataFrame, expectedSchemaCatalogStrings: String*): Unit = {
--- End diff --
`checkScanSchemata` -> `checkScanSchema`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r149293915
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a nested field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
+ val dataSchemaFieldNames = dataSchema.fieldNames.toSet
+ val prunedDataSchema =
+ StructType(prunedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))
+
+ // If the data schema is different from the pruned data schema, continue. Otherwise,
+ // return [[op]]. We effect this comparison by counting the number of "leaf" fields in
+ // each schemata, assuming the fields in [[prunedDataSchema]] are a subset of the fields
+ // in [[dataSchema]].
+ if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
+ val prunedParquetRelation =
+ hadoopFsRelation.copy(dataSchema = prunedDataSchema)(hadoopFsRelation.sparkSession)
+
+ // We need to replace the expression ids of the pruned relation output attributes
+ // with the expression ids of the original relation output attributes so that
+ // references to the original relation's output are not broken
+ val outputIdMap = l.output.map(att => (att.name, att.exprId)).toMap
+ val prunedRelationOutput =
+ prunedParquetRelation
+ .schema
+ .toAttributes
+ .map {
+ case att if outputIdMap.contains(att.name) =>
+ att.withExprId(outputIdMap(att.name))
+ case att => att
+ }
+ val prunedRelation =
+ l.copy(relation = prunedParquetRelation, output = prunedRelationOutput,
+ catalogTable = None)
--- End diff --
Because `catalogTable` can contain some data such as statistics, that might be useful. Not sure if we may lose some optimization chance.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98843749
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.types._
+
+/**
+ * A Scala extractor that builds a [[StructField]] from a Catalyst complex type
+ * extractor. This is like the opposite of [[ExtractValue#apply]].
+ */
+object SelectedField {
+ def unapply(expr: Expression): Option[StructField] = {
+ // If this expression is an alias, work on its child instead
+ val unaliased = expr match {
+ case Alias(child, _) => child
+ case expr => expr
+ }
+ selectField(unaliased, None)
+ }
+
+ /**
+ * Converts some chain of complex type extractors into a [[StructField]].
--- End diff --
It is better to add few example input and output in the comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman do you foresee any issues ? planning to backport it to spark 2.2 on personal fork. will probably make jitpack release
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98920657
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
Agreed. I think the best name in this context is `GetStructField`, but that's already taken. I'll keep thinking about a good alternative.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #71343 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71343/testReport)** for PR 16578 at commit [`efc4d76`](https://github.com/apache/spark/commit/efc4d7662c98a5ee8f719b5839ca389a2a987d97).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r100229358
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
How about `GetStructFieldObject`? Or `GetStructFieldRef`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r140226905
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a proper field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
--- End diff --
> Can the merging order of the fields affect the comparison later in line 57?
Not the merging order, no. However the selection order can, leading to unnecessary "pruning". I will address this issue and add a test case to verify it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82028/testReport)** for PR 16578 at commit [`3fe0e53`](https://github.com/apache/spark/commit/3fe0e5335c6cf2a7ef767b3e2971e1dc91f6860b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82045/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@viirya I've added a commit to address some of your feedback. I will have another commit to address the others, but I'm not sure when I'll have it in. Hopefully by the end of next week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148731634
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/planning/SelectedFieldSuite.scala ---
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.scalatest.BeforeAndAfterAll
+import org.scalatest.exceptions.TestFailedException
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.NamedExpression
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.types._
+
+// scalastyle:off line.size.limit
+class SelectedFieldSuite extends SparkFunSuite with BeforeAndAfterAll {
+ // The test schema as a tree string, i.e. `schema.treeString`
+ // root
+ // |-- col1: string (nullable = false)
+ // |-- col2: struct (nullable = true)
+ // | |-- field1: integer (nullable = true)
+ // | |-- field2: array (nullable = true)
+ // | | |-- element: integer (containsNull = false)
+ // | |-- field3: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | | |-- subfield3: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = true)
+ // | |-- field4: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = false)
+ // | |-- field5: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: struct (nullable = false)
+ // | | | | |-- subsubfield1: integer (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | | | |-- subfield2: struct (nullable = true)
+ // | | | | |-- subsubfield1: struct (nullable = true)
+ // | | | | | |-- subsubsubfield1: string (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field6: struct (nullable = true)
+ // | | |-- subfield1: string (nullable = false)
+ // | | |-- subfield2: string (nullable = true)
+ // | |-- field7: struct (nullable = true)
+ // | | |-- subfield1: struct (nullable = true)
+ // | | | |-- subsubfield1: integer (nullable = true)
+ // | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field8: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: array (valueContainsNull = false)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: array (nullable = true)
+ // | | | | | |-- element: integer (containsNull = false)
+ // | |-- field9: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: integer (valueContainsNull = false)
+ // |-- col3: array (nullable = false)
+ // | |-- element: struct (containsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col4: map (nullable = false)
+ // | |-- key: string
+ // | |-- value: struct (valueContainsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col5: array (nullable = true)
+ // | |-- element: map (containsNull = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col6: map (nullable = true)
+ // | |-- key: string
+ // | |-- value: array (valueContainsNull = true)
+ // | | |-- element: struct (containsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col7: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: integer (nullable = false)
+ // | | |-- field2: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | |-- field3: array (nullable = true)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = false)
+ // |-- col8: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: array (nullable = false)
+ // | | | |-- element: integer (containsNull = false)
+ private val schema =
+ StructType(
+ StructField("col1", StringType, nullable = false) ::
+ StructField("col2", StructType(
+ StructField("field1", IntegerType) ::
+ StructField("field2", ArrayType(IntegerType, containsNull = false)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) ::
+ StructField("subfield3", ArrayType(IntegerType)) :: Nil)), nullable = false) ::
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil), valueContainsNull = false)) ::
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil), nullable = false) ::
+ StructField("subfield2", StructType(
+ StructField("subsubfield1", StructType(
+ StructField("subsubsubfield1", StringType) :: Nil)) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)), nullable = false) ::
+ StructField("field6", StructType(
+ StructField("subfield1", StringType, nullable = false) ::
+ StructField("subfield2", StringType) :: Nil)) ::
+ StructField("field7", StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)) ::
+ StructField("field8", MapType(StringType, ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil)), valueContainsNull = false)) ::
+ StructField("field9", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil)) ::
+ StructField("col3", ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), containsNull = false), nullable = false) ::
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), valueContainsNull = false), nullable = false) ::
+ StructField("col5", ArrayType(MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), valueContainsNull = false))) ::
+ StructField("col6", MapType(StringType, ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), containsNull = false))) ::
+ StructField("col7", ArrayType(StructType(
+ StructField("field1", IntegerType, nullable = false) ::
+ StructField("field2", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil))) :: Nil))) ::
+ StructField("col8", ArrayType(StructType(
+ StructField("field1", ArrayType(IntegerType, containsNull = false), nullable = false) :: Nil))) :: Nil)
+
+ private val testRelation = LocalRelation(schema.toAttributes)
+
+ override def beforeAll() {
+ val indentedSchema = schema.treeString.split('\n').map(x => " " + x).mkString("\n") + "\n"
+ // Print the test schema at the beginning of the test
+ // scalastyle:off println
+ println("Testing SelectedField extractor using the following schema:\n" + indentedSchema)
+ // scalastyle:on println
--- End diff --
Yes, I suppose it's enough to embed the tree string as a comment above, which was basically a copy-n-paste of the `println`. I'll get rid of this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82047/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82382/testReport)** for PR 16578 at commit [`be1a257`](https://github.com/apache/spark/commit/be1a257576df76803524a944ff0e26a873d80b30).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
cc @rxin @ericl @cloud-fan
@marmbrus I would love to get your feedback on this if you have the time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16578
Retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148446396
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/JoinFieldExtractionPushdown.scala ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in a projection over a join
+ * and its join condition. The original [[expressions.GetStructField]] expressions are replaced
+ * with references to the pushed down aliases.
+ */
+object JoinFieldExtractionPushdown extends FieldExtractionPushdown {
--- End diff --
Oh, I just meant we may skip most of the rule if it can't be applied. Please see my comment below.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman @viirya from my understanding current workaround is for case when reading columns which are not in file schema
> Parquet-mr will throw an exception if we try to read a superset of the file's schema.
Isn't it somehow dependent on schema evolution setting? http://spark.apache.org/docs/latest/sql-programming-guide.html#schema-merging
> Since schema merging is a relatively expensive operation, and is not a necessity in most cases, we turned it off by default starting from 1.5.0. You may enable it by
> * setting data source option mergeSchema to true when reading Parquet files (as shown in the examples below), or
> * setting the global SQL option spark.sql.parquet.mergeSchema to true.
Wouldn't it work fine with `spark.sql.parquet.mergeSchema` enabled?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82381/testReport)** for PR 16578 at commit [`88786d3`](https://github.com/apache/spark/commit/88786d3eaf9d3a3d2c80c2235c5014074ade3dc1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79534/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #79536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79536/testReport)** for PR 16578 at commit [`81cca70`](https://github.com/apache/spark/commit/81cca702dbe0c1f8a5f9dab30406cac8d5e76362).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@Gauravshah That's very encouraging to see.
I suspect this patch will require some pretty heavy scrutiny. Hopefully it will make it into 2.3. If not, I hope that interested users will be able to port the patch to their build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r180495402
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ---
@@ -151,6 +151,9 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
// The following batch should be executed after batch "Join Reorder" and "LocalRelation".
Batch("Check Cartesian Products", Once,
CheckCartesianProducts) :+
+ Batch("Field Extraction Pushdown", fixedPoint,
+ AggregateFieldExtractionPushdown,
+ JoinFieldExtractionPushdown) :+
--- End diff --
@mallman Could you split these new optimizer rules to two PRs first?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
@marmbrus can we start the review process ? so that it can make it for the next release ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r149880006
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
nit: are we confident enough to set it as true by default?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16578
As I mentioned before, we still don't have enough eyes on this change so far.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #77326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77326/testReport)** for PR 16578 at commit [`9f2f340`](https://github.com/apache/spark/commit/9f2f3409172ba09d15494f9faf861bb6ad683911).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98809992
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
Or combine it with `GetStructField` case class extractor?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r142119164
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala ---
@@ -77,20 +77,21 @@ trait QueryPlanConstraints { self: LogicalPlan =>
constraint match {
// When the root is IsNotNull, we can push IsNotNull through the child null intolerant
// expressions
- case IsNotNull(expr) => scanNullIntolerantAttribute(expr).map(IsNotNull(_))
+ case IsNotNull(expr) => scanNullIntolerantField(expr).map(IsNotNull(_))
// Constraints always return true for all the inputs. That means, null will never be returned.
// Thus, we can infer `IsNotNull(constraint)`, and also push IsNotNull through the child
// null intolerant expressions.
- case _ => scanNullIntolerantAttribute(constraint).map(IsNotNull(_))
+ case _ => scanNullIntolerantField(constraint).map(IsNotNull(_))
--- End diff --
> I don't believe this is covered in the current unit tests, so I will add or modify a test to cover it.
I added the following test to cover this case:
https://github.com/apache/spark/blob/88786d3eaf9d3a3d2c80c2235c5014074ade3dc1/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala#L62-L79
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151477529
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
Ah. Sounds reasonable. Do you know how to do that? Is there a precedent I can follow? I'm not aware of one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
I just pushed a revision of `SelectedField.scala`. Let's see what Jenkins says. I expect it to pass, and assuming it does I will return the ball to the reviewers' court.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> * There is a minor change needed though, "parquetFormat: ParquetFileFormat" should be replaced by "fileFormat: FileFormat" as there is no dependency on the actual ParquetFileFormat class defined in parquet package
https://github.com/apache/spark/pull/16578/files?diff=unified#diff-3bad814b3336a83f360d7395bd740759R38
This is true. I hesitate to weaken the match to all instances of `FileFormat` because the only format I have extensive experience and knowledge of is Parquet. There are other formats that could expand upon this work, such as ORC and ROOT, but I have no practical experience working with those. I'd prefer someone who does have such experience build on this PR to make it work with those file formats.
Incidentally, I added a `ColumnarFileFormat` trait in this PR. You might consider it a marker for columnar file formats, but all it really does right now is compute the number of physical columns read for a given catalyst schema. This value is used in the description of a physical plan to help a dev/user ensure that their expected column pruning is in fact occurring.
> * And may be renaming this ParquetSchemaPruning and taking it outside of the parquet package as it is quite more general than just for parquet, otherwise I have to add a special Rule here, https://github.com/apache/spark/pull/16578/files?diff=unified#diff-2370d8ed85930c93ef8e5ce67abca53fR35 ???
Moving and renaming `ParquetSchemaPruning` makes sense if it's generalized to other file formats.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
@rxin Any thoughts on a review of this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148722822
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala ---
@@ -127,8 +127,8 @@ private[parquet] class ParquetRowConverter(
extends ParquetGroupConverter(updater) with Logging {
assert(
- parquetType.getFieldCount == catalystType.length,
- s"""Field counts of the Parquet schema and the Catalyst schema don't match:
+ parquetType.getFieldCount <= catalystType.length,
--- End diff --
In `ParquetReadSupport.scala`, when `parquetMrCompatibility` is `true`, we intersect the clipped parquet schema with the underlying parquet file's schema. This can result in a requested parquet schema with fewer fields than the requested catalyst schema.
For example, in the case of a partitioned table where we select a column which doesn't exist in the schema of one partition's files, we will remove the missing columns from the requested parquet schema.
This scenario is illustrated and tested by the "partial schema intersection - select missing subfield" test in `ParquetSchemaPruningSuite.scala`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #83532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83532/testReport)** for PR 16578 at commit [`48a509e`](https://github.com/apache/spark/commit/48a509e8602ed44a4a0fd5268d91d917bb8e0748).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16578
yes!
sorry about the delay, I think there's a lot of interests in this PR.
@gatorsmile @viirya ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #83387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83387/testReport)** for PR 16578 at commit [`71c1762`](https://github.com/apache/spark/commit/71c17622261c69503c2a2bd80c769bd664df3d9d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151917066
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_SCHEMA_PRUNING_ENABLED =
+ buildConf("spark.sql.nestedSchemaPruning.enabled")
+ .internal()
+ .doc("Prune nested fields from a logical relation's output which are unnecessary in " +
+ "satisfying a query. This optimization allows columnar file format readers to avoid " +
+ "reading unnecessary nested column data.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
So maybe at least make it true for some core sql/parquet test suits?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman not sure how can I help to push it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r98833772
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GetStructField}
+import org.apache.spark.sql.types.StructField
+
+/**
+ * A Scala extractor that extracts the child expression and struct field from a [[GetStructField]].
+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {
--- End diff --
But we can have a better name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #71339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71339/testReport)** for PR 16578 at commit [`8925372`](https://github.com/apache/spark/commit/89253722bde65b02467d05f2b64c4fdbef3b885d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class FieldExtractionPushdown extends Rule[LogicalPlan] `
* `case class ProjectionOverSchema(schema: StructType) `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by ianoc <gi...@git.apache.org>.
Github user ianoc commented on the issue:
https://github.com/apache/spark/pull/16578
Given it has one or two deep review's already, can someone just rubber stamp this in a bias to shipping? Its been stalled more or less since July waiting on reviewers.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
Dammit. I forgot to commit a couple of build fixes. Fixed commit coming momentarily...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148723190
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ProjectionOverSchema, SelectedField}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+/**
+ * Prunes unnecessary Parquet columns given a [[PhysicalOperation]] over a
+ * [[ParquetRelation]]. By "Parquet column", we mean a column as defined in the
+ * Parquet format. In Spark SQL, a root-level Parquet column corresponds to a
+ * SQL column, and a nested Parquet column corresponds to a [[StructField]].
+ */
+private[sql] object ParquetSchemaPruning extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ l @ LogicalRelation(hadoopFsRelation @ HadoopFsRelation(_, partitionSchema,
+ dataSchema, _, parquetFormat: ParquetFileFormat, _), _, _, _)) =>
+ val projectionFields = projects.flatMap(getFields)
+ val filterFields = filters.flatMap(getFields)
+ val requestedFields = (projectionFields ++ filterFields).distinct
+
+ // If [[requestedFields]] includes a nested field, continue. Otherwise,
+ // return [[op]]
+ if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+ val prunedSchema = requestedFields
+ .map { case (field, _) => StructType(Array(field)) }
+ .reduceLeft(_ merge _)
+ val dataSchemaFieldNames = dataSchema.fieldNames.toSet
+ val prunedDataSchema =
+ StructType(prunedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))
+
+ // If the data schema is different from the pruned data schema, continue. Otherwise,
+ // return [[op]]. We effect this comparison by counting the number of "leaf" fields in
+ // each schemata, assuming the fields in [[prunedDataSchema]] are a subset of the fields
+ // in [[dataSchema]].
+ if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
+ val prunedParquetRelation =
+ hadoopFsRelation.copy(dataSchema = prunedDataSchema)(hadoopFsRelation.sparkSession)
+
+ // We need to replace the expression ids of the pruned relation output attributes
+ // with the expression ids of the original relation output attributes so that
+ // references to the original relation's output are not broken
+ val outputIdMap = l.output.map(att => (att.name, att.exprId)).toMap
+ val prunedRelationOutput =
+ prunedParquetRelation
+ .schema
+ .toAttributes
+ .map {
+ case att if outputIdMap.contains(att.name) =>
+ att.withExprId(outputIdMap(att.name))
+ case att => att
+ }
+ val prunedRelation =
+ l.copy(relation = prunedParquetRelation, output = prunedRelationOutput,
+ catalogTable = None)
--- End diff --
I just made a judgment call here based on the idea that by modifying the logical relation it's no longer the same as the catalog table. What do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148455060
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/planning/SelectedFieldSuite.scala ---
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.scalatest.BeforeAndAfterAll
+import org.scalatest.exceptions.TestFailedException
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.NamedExpression
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.types._
+
+// scalastyle:off line.size.limit
+class SelectedFieldSuite extends SparkFunSuite with BeforeAndAfterAll {
+ // The test schema as a tree string, i.e. `schema.treeString`
+ // root
+ // |-- col1: string (nullable = false)
+ // |-- col2: struct (nullable = true)
+ // | |-- field1: integer (nullable = true)
+ // | |-- field2: array (nullable = true)
+ // | | |-- element: integer (containsNull = false)
+ // | |-- field3: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | | |-- subfield3: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = true)
+ // | |-- field4: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = false)
+ // | |-- field5: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: struct (nullable = false)
+ // | | | | |-- subsubfield1: integer (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | | | |-- subfield2: struct (nullable = true)
+ // | | | | |-- subsubfield1: struct (nullable = true)
+ // | | | | | |-- subsubsubfield1: string (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field6: struct (nullable = true)
+ // | | |-- subfield1: string (nullable = false)
+ // | | |-- subfield2: string (nullable = true)
+ // | |-- field7: struct (nullable = true)
+ // | | |-- subfield1: struct (nullable = true)
+ // | | | |-- subsubfield1: integer (nullable = true)
+ // | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field8: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: array (valueContainsNull = false)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: array (nullable = true)
+ // | | | | | |-- element: integer (containsNull = false)
+ // | |-- field9: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: integer (valueContainsNull = false)
+ // |-- col3: array (nullable = false)
+ // | |-- element: struct (containsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col4: map (nullable = false)
+ // | |-- key: string
+ // | |-- value: struct (valueContainsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col5: array (nullable = true)
+ // | |-- element: map (containsNull = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col6: map (nullable = true)
+ // | |-- key: string
+ // | |-- value: array (valueContainsNull = true)
+ // | | |-- element: struct (containsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col7: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: integer (nullable = false)
+ // | | |-- field2: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | |-- field3: array (nullable = true)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = false)
+ // |-- col8: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: array (nullable = false)
+ // | | | |-- element: integer (containsNull = false)
+ private val schema =
+ StructType(
+ StructField("col1", StringType, nullable = false) ::
+ StructField("col2", StructType(
+ StructField("field1", IntegerType) ::
+ StructField("field2", ArrayType(IntegerType, containsNull = false)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) ::
+ StructField("subfield3", ArrayType(IntegerType)) :: Nil)), nullable = false) ::
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil), valueContainsNull = false)) ::
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil), nullable = false) ::
+ StructField("subfield2", StructType(
+ StructField("subsubfield1", StructType(
+ StructField("subsubsubfield1", StringType) :: Nil)) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)), nullable = false) ::
+ StructField("field6", StructType(
+ StructField("subfield1", StringType, nullable = false) ::
+ StructField("subfield2", StringType) :: Nil)) ::
+ StructField("field7", StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)) ::
+ StructField("field8", MapType(StringType, ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil)), valueContainsNull = false)) ::
+ StructField("field9", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil)) ::
+ StructField("col3", ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), containsNull = false), nullable = false) ::
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), valueContainsNull = false), nullable = false) ::
+ StructField("col5", ArrayType(MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), valueContainsNull = false))) ::
+ StructField("col6", MapType(StringType, ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), containsNull = false))) ::
+ StructField("col7", ArrayType(StructType(
+ StructField("field1", IntegerType, nullable = false) ::
+ StructField("field2", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil))) :: Nil))) ::
+ StructField("col8", ArrayType(StructType(
+ StructField("field1", ArrayType(IntegerType, containsNull = false), nullable = false) :: Nil))) :: Nil)
+
+ private val testRelation = LocalRelation(schema.toAttributes)
+
+ override def beforeAll() {
+ val indentedSchema = schema.treeString.split('\n').map(x => " " + x).mkString("\n") + "\n"
+ // Print the test schema at the beginning of the test
+ // scalastyle:off println
+ println("Testing SelectedField extractor using the following schema:\n" + indentedSchema)
+ // scalastyle:on println
+ }
+
+ test("should not match an attribute reference") {
+ assertResult(None)(unapplySelect("col1"))
+ assertResult(None)(unapplySelect("col1 as foo"))
+ assertResult(None)(unapplySelect("col2"))
+ }
+
+ test("col2.field2, col2.field2[0] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field2", ArrayType(IntegerType, containsNull = false)) :: Nil))
+ testSelect("col2.field2", expected)
+ testSelect("col2.field2[0] as foo", expected)
+ }
+
+ test("col2.field9, col2.field9['foo'] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field9", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil))
+ testSelect("col2.field9", expected)
+ testSelect("col2.field9['foo'] as foo", expected)
+ }
+
+ test("col2.field3.subfield3, col2.field3[0].subfield3 as foo, col2.field3.subfield3[0] as foo, col2.field3[0].subfield3[0] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield3", ArrayType(IntegerType)) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field3.subfield3", expected)
+ testSelect("col2.field3[0].subfield3 as foo", expected)
+ testSelect("col2.field3.subfield3[0] as foo", expected)
+ testSelect("col2.field3[0].subfield3[0] as foo", expected)
+ }
+
+ test("col2.field3.subfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field3.subfield1", expected)
+ }
+
+ test("col2.field5.subfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil), nullable = false) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field5.subfield1", expected)
+ }
+
+ test("col3.field1.subfield1") {
+ val expected =
+ StructField("col3", ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) :: Nil), containsNull = false), nullable = false)
+ testSelect("col3.field1.subfield1", expected)
+ }
+
+ test("col3.field2['foo'] as foo") {
+ val expected =
+ StructField("col3", ArrayType(StructType(
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), containsNull = false), nullable = false)
+ testSelect("col3.field2['foo'] as foo", expected)
+ }
+
+ test("col4['foo'].field1.subfield1 as foo") {
+ val expected =
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) :: Nil), valueContainsNull = false), nullable = false)
+ testSelect("col4['foo'].field1.subfield1 as foo", expected)
+ }
+
+ test("col4['foo'].field2['bar'] as foo") {
+ val expected =
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), valueContainsNull = false), nullable = false)
+ testSelect("col4['foo'].field2['bar'] as foo", expected)
+ }
+
+ test("col5[0]['foo'].field1.subfield1 as foo") {
+ val expected =
+ StructField("col5", ArrayType(MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) :: Nil)) :: Nil), valueContainsNull = false)))
+ testSelect("col5[0]['foo'].field1.subfield1 as foo", expected)
+ }
+
+ test("col6['foo'][0].field1.subfield1 as foo") {
+ val expected =
+ StructField("col6", MapType(StringType, ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) :: Nil)) :: Nil), containsNull = false)))
+ testSelect("col6['foo'][0].field1.subfield1 as foo", expected)
+ }
+
+ test("col2.field5.subfield1.subsubfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) :: Nil), nullable = false) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field5.subfield1.subsubfield1", expected)
+ }
+
+ test("col2.field5.subfield2.subsubfield1.subsubsubfield1") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield2", StructType(
+ StructField("subsubfield1", StructType(
+ StructField("subsubsubfield1", StringType) :: Nil)) :: Nil)) :: Nil)), nullable = false) :: Nil))
+ testSelect("col2.field5.subfield2.subsubfield1.subsubsubfield1", expected)
+ }
+
+ test("col2.field4['foo'].subfield1 as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield1", IntegerType) :: Nil), valueContainsNull = false)) :: Nil))
+ testSelect("col2.field4['foo'].subfield1 as foo", expected)
+ }
+
+ test("col2.field4['foo'].subfield2 as foo, col2.field4['foo'].subfield2[0] as foo") {
+ val expected =
+ StructField("col2", StructType(
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil), valueContainsNull = false)) :: Nil))
--- End diff --
I feel those expected schema are hard to read. Can we have a helper function that returns part of the `schema` based on what fields requested?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #82009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82009/testReport)** for PR 16578 at commit [`5adac6b`](https://github.com/apache/spark/commit/5adac6b789823cd6c33dcfc291e9f33dccc04a50).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> Oh crap. I know what happened here. I've been updating some of the unit tests locally, and I've been running changes against these modified tests. I'll fix this and push a commit.
This may mean parquet-mr 1.8.2 does not fix the problem I was having with version 1.7.0. If so I'll likely have to bring back that ugly hack in `ParquetReadSupport.scala`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
We'll need a review from a committer. @cloud-fan @ericl do you have time to review this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89794/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #83128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83128/testReport)** for PR 16578 at commit [`7af925b`](https://github.com/apache/spark/commit/7af925bcf44861119ec305ab9631a3511d8e8bbb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83390/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> But I think we still need other eyes on this too.
Agreed.
@rxin can you help rope anyone else in on this? It's a big PR with a bigger history, but absent some savaging by another reviewer I believe it is close to the finish line.
A lot of people are hoping this can make it into Spark 2.3. It has a huge performance impact for some Spark users, as evidenced by comments on this PR and VideoAmp's own experience. So I'm hoping we can get this merged to master before the 2.3 branch is cut.
Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139312657
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala ---
@@ -77,20 +77,21 @@ trait QueryPlanConstraints { self: LogicalPlan =>
constraint match {
// When the root is IsNotNull, we can push IsNotNull through the child null intolerant
// expressions
- case IsNotNull(expr) => scanNullIntolerantAttribute(expr).map(IsNotNull(_))
+ case IsNotNull(expr) => scanNullIntolerantField(expr).map(IsNotNull(_))
// Constraints always return true for all the inputs. That means, null will never be returned.
// Thus, we can infer `IsNotNull(constraint)`, and also push IsNotNull through the child
// null intolerant expressions.
- case _ => scanNullIntolerantAttribute(constraint).map(IsNotNull(_))
+ case _ => scanNullIntolerantField(constraint).map(IsNotNull(_))
--- End diff --
Previously IsNotNull constraints are Attribute-specific, why do we need to expand it to `ExtractValue`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> Does this take over #14957? If so, we might need Closes #14957 in the PR description for the merge script to close that one or let the author know this takes over that.
I don't know. @xuanyuanking, how do you feel about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #79534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79534/testReport)** for PR 16578 at commit [`9066fb3`](https://github.com/apache/spark/commit/9066fb35cd6c153dcf4ed5529b404ffb73a0deed).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16578
I only looked at the PR description, here are my 2 cents:
Currently column pruning is done with 2 steps in Spark: 1) optimizer generates extra `Project` to prune unnecessary columns as bottom as possible, to reduce the data size between operators. 2) planner extract required columns and push it to data sources.
The first step is generally useful even if the data source doesn't support column pruning, because we can reduce data size between operators(e.g. shuffle). I think it's also true for nested column pruning.
We can implement nested pruning with 2 PRs:
1. improve the current column pruning rule(or add a new rule) to prune nested columns as bottom as possible
2. improve the planner rule to extract the required nested columns and push to parquet.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16578
@abhaynahar I think the reviewers are already included...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16578
**[Test build #79815 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79815/testReport)** for PR 16578 at commit [`b0c67c2`](https://github.com/apache/spark/commit/b0c67c2efb9020a57bcb6955d4d2641a7cbd24d1).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r142117188
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
--- End diff --
As I wrote below, this problem does exist with parquet-mr 1.8.2. Hence, I've reverted back to the version with the `parquetMrCompatibility`-related logic.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/16578
Maybe we also want to get feedback from @liancheng ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16578
I'm going on this again. But I think we still need other eyes on this too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r139331293
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala ---
@@ -63,9 +74,22 @@ private[parquet] class ParquetReadSupport extends ReadSupport[UnsafeRow] with Lo
StructType.fromString(schemaString)
}
- val parquetRequestedSchema =
+ val clippedParquetSchema =
ParquetReadSupport.clipParquetSchema(context.getFileSchema, catalystRequestedSchema)
+ val parquetRequestedSchema = if (parquetMrCompatibility) {
+ // Parquet-mr will throw an exception if we try to read a superset of the file's schema.
+ // Therefore, we intersect our clipped schema with the underlying file's schema
--- End diff --
`clipParquetSchema` can add column paths only exist in catalyst schema into parquet schema. I think those columns are useful but looks like the `intersectParquetGroups` will remove them again?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16578
sounds like this will be very good to have, production-tested with lots of interests -
@hvanhovell @gatorsmile @liancheng
NPE is a problem though. any luck isolating that further?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
Hmmm... this failed again for the same reason. I'll see if I can reproduce locally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16578
I tried this and this is definitely super useful! it's a big patch and most of the people working in this area are either doing something else that's not Spark, or working on a few high priority SPIPs (e.g. vectorized UDFs in Python, data source API v2), so it might take a bit for people to come around to review ...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r140366890
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdown.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+
+/**
+ * Pushes down aliases to [[expressions.GetStructField]] expressions in an aggregate's grouping and
+ * aggregate expressions into a projection over its children. The original
+ * [[expressions.GetStructField]] expressions are replaced with references to the pushed down
+ * aliases.
+ */
+object AggregateFieldExtractionPushdown extends FieldExtractionPushdown {
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case agg @ Aggregate(groupingExpressions, aggregateExpressions, child) =>
+ val expressions = groupingExpressions ++ aggregateExpressions
+ val attributes = AttributeSet(expressions.collect { case att: Attribute => att })
+ val childAttributes = AttributeSet(child.expressions)
+ val fieldExtractors0 =
+ expressions
+ .flatMap(getFieldExtractors)
+ .distinct
+ val fieldExtractors1 =
+ fieldExtractors0
+ .filter(_.collectFirst { case att: Attribute => att }
+ .filter(attributes.contains).isEmpty)
--- End diff --
> Why do we need to trim the extractors which contain attributes referred from `groupingExpressions ++ aggregateExpressions`?
Consider this query:
```sql
select a, a.b, count(1) from r1 group by a, a.b
```
The grouping fields are `a` and `a.b`. `a` is an `Attribute`. `a.b` is a `GetStructField`. Since we need all of `a` to answer this query, it doesn't make sense to attempt to push down `a.b`. At the same time, `fieldExtractors0` includes all `GetStructField` instances. This includes `a.b`. The code you refer to above filters out the `a.b` `GetStructField` because our query requires all of `a`.
If we do not filter out `a.b`, then the child (projection) of the new `Aggregate` will not contain `a` in its `output`. The query planner will barf.
The logic for the creation of the new child projection is here: https://github.com/apache/spark/blob/00ab80c9b78c45c1a8f8c202c5bab04a62cda2ef/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdown.scala#L63-L70
This case is tested by https://github.com/apache/spark/blob/00ab80c9b78c45c1a8f8c202c5bab04a62cda2ef/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdownSuite.scala#L60-L76
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16578
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/16578
we have back-ported it to 2.2, on production by an average it has saved us at least 2x time.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148454584
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/planning/SelectedFieldSuite.scala ---
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.planning
+
+import org.scalatest.BeforeAndAfterAll
+import org.scalatest.exceptions.TestFailedException
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.NamedExpression
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.types._
+
+// scalastyle:off line.size.limit
+class SelectedFieldSuite extends SparkFunSuite with BeforeAndAfterAll {
+ // The test schema as a tree string, i.e. `schema.treeString`
+ // root
+ // |-- col1: string (nullable = false)
+ // |-- col2: struct (nullable = true)
+ // | |-- field1: integer (nullable = true)
+ // | |-- field2: array (nullable = true)
+ // | | |-- element: integer (containsNull = false)
+ // | |-- field3: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | | |-- subfield3: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = true)
+ // | |-- field4: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- subfield1: integer (nullable = true)
+ // | | | |-- subfield2: array (nullable = true)
+ // | | | | |-- element: integer (containsNull = false)
+ // | |-- field5: array (nullable = false)
+ // | | |-- element: struct (containsNull = true)
+ // | | | |-- subfield1: struct (nullable = false)
+ // | | | | |-- subsubfield1: integer (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | | | |-- subfield2: struct (nullable = true)
+ // | | | | |-- subsubfield1: struct (nullable = true)
+ // | | | | | |-- subsubsubfield1: string (nullable = true)
+ // | | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field6: struct (nullable = true)
+ // | | |-- subfield1: string (nullable = false)
+ // | | |-- subfield2: string (nullable = true)
+ // | |-- field7: struct (nullable = true)
+ // | | |-- subfield1: struct (nullable = true)
+ // | | | |-- subsubfield1: integer (nullable = true)
+ // | | | |-- subsubfield2: integer (nullable = true)
+ // | |-- field8: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: array (valueContainsNull = false)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: array (nullable = true)
+ // | | | | | |-- element: integer (containsNull = false)
+ // | |-- field9: map (nullable = true)
+ // | | |-- key: string
+ // | | |-- value: integer (valueContainsNull = false)
+ // |-- col3: array (nullable = false)
+ // | |-- element: struct (containsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col4: map (nullable = false)
+ // | |-- key: string
+ // | |-- value: struct (valueContainsNull = false)
+ // | | |-- field1: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | | |-- subfield2: integer (nullable = true)
+ // | | |-- field2: map (nullable = true)
+ // | | | |-- key: string
+ // | | | |-- value: integer (valueContainsNull = false)
+ // |-- col5: array (nullable = true)
+ // | |-- element: map (containsNull = true)
+ // | | |-- key: string
+ // | | |-- value: struct (valueContainsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col6: map (nullable = true)
+ // | |-- key: string
+ // | |-- value: array (valueContainsNull = true)
+ // | | |-- element: struct (containsNull = false)
+ // | | | |-- field1: struct (nullable = true)
+ // | | | | |-- subfield1: integer (nullable = true)
+ // | | | | |-- subfield2: integer (nullable = true)
+ // |-- col7: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: integer (nullable = false)
+ // | | |-- field2: struct (nullable = true)
+ // | | | |-- subfield1: integer (nullable = false)
+ // | | |-- field3: array (nullable = true)
+ // | | | |-- element: struct (containsNull = true)
+ // | | | | |-- subfield1: integer (nullable = false)
+ // |-- col8: array (nullable = true)
+ // | |-- element: struct (containsNull = true)
+ // | | |-- field1: array (nullable = false)
+ // | | | |-- element: integer (containsNull = false)
+ private val schema =
+ StructType(
+ StructField("col1", StringType, nullable = false) ::
+ StructField("col2", StructType(
+ StructField("field1", IntegerType) ::
+ StructField("field2", ArrayType(IntegerType, containsNull = false)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) ::
+ StructField("subfield3", ArrayType(IntegerType)) :: Nil)), nullable = false) ::
+ StructField("field4", MapType(StringType, StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil), valueContainsNull = false)) ::
+ StructField("field5", ArrayType(StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil), nullable = false) ::
+ StructField("subfield2", StructType(
+ StructField("subsubfield1", StructType(
+ StructField("subsubsubfield1", StringType) :: Nil)) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)), nullable = false) ::
+ StructField("field6", StructType(
+ StructField("subfield1", StringType, nullable = false) ::
+ StructField("subfield2", StringType) :: Nil)) ::
+ StructField("field7", StructType(
+ StructField("subfield1", StructType(
+ StructField("subsubfield1", IntegerType) ::
+ StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)) ::
+ StructField("field8", MapType(StringType, ArrayType(StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", ArrayType(IntegerType, containsNull = false)) :: Nil)), valueContainsNull = false)) ::
+ StructField("field9", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil)) ::
+ StructField("col3", ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), containsNull = false), nullable = false) ::
+ StructField("col4", MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType, nullable = false) ::
+ StructField("subfield2", IntegerType) :: Nil)) ::
+ StructField("field2", MapType(StringType, IntegerType, valueContainsNull = false)) :: Nil), valueContainsNull = false), nullable = false) ::
+ StructField("col5", ArrayType(MapType(StringType, StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), valueContainsNull = false))) ::
+ StructField("col6", MapType(StringType, ArrayType(StructType(
+ StructField("field1", StructType(
+ StructField("subfield1", IntegerType) ::
+ StructField("subfield2", IntegerType) :: Nil)) :: Nil), containsNull = false))) ::
+ StructField("col7", ArrayType(StructType(
+ StructField("field1", IntegerType, nullable = false) ::
+ StructField("field2", StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil)) ::
+ StructField("field3", ArrayType(StructType(
+ StructField("subfield1", IntegerType, nullable = false) :: Nil))) :: Nil))) ::
+ StructField("col8", ArrayType(StructType(
+ StructField("field1", ArrayType(IntegerType, containsNull = false), nullable = false) :: Nil))) :: Nil)
+
+ private val testRelation = LocalRelation(schema.toAttributes)
+
+ override def beforeAll() {
+ val indentedSchema = schema.treeString.split('\n').map(x => " " + x).mkString("\n") + "\n"
+ // Print the test schema at the beginning of the test
+ // scalastyle:off println
+ println("Testing SelectedField extractor using the following schema:\n" + indentedSchema)
+ // scalastyle:on println
--- End diff --
I think we can get rid of this print now? Do we want to print it in every test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r148687395
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FieldExtractionPushdown.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, GetStructField}
+import org.apache.spark.sql.catalyst.planning.SelectedField
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+
+abstract class FieldExtractionPushdown extends Rule[LogicalPlan] {
--- End diff --
> nit: Does this need to be a Rule?
As opposed to?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Posted by DaimonPl <gi...@git.apache.org>.
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman how about finalizing it as is? IMHO performance improvements are worth more than (possibly) redundant workaround - it could be cleaned later
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org