You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/18 09:40:34 UTC
[GitHub] [spark] c21 opened a new pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
c21 opened a new pull request #29804:
URL: https://github.com/apache/spark/pull/29804
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR is to add support to decide bucketed table scan dynamically based on actual query plan. Currently bucketing is enabled by default (`spark.sql.sources.bucketing.enabled`=true), so for all bucketed tables in the query plan, we will use bucket table scan (all input files per the bucket will be read by same task). This has the drawback that if the bucket table scan is not benefitting at all (no join/groupby/etc in the query), we don't need to use bucket table scan as it would restrict the # of tasks to be # of buckets and might hurt parallelism.
The feature is to add a physical plan rule right after `EnsureRequirements`:
The rule goes through plan nodes. For all operators which has "interesting partition" (i.e., require `ClusteredDistribution` or `HashClusteredDistribution`), check if the sub-plan for operator has `Exchange` and bucketed table scan (and only allow certain operators in plan (i.e. `Scan/Filter/Project/Sort/PartialAgg/etc`.), see details in `PlanBucketing.canDisableBucketedScan`). If yes, disable the bucketed table scan in the sub-plan.
Why the algorithm works is that if there's a shuffle between the bucketed table scan and operator with interesting partition, then bucketed table scan partitioning will be destroyed by the shuffle operator in the middle, and we don't need bucketed table scan for sure.
The idea of "interesting partition" is inspired from "interesting order" in "Access Path Selection in a Relational Database Management System"(http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf), after discussion with @cloud-fan .
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
To avoid unnecessary bucketed scan in the query, and this is prerequisite for https://github.com/apache/spark/pull/29625 (decide bucketed sorted scan dynamically will be added later in that PR).
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
A new config `spark.sql.sources.dynamic.decide.bucketing.enabled` is introduced which set to true by default (the rule is enabled by default). User can opt-out by disabling the rule, as we found in prod, some users rely on assumption of # of tasks == # of buckets when reading bucket table to precisely control # of tasks. This is a bad assumption but it does happen on our side, so leave a config here to allow them opt-out for the feature.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added unit tests in `BucketedReadSuite.scala`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697957691
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492659404
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
Review comment:
nit: `DisableUnnecessaryBucketScan`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497155234
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * 2.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
Review comment:
nit: add space after 1. and 2.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * 2.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * Examples:
+ * 1.no [[hasInterestingPartition]] operator:
Review comment:
ditto.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * 2.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * Examples:
+ * 1.no [[hasInterestingPartition]] operator:
+ * Project
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.join:
Review comment:
ditto.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r500033528
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,17 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1. query does not have operators to utilize " +
+ "bucketing (e.g. join, group-by, etc), or 2. there's an exchange operator between these " +
+ s"operators and table scan. Note when '${BUCKETING_ENABLED.key}' is set to " +
+ "false, this configuration does not take any effect.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(false)
Review comment:
we can follow AQE and only disable it for table cache. See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala#L82
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695115619
**[Test build #128883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128883/testReport)** for PR 29804 at commit [`7ffcdbc`](https://github.com/apache/spark/commit/7ffcdbc288679238a4c8f16446f8259396a34a8c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697957150
**[Test build #129044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129044/testReport)** for PR 29804 at commit [`26c4338`](https://github.com/apache/spark/commit/26c4338f2ce480672fe779e5ea84b1d74f936e37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494851172
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,25 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
+ // TODO(SPARK-32986): Add bucketed scan info in explain output of FileSourceScanExec
+ if (bucketedScan) {
+ val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
Review comment:
nit: instead of writing
```
val withSelectedBucketsCount = some_code
withSelectedBucketsCount
```
we can just write `some_code`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492665465
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
Review comment:
> check if the sub-plan for operator has [[Exchange]]
Can we make it more fine-grained? It's possible that the exchange and the bucket scan are not in the same lineage:
```
node with interesting partition
|
binary node
/ \
Exchange Bucket Scan
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694897510
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128873/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494083795
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
@maropu - sure, filed https://issues.apache.org/jira/browse/SPARK-32985 .
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,43 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ test("SPARK-32859: disable unnecessary bucketed table scan based on query plan") {
+ withTable("t1", "t2") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t2")
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ Seq(
+ ("SELECT * FROM t1 JOIN t2 ON t1.i = t2.i", 1, 2),
+ ("SELECT * FROM t1 JOIN t2 ON t1.i = t2.j", 1, 2),
+ ("SELECT * FROM t1 JOIN t2 ON t1.j = t2.j", 0, 2),
+ ("SELECT SUM(i) FROM t1 GROUP BY i", 1, 1),
+ ("SELECT SUM(i) FROM t1 GROUP BY j", 0, 1),
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
+ ("SELECT * FROM t1 WHERE j = 1", 0, 1),
Review comment:
@maropu - sure. Sorry about missing the comment somehow, not shown up in github page. Added and split tests into 4 (basic, multiple join, multiple bucketed columns, other operators/cases). Thanks.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
+ val numSelectedBuckets = optionalBucketSet.map { b =>
+ b.cardinality()
+ } getOrElse {
+ spec.numBuckets
+ }
+ metadata += ("SelectedBucketsCount" ->
+ (s"$numSelectedBuckets out of ${spec.numBuckets}" +
+ optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
}
- metadata + ("SelectedBucketsCount" ->
- (s"$numSelectedBuckets out of ${spec.numBuckets}" +
- optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
- } getOrElse {
- metadata
+ } else if (disableBucketedScan) {
+ metadata += ("DisableBucketedScan" -> "true")
Review comment:
Added a TODO comment for followup jira - https://issues.apache.org/jira/browse/SPARK-32986 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496854445
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
Review comment:
@cloud-fan - added.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
Review comment:
@cloud-fan - updated.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
Review comment:
@cloud-fan - updated.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
Review comment:
@cloud-fan - sure, moved to a separate test suite - `DisableUnnecessaryBucketedScanSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697176375
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494083795
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
@maropu - sure, filed https://issues.apache.org/jira/browse/SPARK-32985 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701020808
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701731393
**[Test build #129295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129295/testReport)** for PR 29804 at commit [`b29f688`](https://github.com/apache/spark/commit/b29f688fc4a06bc35effa6d24632d2a64501c9fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r500100742
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,17 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1. query does not have operators to utilize " +
+ "bucketing (e.g. join, group-by, etc), or 2. there's an exchange operator between these " +
+ s"operators and table scan. Note when '${BUCKETING_ENABLED.key}' is set to " +
+ "false, this configuration does not take any effect.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(false)
Review comment:
@cloud-fan - thanks for pointing it out. Created https://issues.apache.org/jira/browse/SPARK-33075 for followup, cc @viirya in case there's any other regression for enabling auto bucketed scan, except cached query.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496077990
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
Review comment:
> 2.The sub-plan from root to bucketed table scan, does not contain [[hasInterestingPartition]] operator.
For the example 2, doesn't top HashAggregate have interesting partition? And there is also an Exchange, Is it an example for the condition 2?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
Review comment:
> 2.The sub-plan from root to bucketed table scan, does not contain [[hasInterestingPartition]] operator.
For the example 2, doesn't top HashAggregate have interesting partition? And there is also an Exchange, is it an example for the condition 2?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494933595
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
+ * [[hasInterestingPartition]].
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withNonAllowedNodes The traversed plan has operators not to be
+ * [[isAllowedUnaryExecNode]].
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withNonAllowedNodes: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withNonAllowedNodes` as false.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, false))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withNonAllowedNodes))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ val isSafeToDisableWithInterestingPartition =
Review comment:
The code can be even simpler if we use `withAllowedNode` as parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701244871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268392
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268392
**[Test build #129075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129075/testReport)** for PR 29804 at commit [`336337a`](https://github.com/apache/spark/commit/336337a1bc2827d7e05f7d83657a91438325140e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701244848
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33884/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496651792
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
Review comment:
Can we move the new tests to a new suite? They are more about bucket scan planning, instead of bucketed read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695134989
Looks an interesting idea, @c21 !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697176303
**[Test build #129008 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129008/testReport)** for PR 29804 at commit [`b45ee8b`](https://github.com/apache/spark/commit/b45ee8bac960dae0089325577f44c42c94e18032).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695115867
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu edited a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu edited a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695134989
Looks an interesting idea, @c21 ! I'll check this later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497153482
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
Review comment:
yea
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697163781
**[Test build #129008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129008/testReport)** for PR 29804 at commit [`b45ee8b`](https://github.com/apache/spark/commit/b45ee8bac960dae0089325577f44c42c94e18032).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698266541
@cloud-fan and @maropu - the PR is ready for review again after adding more unit tests and addressed all comments. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493230189
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
+ val numSelectedBuckets = optionalBucketSet.map { b =>
+ b.cardinality()
+ } getOrElse {
+ spec.numBuckets
+ }
+ metadata += ("SelectedBucketsCount" ->
+ (s"$numSelectedBuckets out of ${spec.numBuckets}" +
+ optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
}
- metadata + ("SelectedBucketsCount" ->
- (s"$numSelectedBuckets out of ${spec.numBuckets}" +
- optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
- } getOrElse {
- metadata
+ } else if (disableBucketedScan) {
+ metadata += ("DisableBucketedScan" -> "true")
Review comment:
At least we don't need to do it in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493242080
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: ProjectExec | _: FilterExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
Review comment:
instead of checking the mode, shall we just check `requiredChildDistributionExpressions`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037560
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493222439
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
Review comment:
@maropu - sure, updated. Was just following the reference in [CBO join reorder](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala#L117). Updated the link there as well.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -339,7 +343,7 @@ case class FileSourceScanExec(
location.getClass.getSimpleName +
Utils.buildLocationMetadata(location.rootPaths, maxMetadataValueLength)
val metadata =
- Map(
+ HashMap(
Review comment:
@maropu - updated. Was just following other code in the same file.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
+ }
+
+ private def disableAllBucketedScan(plan: SparkPlan): SparkPlan = {
+ plan.transformUp {
+ case scan: FileSourceScanExec if isBucketedScanWithoutFilter(scan) =>
+ scan.copy(disableBucketedScan = true)
+ }
+ }
+
+ def apply(plan: SparkPlan): SparkPlan = {
+ if (!conf.bucketingEnabled || !conf.autoBucketedScanEnabled) {
Review comment:
@maropu - sure, updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
Review comment:
@maropu - `FileSourceScanExec` should be pattern matched earlier in `disableBucketWithInterestingPartition` before `isAllowedUnaryExecNode` check. But thanks for catch, we don't need to add `Exchange` and `FileSourceScanExec` here anyway, removed both.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
Review comment:
@maropu - this is good question. I think currently only matching `Exchange` will also do the same job. Either way is fine with me. `InterestingPartition` and later on `InterestingOrder` (for bucketed sorted scan) looks like more general and we can extend them later.
One interesting extension I can think of - if bucketed scan parallelism is too low (to few # of buckets), we may decide to not do a bucketed scan for join to trade-off for query run-time vs extra shuffle cost (in this case, there's no `Exchange` before join).
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
@maropu - this is a good question, and I think it is kind of out of scope for this PR and needs more thoughts later. We don't have a cost model to decide whether to do (bucketed filter + bucketed scan) vs (normal filter + non-bucketed scan). It can depend on number of buckets, size of filtered buckets, CPU cost for filter, etc.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
Review comment:
@maropu - just for my own education, why does it matter? Updated anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268995
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495702507
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
Review comment:
@maropu - sure, just for my own education, what does it indicate to make a config internal/external?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
Review comment:
@maropu - sure, wondering what do you think of below?
```
When true, decide whether to do bucketed scan on input tables
based on query plan automatically. Do not use bucketed scan if
(1).query does not have operators to utilize bucketing (e.g. join, group-by, etc),
or (2).there's an exchange operator between these operators and table scan.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496084586
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
I already added a comment in last iteration. Please suggest concretely for alternative comment if it's not looking good. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037938
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700857633
**[Test build #129247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129247/testReport)** for PR 29804 at commit [`fbe0c06`](https://github.com/apache/spark/commit/fbe0c06a997aa8dfd7135b599d0bd01e9366b8ae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037560
**[Test build #128998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128998/testReport)** for PR 29804 at commit [`6a8c7a5`](https://github.com/apache/spark/commit/6a8c7a537c08dc5ef825b927f34dae669014b757).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496845997
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
Review comment:
@cloud-fan , @maropu - I changed the code during iterations. The current change is just adding a `if (bucketedScan) { ... } else { ... }` on top of original code, where we still need to use `map` as it's returning value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492656293
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
or `dynamicBucketScan`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
or `enableDynamicBucketScan`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492657236
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -165,6 +166,7 @@ case class FileSourceScanExec(
partitionFilters: Seq[Expression],
optionalBucketSet: Option[BitSet],
optionalNumCoalescedBuckets: Option[Int],
+ optionalDynamicDecideBucketing: Option[Boolean],
Review comment:
Why does it need to be `Option[Boolean]`? I'd expect a simple `disableBucketScan: Boolean = false`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701159093
**[Test build #129262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129262/testReport)** for PR 29804 at commit [`4027175`](https://github.com/apache/spark/commit/4027175ecade7b4dc8a555164cdc0b201b0845de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694897496
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698266541
@cloud-fan and @maropu - the PR is ready for review again after adding more unit tests and addressed all comments. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697212022
**[Test build #129021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129021/testReport)** for PR 29804 at commit [`9aa0266`](https://github.com/apache/spark/commit/9aa02664bcb6ca0eb3f72401026d95bc97c34088).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697039785
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496645430
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
Review comment:
I'd put this as item 1, as it's the simplest case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497904900
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
Review comment:
removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701202125
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r491242019
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -165,6 +166,7 @@ case class FileSourceScanExec(
partitionFilters: Seq[Expression],
optionalBucketSet: Option[BitSet],
optionalNumCoalescedBuckets: Option[Int],
+ optionalDynamicDecideBucketing: Option[Boolean],
Review comment:
nit: could you add this param in `metadata` for better explain output?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697300907
**[Test build #129021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129021/testReport)** for PR 29804 at commit [`9aa0266`](https://github.com/apache/spark/commit/9aa02664bcb6ca0eb3f72401026d95bc97c34088).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694767426
**[Test build #128873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128873/testReport)** for PR 29804 at commit [`a7193cf`](https://github.com/apache/spark/commit/a7193cfde4978aab052efe0335a86a8cf0705b39).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494854327
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
Review comment:
Rule 2 is very clear. I think we can simplify rule 1 to
```
The sub-plan from the nearest downstream [[hasInterestingPartition]] operator to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators and at least one [[Exchange]].
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494852129
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
Review comment:
nit: either `(1)` or `1.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700855626
> Can we apply this feature in AQE? seems we just need to add this rule to AdaptiveSparkPlanExec.queryStagePreparationRules. This can be done in followup.
@cloud-fan - thanks, created JIRA https://issues.apache.org/jira/browse/SPARK-33027 as the followup. Will do it next week after vocation, thanks.
All comments are addressed, thanks cc @maropu .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495714902
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
If `spark.sql.sources.bucketing.enabled` is disabled, does this config still work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496085453
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
Review comment:
This is not an example for option 2. Let me know if we want to add one example in comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493249493
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
Review comment:
let's make the rule clear. I think there are 2 cases when doing traverse here:
1. When we are looking for node with interesting partition, we don't need a whitelist, we can just go through any node. If we hit a file scan, disable its bucketed scan.
2. When we hit a node with interesting partition, and walk through its sub-plan, we need a whitelist and stop earlier if we hit a node outside of the whitelist.
I'd expect to see code like
```
if (withInterestingPartition) {
if (isAllowedUnaryExecNode(o)) {
...
} else {
// stop traversing down here.
o
}
} else {
o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497239920
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
+ "bucketing (e.g. join, group-by, etc), or 2.there's an exchange operator between these " +
+ "operators and table scan. Note when spark.sql.sources.bucketing.enabled is set to " +
Review comment:
updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * 2.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * Examples:
+ * 1.no [[hasInterestingPartition]] operator:
Review comment:
updated.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
Review comment:
@maropu , @viirya - sure, updated.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
+ "bucketing (e.g. join, group-by, etc), or 2.there's an exchange operator between these " +
Review comment:
updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * 2.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * Examples:
+ * 1.no [[hasInterestingPartition]] operator:
+ * Project
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.join:
Review comment:
updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * 2.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
Review comment:
updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701825592
Addressed all comments and updated PR description to reflect latest status of thing. cc @maropu thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493985269
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
Yea, okay. Could you file jira later, @c21 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701151447
@viirya - thanks for pointing out. With query cache, e.g. dataframe user calls `persist()`, we will store the query data as logical operator `InMemoryRelation` and later on with physical operator `InMemoryTableScanExec`. So if user caches the query which only reads the bucketed table, it will have regression later when user join/group-by/etc on cached table data.
I am fine with disabling the feature by default, and let user opt-in case by case. As normally for SQL users (not dataframe), the cache query should not be very popular. WDYT? @maropu . thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495030578
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
Review comment:
@cloud-fan - updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,25 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
+ // TODO(SPARK-32986): Add bucketed scan info in explain output of FileSourceScanExec
+ if (bucketedScan) {
+ val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
Review comment:
@cloud-fan - updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
+ * [[hasInterestingPartition]].
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withNonAllowedNodes The traversed plan has operators not to be
+ * [[isAllowedUnaryExecNode]].
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withNonAllowedNodes: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withNonAllowedNodes` as false.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, false))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withNonAllowedNodes))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ val isSafeToDisableWithInterestingPartition =
Review comment:
@cloud-fan - udpated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
Review comment:
@cloud-fan - updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494933217
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
+ * [[hasInterestingPartition]].
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withNonAllowedNodes The traversed plan has operators not to be
+ * [[isAllowedUnaryExecNode]].
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withNonAllowedNodes: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withNonAllowedNodes` as false.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, false))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withNonAllowedNodes))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ val isSafeToDisableWithInterestingPartition =
Review comment:
We can make the condition simpler:
```
if (!withInterestingPartition || (withExchange && !withNonAllowedNodes))
```
The logic is:
1. if it's not under interesting partition operator, we can always disable bucket scan
2. otherwise, we require at least one exchange in the subquery, and no nonAllowedNode
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699990030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268995
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697212753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701210307
**[Test build #129267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129267/testReport)** for PR 29804 at commit [`f2ceacd`](https://github.com/apache/spark/commit/f2ceacd65c7a8fcfab9f68e1a784a911c52a136d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701170865
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33879/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497895103
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
Review comment:
I think there is no table "t4"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701931635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699113855
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697039469
**[Test build #128999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128999/testReport)** for PR 29804 at commit [`a15f864`](https://github.com/apache/spark/commit/a15f8644317cc904d206aaffc5ccae42440787e4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697163781
**[Test build #129008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129008/testReport)** for PR 29804 at commit [`b45ee8b`](https://github.com/apache/spark/commit/b45ee8bac960dae0089325577f44c42c94e18032).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493984368
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
+ val numSelectedBuckets = optionalBucketSet.map { b =>
+ b.cardinality()
+ } getOrElse {
+ spec.numBuckets
+ }
+ metadata += ("SelectedBucketsCount" ->
+ (s"$numSelectedBuckets out of ${spec.numBuckets}" +
+ optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
}
- metadata + ("SelectedBucketsCount" ->
- (s"$numSelectedBuckets out of ${spec.numBuckets}" +
- optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
- } getOrElse {
- metadata
+ } else if (disableBucketedScan) {
+ metadata += ("DisableBucketedScan" -> "true")
Review comment:
> It's kind of the reason why there is no bucket scan in this node. The reason can be: 1. the table is not bucketed. 2. the bucket column is not read. 3. the planner decides to disable it as it has no benefits.
The intention in my comment meant that users need to be able to see why bucket scans are disabled as Wenchen pointed it out above. Anyway, the followup looks okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699007851
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33735/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699007877
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697212022
**[Test build #129021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129021/testReport)** for PR 29804 at commit [`9aa0266`](https://github.com/apache/spark/commit/9aa02664bcb6ca0eb3f72401026d95bc97c34088).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699990030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699007877
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701020808
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695156470
**[Test build #128883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128883/testReport)** for PR 29804 at commit [`7ffcdbc`](https://github.com/apache/spark/commit/7ffcdbc288679238a4c8f16446f8259396a34a8c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698050224
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495886405
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
In my opnion, it is okay for this PR to focus on a basic (minimal) support for the auto bucket scan. In followup activities, I think we can optimize it step-by-step by adding test cases and checking performance improvements... (Anyway, it would be better to leave some comment there about it as @viirya suggested above)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493227784
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -257,7 +260,8 @@ case class FileSourceScanExec(
// exposed for testing
Review comment:
We can remove this comment now. It's used in line 355
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493231175
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
+ val numSelectedBuckets = optionalBucketSet.map { b =>
+ b.cardinality()
+ } getOrElse {
+ spec.numBuckets
+ }
+ metadata += ("SelectedBucketsCount" ->
+ (s"$numSelectedBuckets out of ${spec.numBuckets}" +
+ optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
}
- metadata + ("SelectedBucketsCount" ->
- (s"$numSelectedBuckets out of ${spec.numBuckets}" +
- optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
- } getOrElse {
- metadata
+ } else if (disableBucketedScan) {
+ metadata += ("DisableBucketedScan" -> "true")
Review comment:
@cloud-fan - how about I update the explain with a `BucketedScan: [NON_BUCKETED, BUCKETED_COLUMNS_NOT_READ, DISABLED]` in a followup PR? cc @maropu .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037938
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701748765
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699988372
**[Test build #129173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129173/testReport)** for PR 29804 at commit [`e0c76c9`](https://github.com/apache/spark/commit/e0c76c98c7e56a5e443bdc2534f8f10db5c98fc7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699116328
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694896487
**[Test build #128873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128873/testReport)** for PR 29804 at commit [`a7193cf`](https://github.com/apache/spark/commit/a7193cfde4978aab052efe0335a86a8cf0705b39).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495707216
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
Review comment:
IIUC we don't have any strict rule for that. But, I think this new rule works well in most queries, so adding this as external looks less meaning because I think most users don't turn this feature off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697176375
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495721418
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
Can you add description why this is needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701210307
**[Test build #129267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129267/testReport)** for PR 29804 at commit [`f2ceacd`](https://github.com/apache/spark/commit/f2ceacd65c7a8fcfab9f68e1a784a911c52a136d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697212753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701086045
@viirya - wondering could you give an example that this feature can cause regression? Note this feature is to disable bucketed scan but not enable more. If there's further improvement (not regression fix) in your mind, happy to explore more separately as followup, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701159093
**[Test build #129262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129262/testReport)** for PR 29804 at commit [`4027175`](https://github.com/apache/spark/commit/4027175ecade7b4dc8a555164cdc0b201b0845de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701110070
I think it is easy to encounter such example. For case 1, the sub-plan from root to bucketed table scan, does not contain [[hasInterestingPartition]] operator. If we cache a query plan like that, but we have other query uses the cached query plan, and the other query has an operator with [[hasInterestingPartition]] on top of the cached query. Then we won't do bucket scan even bucket scan can benefit the later query. It seems to me, this feature can easily cause unintentional regression like that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495714902
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
If `spark.sql.sources.bucketing.enabled` is disabled, does this config still work? Please describe it in the config doc too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695115867
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492657749
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
Review comment:
`bucketing` -> `bucket scan`, or `bucketed table scan`, just use a consistent terminology.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494851172
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,25 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
+ // TODO(SPARK-32986): Add bucketed scan info in explain output of FileSourceScanExec
+ if (bucketedScan) {
+ val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
Review comment:
nit: instead of writing
```
val withSelectedBucketsCount = some_code
withSelectedBucketsCount
```
we can just write `some_code`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
Review comment:
nit: either `(1)` or `1.`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
Review comment:
Rule 2 is very clear. I think we can simplify rule 1 to
```
The sub-plan from the nearest downstream [[hasInterestingPartition]] operator to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators and at least one [[Exchange]].
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
+ * [[hasInterestingPartition]].
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withNonAllowedNodes The traversed plan has operators not to be
+ * [[isAllowedUnaryExecNode]].
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withNonAllowedNodes: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withNonAllowedNodes` as false.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, false))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withNonAllowedNodes))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ val isSafeToDisableWithInterestingPartition =
Review comment:
We can make the condition simpler:
```
if (!withInterestingPartition || (withExchange && !withNonAllowedNodes))
```
The logic is:
1. if it's not under interesting partition operator, we can always disable bucket scan
2. otherwise, we require at least one exchange in the subquery, and no nonAllowedNode
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * (1).The sub-plan from root to bucketed table scan, only contains operator which
+ * [[hasInterestingPartition]], [[Exchange]], and allowed single-child operators
+ * ([[isAllowedUnaryExecNode]]).
+ *
+ * (2).The sub-plan from root to bucketed table scan, does not contain operator which
+ * [[hasInterestingPartition]].
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withNonAllowedNodes The traversed plan has operators not to be
+ * [[isAllowedUnaryExecNode]].
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withNonAllowedNodes: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withNonAllowedNodes` as false.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, false))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withNonAllowedNodes))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ val isSafeToDisableWithInterestingPartition =
Review comment:
The code can be even simpler if we use `withAllowedNode` as parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492656178
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
+1
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
or `dynamicBucketScan`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
or `enableDynamicBucketScan`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -165,6 +166,7 @@ case class FileSourceScanExec(
partitionFilters: Seq[Expression],
optionalBucketSet: Option[BitSet],
optionalNumCoalescedBuckets: Option[Int],
+ optionalDynamicDecideBucketing: Option[Boolean],
Review comment:
Why does it need to be `Option[Boolean]`? I'd expect a simple `disableBucketScan: Boolean = false`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
Review comment:
`bucketing` -> `bucket scan`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
Review comment:
`bucketing` -> `bucket scan`, or `bucketed table scan`, just use a consistent terminology.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
Review comment:
nit: `DisableUnnecessaryBucketScan`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
+ private def disableBucketWithInterestingPartition(plan: SparkPlan): SparkPlan = {
+ var hasPlanWithInterestingPartition = false
+
+ val newPlan = plan.transformUp {
+ case p if hasInterestingPartition(p) =>
+ hasPlanWithInterestingPartition = true
Review comment:
This looks tricky and fragile. We shouldn't call `transformUp` and update a global state. Can we write a recursive method to do bottom-up tree traverse manually? using `TreeNode.mapChildren.`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
Review comment:
> check if the sub-plan for operator has [[Exchange]]
Can we make it more fine-grained? It's possible that the exchange and the bucket scan are not in the same lineage:
```
node with interesting partition
|
binary node
/ \
Exchange Bucket Scan
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
Review comment:
The above quey is not valid, I just put it here as an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-702516837
Thank you @maropu , @viirya and @cloud-fan for all discussion and help!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494215013
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
+ val numSelectedBuckets = optionalBucketSet.map { b =>
+ b.cardinality()
+ } getOrElse {
+ spec.numBuckets
+ }
+ metadata += ("SelectedBucketsCount" ->
+ (s"$numSelectedBuckets out of ${spec.numBuckets}" +
+ optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
}
- metadata + ("SelectedBucketsCount" ->
- (s"$numSelectedBuckets out of ${spec.numBuckets}" +
- optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
- } getOrElse {
- metadata
+ } else if (disableBucketedScan) {
+ metadata += ("DisableBucketedScan" -> "true")
Review comment:
Added a TODO comment for followup jira - https://issues.apache.org/jira/browse/SPARK-32986 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493097869
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -339,7 +343,7 @@ case class FileSourceScanExec(
location.getClass.getSimpleName +
Utils.buildLocationMetadata(location.rootPaths, maxMetadataValueLength)
val metadata =
- Map(
+ HashMap(
Review comment:
nit: `HashMap` -> `mutable.HashMap` for readability.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
Review comment:
nit: could we use a link to the ACM page instead? https://dl.acm.org/doi/10.1145/582095.582099
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
Review comment:
`map` -> `foreach`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
+ }
+
+ private def disableAllBucketedScan(plan: SparkPlan): SparkPlan = {
+ plan.transformUp {
+ case scan: FileSourceScanExec if isBucketedScanWithoutFilter(scan) =>
+ scan.copy(disableBucketedScan = true)
+ }
+ }
+
+ def apply(plan: SparkPlan): SparkPlan = {
+ if (!conf.bucketingEnabled || !conf.autoBucketedScanEnabled) {
Review comment:
If a plan doesn't have any bucket spec, we do nothing in this rule? Could we filter out the case at the beginning?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
Review comment:
> _: FileSourceScanExec
Is the case only allowed if `FileSourceScanExec` has a relation with bucket specs?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
What if a scan operator reads most buckets? e.g., 999 of 1000 buckets. We select bucket scans even in this case?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
Review comment:
Just a question; I read the PR description and I thought first that this rule is to find exchange plan nodes (inserted by `EnsureRequirements`) having bucket scans and then it disables the scans if the exchanges make bucket read meaningless. What I imagined is like this;
```
private def disableBucketWithInterestingPartition(...): SparkPlan = {
plan match {
case exchange: Exchange if isAllowedUnaryExecNode(exchange.child) =>
val newPlan = (disable bucket scan if the scan is meaningless)
exchange.withNewChildren(newPlan:: Nil)
case o =>
o.mapChildren(disableBucketWithInterestingPartition(...))
}
}
```
But, the current code looks more general. Any reason to propagate required distributions in a top-down manner?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701232525
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33884/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698977132
**[Test build #129115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129115/testReport)** for PR 29804 at commit [`057c021`](https://github.com/apache/spark/commit/057c021aeff739cc2d96eb374538f83b08d85654).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492657749
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
Review comment:
`bucketing` -> `bucket scan`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701176687
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33879/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700401382
Let me know if anything else needs to be addressed before merge, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701338424
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-702456924
okay, it looks okay as a first step for this feature, I think. Thanks for the valuable work, @c21! I'll merge this. If more comments are given by @cloud-fan, please address them in followup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496634575
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
Review comment:
Yea, please only use `map` when you care about the return value. `foreach` is better if you just want to do some calculation if `Option` is `Some`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037560
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496072631
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
We should make the code clear for developer and maintainer, so leaving some comment is nicer if we want to constrain the scope of this rule for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700887717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33864/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495698286
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumBucketedScanEnabled: Int,
Review comment:
nit: This param name looks a bit confusing and this means the number of bucketed scans when `AUTO_BUCKETED_SCAN_ENABLED` enabled?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
Review comment:
btw, we need to make this config external? If we just add this config for keeping the current behaviour, is it okay to add it as internal one?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
Review comment:
Since our user documents are generated based on this statement, could you describe a bit more about how to decide whether to do bucketed scans or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698408820
**[Test build #129075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129075/testReport)** for PR 29804 at commit [`336337a`](https://github.com/apache/spark/commit/336337a1bc2827d7e05f7d83657a91438325140e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492663014
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
+ private def disableBucketWithInterestingPartition(plan: SparkPlan): SparkPlan = {
+ var hasPlanWithInterestingPartition = false
+
+ val newPlan = plan.transformUp {
+ case p if hasInterestingPartition(p) =>
+ hasPlanWithInterestingPartition = true
Review comment:
This looks tricky and fragile. We shouldn't call `transformUp` and update a global state. Can we write a recursive method to do bottom-up tree traverse manually? using `TreeNode.mapChildren.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701176700
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497135535
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
Review comment:
super nit: Probably, please add a space: `1. query...`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
+ "bucketing (e.g. join, group-by, etc), or 2.there's an exchange operator between these " +
Review comment:
ditto: `2. there's`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,18 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .internal()
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically. Do not use bucketed scan if 1.query does not have operators to utilize " +
+ "bucketing (e.g. join, group-by, etc), or 2.there's an exchange operator between these " +
+ "operators and table scan. Note when spark.sql.sources.bucketing.enabled is set to " +
Review comment:
`spark.sql.sources.bucketing.enabled` -> `${BUCKETING_ENABLED.key}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701930532
**[Test build #129295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129295/testReport)** for PR 29804 at commit [`b29f688`](https://github.com/apache/spark/commit/b29f688fc4a06bc35effa6d24632d2a64501c9fd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698410094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694768118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r491243552
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
How about adding this param under `spark.sql.sources.bucketing.xxx`? `spark.sql.sources.bucketing.enableAutoBucketScan` or something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694767426
**[Test build #128873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128873/testReport)** for PR 29804 at commit [`a7193cf`](https://github.com/apache/spark/commit/a7193cfde4978aab052efe0335a86a8cf0705b39).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492919052
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -165,6 +166,7 @@ case class FileSourceScanExec(
partitionFilters: Seq[Expression],
optionalBucketSet: Option[BitSet],
optionalNumCoalescedBuckets: Option[Int],
+ optionalDynamicDecideBucketing: Option[Boolean],
Review comment:
@maropu - sure, updated. Added a new field `DisableBucketedScan` for explain. This will be printed out when bucketed scan is disable. In addition, changed to not print out the other field `SelectedBucketsCount` if this is not bucketed scan.
@cloud-fan - sure, updated. Was just trying to keep consistent with other bucketing parameters here, but an `Option` is not necessary.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
Review comment:
@cloud-fan - updated.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
@maropu , @cloud-fan - sure, updated. Change the config name to be `spark.sql.sources.bucketing.autoBucketedScan.enabled` as most of our configs are with `.enabled` at the end.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
Review comment:
@cloud-fan - updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
+ private def disableBucketWithInterestingPartition(plan: SparkPlan): SparkPlan = {
+ var hasPlanWithInterestingPartition = false
+
+ val newPlan = plan.transformUp {
+ case p if hasInterestingPartition(p) =>
+ hasPlanWithInterestingPartition = true
Review comment:
@cloud-fan - sure, I feel the same way and sorry for the laziness. I updated to use `TreeNode.mapChildren` with only one round of pre-order traversal of plan (`disableBucketWithInterestingPartition`). Wondering does it look better? Thanks.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
Review comment:
@cloud-fan - sorry if I misunderstand anything, I have check to only allow certain `UnaryExecNode` operators in sub-plan, so the single lineage property is guaranteed. I updated the comment to call it out explicitly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701731393
**[Test build #129295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129295/testReport)** for PR 29804 at commit [`b29f688`](https://github.com/apache/spark/commit/b29f688fc4a06bc35effa6d24632d2a64501c9fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699116328
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037938
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493877941
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
Review comment:
@cloud-fan - as discussed offline, we still need to traverse sub-plan for node with interesting partition, even though this node cannot disable bucketed scan (e.g. multiple join cases).
Updated the code to disable bucketed table scan if:
1. The sub-plan from root to bucketed table scan, only contains node with interesting partition, exchange, and allowed whitelisted single-child nodes.
2. The sub-plan from root to bucketed table scan, does not contain node with interesting partition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496926590
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
Review comment:
Yeah, thanks for adding it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701202125
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700650620
Can we apply this feature in AQE? seems we just need to add this rule to `AdaptiveSparkPlanExec.queryStagePreparationRules`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700887683
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33864/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-702457642
Thanks, @c21 @viirya ! Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495721418
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
Can you add description why this is needed? HasInterestingPartition and at lease one Exchange sounds obvious condition, but this allowed unary exec node is not. Why we can disable bucketed scan only if those exec nodes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701202144
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129262/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699879721
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33788/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697176387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129008/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493097869
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -339,7 +343,7 @@ case class FileSourceScanExec(
location.getClass.getSimpleName +
Utils.buildLocationMetadata(location.rootPaths, maxMetadataValueLength)
val metadata =
- Map(
+ HashMap(
Review comment:
nit: `HashMap` -> `mutable.HashMap` for readability.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
Review comment:
nit: could we use a link to the ACM page instead? https://dl.acm.org/doi/10.1145/582095.582099
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
Review comment:
`map` -> `foreach`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
+ }
+
+ private def disableAllBucketedScan(plan: SparkPlan): SparkPlan = {
+ plan.transformUp {
+ case scan: FileSourceScanExec if isBucketedScanWithoutFilter(scan) =>
+ scan.copy(disableBucketedScan = true)
+ }
+ }
+
+ def apply(plan: SparkPlan): SparkPlan = {
+ if (!conf.bucketingEnabled || !conf.autoBucketedScanEnabled) {
Review comment:
If a plan doesn't have any bucket spec, we do nothing in this rule? Could we filter out the case at the beginning?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
Review comment:
> _: FileSourceScanExec
Is the case only allowed if `FileSourceScanExec` has a relation with bucket specs?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
What if a scan operator reads most buckets? e.g., 999 of 1000 buckets. We select bucket scans even in this case?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
Review comment:
Just a question; I read the PR description and I thought first that this rule is to find exchange plan nodes (inserted by `EnsureRequirements`) having bucket scans and then it disables the scans if the exchanges make bucket read meaningless. What I imagined is like this;
```
private def disableBucketWithInterestingPartition(...): SparkPlan = {
plan match {
case exchange: Exchange if isAllowedUnaryExecNode(exchange.child) =>
val newPlan = (disable bucket scan if the scan is meaningless)
exchange.withNewChildren(newPlan:: Nil)
case o =>
o.mapChildren(disableBucketWithInterestingPartition(...))
}
}
```
But, the current code looks more general. Any reason to propagate required distributions in a top-down manner?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701201838
**[Test build #129262 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129262/testReport)** for PR 29804 at commit [`4027175`](https://github.com/apache/spark/commit/4027175ecade7b4dc8a555164cdc0b201b0845de).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701748753
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33911/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496647273
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
Review comment:
Yea, let's add an example for it as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701209877
> Could you drop .internal accordingly if users need to care about the added option?
@maropu - sure, removed `internal()`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495701179
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumBucketedScanEnabled: Int,
Review comment:
@maropu - yes. I agree this is bit confusing too, do you have some better name in mind? e.g. `expectedNumScanWithAutoScanEnabled` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493991276
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,43 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ test("SPARK-32859: disable unnecessary bucketed table scan based on query plan") {
+ withTable("t1", "t2") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t2")
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ Seq(
+ ("SELECT * FROM t1 JOIN t2 ON t1.i = t2.i", 1, 2),
+ ("SELECT * FROM t1 JOIN t2 ON t1.i = t2.j", 1, 2),
+ ("SELECT * FROM t1 JOIN t2 ON t1.j = t2.j", 0, 2),
+ ("SELECT SUM(i) FROM t1 GROUP BY i", 1, 1),
+ ("SELECT SUM(i) FROM t1 GROUP BY j", 0, 1),
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
+ ("SELECT * FROM t1 WHERE j = 1", 0, 1),
Review comment:
I left two comments about the test;
- Could you add more test cases, e.g., multiple join cases, multiple bucket column cases, ...?
- Could you split this single test unit into multiple ones having meaningful test titles?, e.g., `test("SPARK-32859: disable unnecessary bucketed table scan based on query plan - multiple join test")`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698997290
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33735/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698050224
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268995
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701176700
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701110070
I think it is easy to encounter such example. For case 1, the sub-plan from root to bucketed table scan, does not contain [[hasInterestingPartition]] operator. If we cache a query plan like that, but we have other query uses the cached query plan, and the other query has an operator with [[hasInterestingPartition]] on top of the cached query. Then we won't do bucket scan even bucket scan can benefit the later query. It seems to me, this feature can easily cause unintentional regression like that.
Maybe we should disable it by default? This feature can confuse users easily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497899976
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(8, "i").saveAsTable("t2")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t3")
+
+ Seq(
+ // Read bucketed table
+ ("SELECT * FROM t1", 0, 1),
+ ("SELECT i FROM t1", 0, 1),
+ ("SELECT j FROM t1", 0, 0),
+ // Filter on bucketed column
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
Review comment:
If there's filter on bucketed column, then we can use bucket filter push down to only read targeted bucket files, so we decide not to disable bucketed scan. See `DisableUnnecessaryBucketedScan.isBucketedScanWithoutFilter()`, I already added comment into it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495701717
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumBucketedScanEnabled: Int,
Review comment:
yea, it looks better ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497898501
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
Review comment:
@viirya - yes, this is typo, will remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu closed pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu closed pull request #29804:
URL: https://github.com/apache/spark/pull/29804
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694897496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699892158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697957691
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495758770
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumBucketedScanEnabled: Int,
Review comment:
@maropu - sure, updated.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
Review comment:
@maropu - sure, updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
@viirya - this is good question. I agree we can be more bold and we probably don't need a whitelist operators here, e.g. SMJ - shuffle - BHJ - scan, SMJ - Shuffle - union - Scan (and another scan) should also work, but my feeling is to start with more confidence change first and improve later. With a whitelist operators here, we have a high confidence that this feature should work without introducing regression, but much less confidence if we allow arbitrary operators in the middle (at least for me). For now, to be honest, I cannot find a case why arbitrary operators cannot work. But I want to play safer in the beginning and any future improvement for this is much welcomed. cc @cloud-fan and @maropu for thoughts.
Added a comment for now.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
@viirya - this config will not take any effect in that case, updated.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
+ "automatically.")
+ .version("3.1.0")
Review comment:
@maropu - sure, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699892135
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33788/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700650620
Can we apply this feature in AQE? seems we just need to add this rule to `AdaptiveSparkPlanExec.queryStagePreparationRules`. This can be done in followup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495721418
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if:
+ *
+ * 1.The sub-plan from the nearest downstream [[hasInterestingPartition]] operator
+ * to the bucketed table scan, contains only [[isAllowedUnaryExecNode]] operators
+ * and at least one [[Exchange]].
+ *
+ * 2.The sub-plan from root to bucketed table scan, does not contain
+ * [[hasInterestingPartition]] operator.
+ *
+ * Examples:
+ * 1.join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * 2.aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ * @param withAllowedNode The traversed plan has only [[isAllowedUnaryExecNode]] operators.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean,
+ withAllowedNode: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operator with interesting partition, propagates `withInterestingPartition` as true
+ // to its children, and resets `withExchange` and `withAllowedNode`.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false, true))
+ case exchange: Exchange =>
+ // Exchange operator propagates `withExchange` as true to its child.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true, withAllowedNode))
+ case scan: FileSourceScanExec =>
+ if (isBucketedScanWithoutFilter(scan)) {
+ if (!withInterestingPartition || (withExchange && withAllowedNode)) {
+ scan.copy(disableBucketedScan = true)
+ } else {
+ scan
+ }
+ } else {
+ scan
+ }
+ case o =>
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _,
+ withInterestingPartition,
+ withExchange,
+ withAllowedNode && isAllowedUnaryExecNode(o)))
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
Review comment:
Can you add description why this is needed? HasInterestingPartition and at lease one Exchange sounds obvious condition, but this allowed unary exec node is not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701748765
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694767363
cc @cloud-fan and @sameeragarwal if you guys have time to take a look, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493390415
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
Review comment:
Yea, I remember the previous discussion: https://issues.apache.org/jira/browse/SPARK-16694
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493283306
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (https://dl.acm.org/doi/10.1145/582095.582099).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: ProjectExec | _: FilterExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
Review comment:
@cloud-fan - sure, updated. Also thanks for catch, I think `requiredChildDistributionExpressions` covers all cases where we can have partial aggregate without any aggregate expression (e.g. `SELECT i from t GROUP BY i`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268392
**[Test build #129075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129075/testReport)** for PR 29804 at commit [`336337a`](https://github.com/apache/spark/commit/336337a1bc2827d7e05f7d83657a91438325140e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699857418
**[Test build #129173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129173/testReport)** for PR 29804 at commit [`e0c76c9`](https://github.com/apache/spark/commit/e0c76c98c7e56a5e443bdc2534f8f10db5c98fc7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695156624
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698977132
**[Test build #129115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129115/testReport)** for PR 29804 at commit [`057c021`](https://github.com/apache/spark/commit/057c021aeff739cc2d96eb374538f83b08d85654).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r494214468
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,43 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ test("SPARK-32859: disable unnecessary bucketed table scan based on query plan") {
+ withTable("t1", "t2") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t2")
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ Seq(
+ ("SELECT * FROM t1 JOIN t2 ON t1.i = t2.i", 1, 2),
+ ("SELECT * FROM t1 JOIN t2 ON t1.i = t2.j", 1, 2),
+ ("SELECT * FROM t1 JOIN t2 ON t1.j = t2.j", 0, 2),
+ ("SELECT SUM(i) FROM t1 GROUP BY i", 1, 1),
+ ("SELECT SUM(i) FROM t1 GROUP BY j", 0, 1),
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
+ ("SELECT * FROM t1 WHERE j = 1", 0, 1),
Review comment:
@maropu - sure. Sorry about missing the comment somehow, not shown up in github page. Added and split tests into 4 (basic, multiple join, multiple bucketed columns, other operators/cases). Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699857418
**[Test build #129173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129173/testReport)** for PR 29804 at commit [`e0c76c9`](https://github.com/apache/spark/commit/e0c76c98c7e56a5e443bdc2534f8f10db5c98fc7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698972682
**[Test build #129114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129114/testReport)** for PR 29804 at commit [`1012334`](https://github.com/apache/spark/commit/1012334e2e24be94375be678054156b12e1ec2d5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701931635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r496650299
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
Review comment:
nit: 4 space indentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700857633
**[Test build #129247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129247/testReport)** for PR 29804 at commit [`fbe0c06`](https://github.com/apache/spark/commit/fbe0c06a997aa8dfd7135b599d0bd01e9366b8ae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492656178
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701458412
@viirya - wondering any other comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695115619
**[Test build #128883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128883/testReport)** for PR 29804 at commit [`7ffcdbc`](https://github.com/apache/spark/commit/7ffcdbc288679238a4c8f16446f8259396a34a8c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700876562
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33864/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699112822
**[Test build #129114 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129114/testReport)** for PR 29804 at commit [`1012334`](https://github.com/apache/spark/commit/1012334e2e24be94375be678054156b12e1ec2d5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701337640
**[Test build #129267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129267/testReport)** for PR 29804 at commit [`f2ceacd`](https://github.com/apache/spark/commit/f2ceacd65c7a8fcfab9f68e1a784a911c52a136d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701019537
**[Test build #129247 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129247/testReport)** for PR 29804 at commit [`fbe0c06`](https://github.com/apache/spark/commit/fbe0c06a997aa8dfd7135b599d0bd01e9366b8ae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497894191
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(8, "i").saveAsTable("t2")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t3")
+
+ Seq(
+ // Read bucketed table
+ ("SELECT * FROM t1", 0, 1),
+ ("SELECT i FROM t1", 0, 1),
+ ("SELECT j FROM t1", 0, 0),
+ // Filter on bucketed column
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
Review comment:
Isn't this case 1 "no [[hasInterestingPartition]] operator"? Why there is one bucket scan?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493233965
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Disable unnecessary bucketed table scan based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and AUTO_BUCKETED_SCAN_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan. If yes, disable the bucketed table scan in the sub-plan.
+ * Only allow certain operators in sub-plan, which guarantees each sub-plan is single lineage
+ * (i.e., each operator has only one child). See details in
+ * [[disableBucketWithInterestingPartition]]).
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class DisableUnnecessaryBucketedScan(conf: SQLConf) extends Rule[SparkPlan] {
+
+ /**
+ * Disable bucketed table scan with pre-order traversal of plan.
+ *
+ * @param withInterestingPartition The traversed plan has operator with interesting partition.
+ * @param withExchange The traversed plan has [[Exchange]] operator.
+ */
+ private def disableBucketWithInterestingPartition(
+ plan: SparkPlan,
+ withInterestingPartition: Boolean,
+ withExchange: Boolean): SparkPlan = {
+ plan match {
+ case p if hasInterestingPartition(p) =>
+ // Operators with interesting partition, propagates `withInterestingPartition` as true
+ // to its children.
+ p.mapChildren(disableBucketWithInterestingPartition(_, true, false))
+ case exchange: Exchange if withInterestingPartition =>
+ // Exchange operator propagates `withExchange` as true to its child
+ // if the plan has interesting partition.
+ exchange.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, true))
+ case scan: FileSourceScanExec
+ if withInterestingPartition && withExchange && isBucketedScanWithoutFilter(scan) =>
+ // Disable bucketed table scan if the plan has interesting partition,
+ // and [[Exchange]] in the plan.
+ scan.copy(disableBucketedScan = true)
+ case o =>
+ if (isAllowedUnaryExecNode(o)) {
+ // Propagates `withInterestingPartition` and `withExchange` from parent
+ // for only allowed single-child nodes.
+ o.mapChildren(disableBucketWithInterestingPartition(
+ _, withInterestingPartition, withExchange))
+ } else {
+ o.mapChildren(disableBucketWithInterestingPartition(_, false, false))
+ }
+ }
+ }
+
+ private def hasInterestingPartition(plan: SparkPlan): Boolean = {
+ plan.requiredChildDistribution.exists {
+ case _: ClusteredDistribution | _: HashClusteredDistribution => true
+ case _ => false
+ }
+ }
+
+ private def isAllowedUnaryExecNode(plan: SparkPlan): Boolean = {
+ plan match {
+ case _: SortExec | _: Exchange | _: ProjectExec | _: FilterExec |
+ _: FileSourceScanExec => true
+ case partialAgg: BaseAggregateExec =>
+ val modes = partialAgg.aggregateExpressions.map(_.mode)
+ modes.nonEmpty && modes.forall(mode => mode == Partial || mode == PartialMerge)
+ case _ => false
+ }
+ }
+
+ private def isBucketedScanWithoutFilter(scan: FileSourceScanExec): Boolean = {
+ // Do not disable bucketed table scan if it has filter pruning,
+ // because bucketed table scan is still useful here to save CPU/IO cost with
+ // only reading selected bucket files.
+ scan.bucketedScan && scan.optionalBucketSet.isEmpty
Review comment:
I'm fine with it for now. Technically I think filter by bucket ID and bucketed scan don't need to be coupled. We can always filter files by bucket id, and then do bucketed scan or not according to this rule.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698268392
**[Test build #129075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129075/testReport)** for PR 29804 at commit [`336337a`](https://github.com/apache/spark/commit/336337a1bc2827d7e05f7d83657a91438325140e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-695156624
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698410094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495707732
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables based on query plan " +
Review comment:
It looks okay, but I have the same suggestion with https://github.com/apache/spark/pull/29804#discussion_r494852129
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701244871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701742021
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33911/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697301238
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701203694
> I am fine with disabling the feature by default, and let user opt-in case by case. As normally for SQL users (not dataframe), the cache query should not be very popular. WDYT? @maropu . thanks.
Ah, I see and I missed the case. The @viirya suggestion makes sense, so disabling it by default looks okay to me. Could you drop `.internal` accordingly if users need to care about the added option?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699892158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-694768118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697301245
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129021/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r493229395
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -348,20 +352,22 @@ case class FileSourceScanExec(
"DataFilters" -> seqToString(dataFilters),
"Location" -> locationDesc)
- val withSelectedBucketsCount = relation.bucketSpec.map { spec =>
- val numSelectedBuckets = optionalBucketSet.map { b =>
- b.cardinality()
- } getOrElse {
- spec.numBuckets
+ if (bucketedScan) {
+ relation.bucketSpec.map { spec =>
+ val numSelectedBuckets = optionalBucketSet.map { b =>
+ b.cardinality()
+ } getOrElse {
+ spec.numBuckets
+ }
+ metadata += ("SelectedBucketsCount" ->
+ (s"$numSelectedBuckets out of ${spec.numBuckets}" +
+ optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
}
- metadata + ("SelectedBucketsCount" ->
- (s"$numSelectedBuckets out of ${spec.numBuckets}" +
- optionalNumCoalescedBuckets.map { b => s" (Coalesced to $b)"}.getOrElse("")))
- } getOrElse {
- metadata
+ } else if (disableBucketedScan) {
+ metadata += ("DisableBucketedScan" -> "true")
Review comment:
Do we need it? It's kind of the reason why there is no bucket scan in this node. The reason can be: 1. the table is not bucketed. 2. the bucket column is not read. 3. the planner decides to disable it as it has no benefits.
If we do need the reason, we should make it completed. Let's not just put the `disableBucketedScan` flag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697164192
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698972682
**[Test build #129114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129114/testReport)** for PR 29804 at commit [`1012334`](https://github.com/apache/spark/commit/1012334e2e24be94375be678054156b12e1ec2d5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-698050224
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700887702
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-700887702
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492919052
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -165,6 +166,7 @@ case class FileSourceScanExec(
partitionFilters: Seq[Expression],
optionalBucketSet: Option[BitSet],
optionalNumCoalescedBuckets: Option[Int],
+ optionalDynamicDecideBucketing: Option[Boolean],
Review comment:
@maropu - sure, updated. Added a new field `DisableBucketedScan` for explain. This will be printed out when bucketed scan is disable. In addition, changed to not print out the other field `SelectedBucketsCount` if this is not bucketed scan.
@cloud-fan - sure, updated. Was just trying to keep consistent with other bucketing parameters here, but an `Option` is not necessary.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
Review comment:
@cloud-fan - updated.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2764,6 +2764,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val DYNAMIC_DECIDE_BUCKETING_ENABLED =
+ buildConf("spark.sql.sources.dynamic.decide.bucketing.enabled")
Review comment:
@maropu , @cloud-fan - sure, updated. Change the config name to be `spark.sql.sources.bucketing.autoBucketedScan.enabled` as most of our configs are with `.enabled` at the end.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
Review comment:
@cloud-fan - updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
+ * bucketed table scan (and only allow certain operators in plan, see details in
+ * [[canDisableBucketedScan]]). If yes, disable the bucketed table scan in the sub-plan.
+ *
+ * Examples:
+ * (1).join:
+ * SortMergeJoin(t1.i = t2.j)
+ * / \
+ * Sort(i) Sort(j)
+ * / \
+ * Shuffle(i) Scan(t2: i, j)
+ * / (bucketed on column j, enable bucketed scan)
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * (2).aggregate:
+ * HashAggregate(i, ..., Final)
+ * |
+ * Shuffle(i)
+ * |
+ * HashAggregate(i, ..., Partial)
+ * |
+ * Filter
+ * |
+ * Scan(t1: i, j)
+ * (bucketed on column j, DISABLE bucketed scan)
+ *
+ * The idea of [[hasInterestingPartition]] is inspired from "interesting order" in
+ * the paper "Access Path Selection in a Relational Database Management System"
+ * (http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf).
+ */
+case class PlanBucketing(conf: SQLConf) extends Rule[SparkPlan] {
+ private def disableBucketWithInterestingPartition(plan: SparkPlan): SparkPlan = {
+ var hasPlanWithInterestingPartition = false
+
+ val newPlan = plan.transformUp {
+ case p if hasInterestingPartition(p) =>
+ hasPlanWithInterestingPartition = true
Review comment:
@cloud-fan - sure, I feel the same way and sorry for the laziness. I updated to use `TreeNode.mapChildren` with only one round of pre-order traversal of plan (`disableBucketWithInterestingPartition`). Wondering does it look better? Thanks.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
Review comment:
@cloud-fan - sorry if I misunderstand anything, I have check to only allow certain `UnaryExecNode` operators in sub-plan, so the single lineage property is guaranteed. I updated the comment to call it out explicitly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701338424
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697037938
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697164192
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497900170
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(8, "i").saveAsTable("t2")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t3")
+
+ Seq(
+ // Read bucketed table
+ ("SELECT * FROM t1", 0, 1),
+ ("SELECT i FROM t1", 0, 1),
+ ("SELECT j FROM t1", 0, 0),
+ // Filter on bucketed column
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
Review comment:
just fyi relevant discussion before - https://github.com/apache/spark/pull/29804#discussion_r493100510 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-697301238
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699113855
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-699115423
**[Test build #129115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129115/testReport)** for PR 29804 at commit [`057c021`](https://github.com/apache/spark/commit/057c021aeff739cc2d96eb374538f83b08d85654).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29804:
URL: https://github.com/apache/spark/pull/29804#issuecomment-701158601
Disabled the config by default and addressed all new format comments. Let me know if there's anything needed to be addressed. Will look into how to workaround cached query problem in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r497901758
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+
+class DisableUnnecessaryBucketedScanWithoutHiveSupportSuite
+ extends DisableUnnecessaryBucketedScanSuite
+ with SharedSparkSession {
+
+ protected override def beforeAll(): Unit = {
+ super.beforeAll()
+ assert(spark.sparkContext.conf.get(CATALOG_IMPLEMENTATION) == "in-memory")
+ }
+}
+
+abstract class DisableUnnecessaryBucketedScanSuite extends QueryTest with SQLTestUtils {
+ import testImplicits._
+
+ private lazy val df1 =
+ (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k").as("df1")
+ private lazy val df2 =
+ (0 until 50).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k").as("df2")
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumScanWithAutoScanEnabled: Int,
+ expectedNumScanWithAutoScanDisabled: Int): Unit = {
+
+ def checkNumBucketedScan(query: String, expectedNumBucketedScan: Int): Unit = {
+ val plan = sql(query).queryExecution.executedPlan
+ val bucketedScan = plan.collect { case s: FileSourceScanExec if s.bucketedScan => s }
+ assert(bucketedScan.length == expectedNumBucketedScan)
+ }
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "true") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanEnabled)
+ val result = sql(query).collect()
+
+ withSQLConf(SQLConf.AUTO_BUCKETED_SCAN_ENABLED.key -> "false") {
+ checkNumBucketedScan(query, expectedNumScanWithAutoScanDisabled)
+ checkAnswer(sql(query), result)
+ }
+ }
+ }
+
+ test("SPARK-32859: disable unnecessary bucketed table scan - basic test") {
+ withTable("t1", "t2", "t3", "t4") {
+ df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
+ df2.write.format("parquet").bucketBy(8, "i").saveAsTable("t2")
+ df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t3")
+
+ Seq(
+ // Read bucketed table
+ ("SELECT * FROM t1", 0, 1),
+ ("SELECT i FROM t1", 0, 1),
+ ("SELECT j FROM t1", 0, 0),
+ // Filter on bucketed column
+ ("SELECT * FROM t1 WHERE i = 1", 1, 1),
Review comment:
Oh, I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r492665836
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/PlanBucketing.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.bucketing
+
+import org.apache.spark.sql.catalyst.expressions.aggregate.{Partial, PartialMerge}
+import org.apache.spark.sql.catalyst.plans.physical.{ClusteredDistribution, HashClusteredDistribution}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec, SortExec, SparkPlan}
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.execution.exchange.Exchange
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Plans bucketing dynamically based on actual physical query plan.
+ * NOTE: this rule is designed to be applied right after [[EnsureRequirements]],
+ * where all [[ShuffleExchangeExec]] and [[SortExec]] have been added to plan properly.
+ *
+ * When BUCKETING_ENABLED and DYNAMIC_DECIDE_BUCKETING_ENABLED are set to true, go through
+ * query plan to check where bucketed table scan is unnecessary, and disable bucketed table
+ * scan if needed.
+ *
+ * For all operators which [[hasInterestingPartition]] (i.e., require [[ClusteredDistribution]]
+ * or [[HashClusteredDistribution]]), check if the sub-plan for operator has [[Exchange]] and
Review comment:
The above quey is not valid, I just put it here as an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org