You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/02 11:02:10 UTC
[GitHub] [spark] beliefer opened a new pull request #34779: [SPARK-37518] inject a early scan pushdown rule
beliefer opened a new pull request #34779:
URL: https://github.com/apache/spark/pull/34779
### What changes were proposed in this pull request?
Currently, Spark supports push down filters, aggregates and limit. All the job is completed by `V2ScanRelationPushDown`.
But `V2ScanRelationPushDown` have a lot limit.
Users want apply custom rule for push down after `V2ScanRelationPushDown` failed.
### Why are the changes needed?
Easy for users to apply custom pushdown rules.
### Does this PR introduce _any_ user-facing change?
'Yes'.
Users can inject custom early scan pushdown rules.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764564803
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
* <li>Check Analysis Rules.</li>
* <li>Optimizer Rules.</li>
* <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>
Review comment:
I feel that the name of `Pre-CBO Rules` is too large and the meaning in the [comment](https://github.com/apache/spark/blob/9596820a473241928b198c1057f4ac662c77dc08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L212) is not the same. Can't limit the flexibility because of this name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984628542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50329/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985139936
**[Test build #145870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145870/testReport)** for PR 34779 at commit [`746278a`](https://github.com/apache/spark/commit/746278a8668bb996f0cbf877b8f02ef4561cc3c2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984574210
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50329/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984775790
**[Test build #145854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145854/testReport)** for PR 34779 at commit [`a3bc014`](https://github.com/apache/spark/commit/a3bc014d8a8d88a11d4909e13605c9b44786c3ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985395023
cc @cloud-fan and @maryannxue FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r761573837
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -95,6 +95,12 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
}
}
+ test("SPARK-37518: inject a early scan push down rule") {
Review comment:
Just references https://github.com/apache/spark/blob/a3bc014d8a8d88a11d4909e13605c9b44786c3ca/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala#L92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985186469
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984790229
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984542641
**[Test build #145854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145854/testReport)** for PR 34779 at commit [`a3bc014`](https://github.com/apache/spark/commit/a3bc014d8a8d88a11d4909e13605c9b44786c3ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-988032869
**[Test build #145986 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145986/testReport)** for PR 34779 at commit [`9596820`](https://github.com/apache/spark/commit/9596820a473241928b198c1057f4ac662c77dc08).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985182073
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985120724
> LGTM(non-binding)
>
> Generally speaking, it is useful. @beliefer Can you give a simple example to show the actual user case?
Spark SQL supports aggregate pushdown only for standard aggregate function. But some databases have some non-standard aggregate function, This PR open a door for flexible customization.
Some databases if good for the other different pushdown type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985120724
> LGTM(non-binding)
>
> Generally speaking, it is useful. @beliefer Can you give a simple example to show the actual user case?
Spark SQL supports aggregate pushdown only for standard aggregate function. But some databases have some non-standard aggregate function, This PR open a door for flexible customization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34779: [SPARK-37518][SQL] Inject an early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-1078530010
OK. Let me close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer closed pull request #34779: [SPARK-37518][SQL] Inject an early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer closed pull request #34779:
URL: https://github.com/apache/spark/pull/34779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r761472725
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -95,6 +95,12 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
}
}
+ test("SPARK-37518: inject a early scan push down rule") {
Review comment:
nit: I think we only require JIRA id for bug fixes and regressions?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -226,6 +226,21 @@ class SparkSessionExtensions {
preCBORules += builder
}
+ private[this] val earlyScanPushDownRules = mutable.Buffer.empty[RuleBuilder]
+
+ private[sql] def buildEarlyScanPushDownRules(session: SparkSession): Seq[Rule[LogicalPlan]] = {
+ earlyScanPushDownRules.map(_.apply(session)).toSeq
+ }
+
+ /**
+ * Inject an optimizer `Rule` builder that rewrites logical plans into the [[SparkSession]].
+ * The injected rules will be executed once after the operator optimization batch and
+ * after any push down optimization rules.
+ */
+ def injectEarlyScanPushDownRules(builder: RuleBuilder): Unit = {
Review comment:
nit: `injectEarlyScanPushDownRules` -> `injectEarlyScanPushDownRule`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985267810
**[Test build #145870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145870/testReport)** for PR 34779 at commit [`746278a`](https://github.com/apache/spark/commit/746278a8668bb996f0cbf877b8f02ef4561cc3c2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r763135859
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -226,6 +226,21 @@ class SparkSessionExtensions {
preCBORules += builder
}
+ private[this] val earlyScanPushDownRules = mutable.Buffer.empty[RuleBuilder]
+
+ private[sql] def buildEarlyScanPushDownRules(session: SparkSession): Seq[Rule[LogicalPlan]] = {
+ earlyScanPushDownRules.map(_.apply(session)).toSeq
+ }
+
+ /**
+ * Inject an optimizer `Rule` builder that rewrites logical plans into the [[SparkSession]].
+ * The injected rules will be executed once after the operator optimization batch and
+ * after any push down optimization rules.
+ */
+ def injectEarlyScanPushDownRule(builder: RuleBuilder): Unit = {
Review comment:
Let's update the classdoc, which only mentions
```
* This current provides the following extension points:
*
* <ul>
* <li>Analyzer Rules.</li>
* <li>Check Analysis Rules.</li>
* <li>Optimizer Rules.</li>
* <li>Pre CBO Rules.</li>
* <li>Planning Strategies.</li>
* <li>Customized Parser.</li>
* <li>(External) Catalog listeners.</li>
* <li>Columnar Rules.</li>
* <li>Adaptive Query Stage Preparation Rules.</li>
* </ul>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-988051347
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145986/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984615070
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50329/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984790229
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985139936
**[Test build #145870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145870/testReport)** for PR 34779 at commit [`746278a`](https://github.com/apache/spark/commit/746278a8668bb996f0cbf877b8f02ef4561cc3c2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-986907143
Since we already have this extension point in`BaseSessionStateBuilder`, I'm OK to expose it in the developer API. My only concern is we should document it clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r763136238
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -226,6 +226,21 @@ class SparkSessionExtensions {
preCBORules += builder
}
+ private[this] val earlyScanPushDownRules = mutable.Buffer.empty[RuleBuilder]
+
+ private[sql] def buildEarlyScanPushDownRules(session: SparkSession): Seq[Rule[LogicalPlan]] = {
+ earlyScanPushDownRules.map(_.apply(session)).toSeq
+ }
+
+ /**
+ * Inject an optimizer `Rule` builder that rewrites logical plans into the [[SparkSession]].
+ * The injected rules will be executed once after the operator optimization batch and
+ * after any push down optimization rules.
+ */
+ def injectEarlyScanPushDownRule(builder: RuleBuilder): Unit = {
Review comment:
We also need to clarify how is this different from `Pre CBO Rules`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987883857
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50462/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764156569
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
* <li>Check Analysis Rules.</li>
* <li>Optimizer Rules.</li>
* <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>
Review comment:
After so many discussions in https://github.com/apache/spark/pull/30808 , I'm really worried about the naming of this new extension point.
In general, this new extension point allows people to inject custom data source operator pushdown rules, which run after the built-in ones. But then the existing `Pre CBO rules` becomes a confusing name, as the pushdown rules are also pre-CBO.
We may need more time to think about the naming, or think if really need custom data source pushdown rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764564803
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
* <li>Check Analysis Rules.</li>
* <li>Optimizer Rules.</li>
* <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>
Review comment:
I feel that the name of `Pre-CBO Rules` is too wide and the meaning in the [comment](https://github.com/apache/spark/blob/9596820a473241928b198c1057f4ac662c77dc08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L212) is not the same. Can't limit the flexibility because of this name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987895054
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50462/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987895054
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50462/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987804269
**[Test build #145986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145986/testReport)** for PR 34779 at commit [`9596820`](https://github.com/apache/spark/commit/9596820a473241928b198c1057f4ac662c77dc08).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984628542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50329/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985280260
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985168763
Thanks for your explanation @beliefer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985186469
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764564803
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
* <li>Check Analysis Rules.</li>
* <li>Optimizer Rules.</li>
* <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>
Review comment:
I feel that the name of `Pre-CBO Rules` is too wide and the meaning in the [comment](https://github.com/apache/spark/blob/9596820a473241928b198c1057f4ac662c77dc08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L212) is not the same. In my opinion we should not limit the flexibility because of this name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987775551
> Since we already have this extension point in`BaseSessionStateBuilder`, I'm OK to expose it in the developer API. My only concern is we should document it clearly.
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-988051347
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145986/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987804269
**[Test build #145986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145986/testReport)** for PR 34779 at commit [`9596820`](https://github.com/apache/spark/commit/9596820a473241928b198c1057f4ac662c77dc08).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987851080
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50462/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984542641
**[Test build #145854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145854/testReport)** for PR 34779 at commit [`a3bc014`](https://github.com/apache/spark/commit/a3bc014d8a8d88a11d4909e13605c9b44786c3ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985161213
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985280260
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #34779: [SPARK-37518][SQL] Inject an early scan pushdown rule
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-1078524146
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org