You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/02 11:02:10 UTC

[GitHub] [spark] beliefer opened a new pull request #34779: [SPARK-37518] inject a early scan pushdown rule

beliefer opened a new pull request #34779:
URL: https://github.com/apache/spark/pull/34779


   ### What changes were proposed in this pull request?
   Currently, Spark supports push down filters, aggregates and limit. All the job is completed by `V2ScanRelationPushDown`.
   But `V2ScanRelationPushDown` have a lot limit.
   Users want apply custom rule for push down after `V2ScanRelationPushDown` failed.
   
   
   ### Why are the changes needed?
   Easy for users to apply custom pushdown rules.
   
   
   ### Does this PR introduce _any_ user-facing change?
   'Yes'.
   Users can inject custom early scan pushdown rules.
   
   
   ### How was this patch tested?
   New tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764564803



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
  * <li>Check Analysis Rules.</li>
  * <li>Optimizer Rules.</li>
  * <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>

Review comment:
       I feel that the name of `Pre-CBO Rules` is too large and the meaning in the [comment](https://github.com/apache/spark/blob/9596820a473241928b198c1057f4ac662c77dc08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L212) is not the same. Can't limit the flexibility because of this name.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984628542


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50329/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985139936


   **[Test build #145870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145870/testReport)** for PR 34779 at commit [`746278a`](https://github.com/apache/spark/commit/746278a8668bb996f0cbf877b8f02ef4561cc3c2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984574210


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50329/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984775790


   **[Test build #145854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145854/testReport)** for PR 34779 at commit [`a3bc014`](https://github.com/apache/spark/commit/a3bc014d8a8d88a11d4909e13605c9b44786c3ca).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985395023


   cc @cloud-fan and @maryannxue FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r761573837



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -95,6 +95,12 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
     }
   }
 
+  test("SPARK-37518: inject a early scan push down rule") {

Review comment:
       Just references https://github.com/apache/spark/blob/a3bc014d8a8d88a11d4909e13605c9b44786c3ca/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala#L92




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985186469


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50345/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984790229


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145854/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984542641


   **[Test build #145854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145854/testReport)** for PR 34779 at commit [`a3bc014`](https://github.com/apache/spark/commit/a3bc014d8a8d88a11d4909e13605c9b44786c3ca).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-988032869


   **[Test build #145986 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145986/testReport)** for PR 34779 at commit [`9596820`](https://github.com/apache/spark/commit/9596820a473241928b198c1057f4ac662c77dc08).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985182073


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50345/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer edited a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985120724


   > LGTM(non-binding)
   > 
   > Generally speaking, it is useful. @beliefer Can you give a simple example to show the actual user case?
   
   Spark SQL supports aggregate pushdown only for standard aggregate function. But some databases have some non-standard aggregate function, This PR open a door for flexible customization.
   
   Some databases if good for the other different pushdown type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985120724


   > LGTM(non-binding)
   > 
   > Generally speaking, it is useful. @beliefer Can you give a simple example to show the actual user case?
   
   Spark SQL supports aggregate pushdown only for standard aggregate function. But some databases have some non-standard aggregate function, This PR open a door for flexible customization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #34779: [SPARK-37518][SQL] Inject an early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-1078530010


   OK. Let me close it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer closed pull request #34779: [SPARK-37518][SQL] Inject an early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer closed pull request #34779:
URL: https://github.com/apache/spark/pull/34779


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r761472725



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -95,6 +95,12 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
     }
   }
 
+  test("SPARK-37518: inject a early scan push down rule") {

Review comment:
       nit: I think we only require JIRA id for bug fixes and regressions?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -226,6 +226,21 @@ class SparkSessionExtensions {
     preCBORules += builder
   }
 
+  private[this] val earlyScanPushDownRules = mutable.Buffer.empty[RuleBuilder]
+
+  private[sql] def buildEarlyScanPushDownRules(session: SparkSession): Seq[Rule[LogicalPlan]] = {
+    earlyScanPushDownRules.map(_.apply(session)).toSeq
+  }
+
+  /**
+   * Inject an optimizer `Rule` builder that rewrites logical plans into the [[SparkSession]].
+   * The injected rules will be executed once after the operator optimization batch and
+   * after any push down optimization rules.
+   */
+  def injectEarlyScanPushDownRules(builder: RuleBuilder): Unit = {

Review comment:
       nit: `injectEarlyScanPushDownRules` -> `injectEarlyScanPushDownRule`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985267810


   **[Test build #145870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145870/testReport)** for PR 34779 at commit [`746278a`](https://github.com/apache/spark/commit/746278a8668bb996f0cbf877b8f02ef4561cc3c2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r763135859



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -226,6 +226,21 @@ class SparkSessionExtensions {
     preCBORules += builder
   }
 
+  private[this] val earlyScanPushDownRules = mutable.Buffer.empty[RuleBuilder]
+
+  private[sql] def buildEarlyScanPushDownRules(session: SparkSession): Seq[Rule[LogicalPlan]] = {
+    earlyScanPushDownRules.map(_.apply(session)).toSeq
+  }
+
+  /**
+   * Inject an optimizer `Rule` builder that rewrites logical plans into the [[SparkSession]].
+   * The injected rules will be executed once after the operator optimization batch and
+   * after any push down optimization rules.
+   */
+  def injectEarlyScanPushDownRule(builder: RuleBuilder): Unit = {

Review comment:
       Let's update the classdoc, which only mentions
   ```
    * This current provides the following extension points:
    *
    * <ul>
    * <li>Analyzer Rules.</li>
    * <li>Check Analysis Rules.</li>
    * <li>Optimizer Rules.</li>
    * <li>Pre CBO Rules.</li>
    * <li>Planning Strategies.</li>
    * <li>Customized Parser.</li>
    * <li>(External) Catalog listeners.</li>
    * <li>Columnar Rules.</li>
    * <li>Adaptive Query Stage Preparation Rules.</li>
    * </ul>
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-988051347


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145986/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984615070


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50329/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984790229


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145854/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985139936


   **[Test build #145870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145870/testReport)** for PR 34779 at commit [`746278a`](https://github.com/apache/spark/commit/746278a8668bb996f0cbf877b8f02ef4561cc3c2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-986907143


   Since we already have this extension point in`BaseSessionStateBuilder`, I'm OK to expose it in the developer API. My only concern is we should document it clearly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r763136238



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -226,6 +226,21 @@ class SparkSessionExtensions {
     preCBORules += builder
   }
 
+  private[this] val earlyScanPushDownRules = mutable.Buffer.empty[RuleBuilder]
+
+  private[sql] def buildEarlyScanPushDownRules(session: SparkSession): Seq[Rule[LogicalPlan]] = {
+    earlyScanPushDownRules.map(_.apply(session)).toSeq
+  }
+
+  /**
+   * Inject an optimizer `Rule` builder that rewrites logical plans into the [[SparkSession]].
+   * The injected rules will be executed once after the operator optimization batch and
+   * after any push down optimization rules.
+   */
+  def injectEarlyScanPushDownRule(builder: RuleBuilder): Unit = {

Review comment:
       We also need to clarify how is this different from `Pre CBO Rules`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987883857


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50462/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764156569



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
  * <li>Check Analysis Rules.</li>
  * <li>Optimizer Rules.</li>
  * <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>

Review comment:
       After so many discussions in https://github.com/apache/spark/pull/30808 , I'm really worried about the naming of this new extension point.
   
   In general, this new extension point allows people to inject custom data source operator pushdown rules, which run after the built-in ones. But then the existing `Pre CBO rules` becomes a confusing name, as the pushdown rules are also pre-CBO.
   
   We may need more time to think about the naming, or think if really need custom data source pushdown rules.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764564803



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
  * <li>Check Analysis Rules.</li>
  * <li>Optimizer Rules.</li>
  * <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>

Review comment:
       I feel that the name of `Pre-CBO Rules` is too wide and the meaning in the [comment](https://github.com/apache/spark/blob/9596820a473241928b198c1057f4ac662c77dc08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L212) is not the same. Can't limit the flexibility because of this name.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987895054


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50462/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987895054


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50462/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987804269


   **[Test build #145986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145986/testReport)** for PR 34779 at commit [`9596820`](https://github.com/apache/spark/commit/9596820a473241928b198c1057f4ac662c77dc08).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984628542


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50329/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985280260


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145870/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985168763


   Thanks for your explanation @beliefer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985186469


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50345/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34779:
URL: https://github.com/apache/spark/pull/34779#discussion_r764564803



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala
##########
@@ -42,6 +42,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, SparkPlan}
  * <li>Check Analysis Rules.</li>
  * <li>Optimizer Rules.</li>
  * <li>Pre CBO Rules.</li>
+ * <li>Early Scan Push-Down</li>

Review comment:
       I feel that the name of `Pre-CBO Rules` is too wide and the meaning in the [comment](https://github.com/apache/spark/blob/9596820a473241928b198c1057f4ac662c77dc08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L212) is not the same. In my opinion we should not limit the flexibility because of this name.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987775551


   > Since we already have this extension point in`BaseSessionStateBuilder`, I'm OK to expose it in the developer API. My only concern is we should document it clearly.
   
   OK


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-988051347


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145986/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987804269


   **[Test build #145986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145986/testReport)** for PR 34779 at commit [`9596820`](https://github.com/apache/spark/commit/9596820a473241928b198c1057f4ac662c77dc08).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-987851080


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50462/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-984542641


   **[Test build #145854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145854/testReport)** for PR 34779 at commit [`a3bc014`](https://github.com/apache/spark/commit/a3bc014d8a8d88a11d4909e13605c9b44786c3ca).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985161213


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50345/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34779: [SPARK-37518][SQL] Inject a early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-985280260


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145870/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] github-actions[bot] commented on pull request #34779: [SPARK-37518][SQL] Inject an early scan pushdown rule

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #34779:
URL: https://github.com/apache/spark/pull/34779#issuecomment-1078524146


   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org