You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/23 09:31:44 UTC

[GitHub] [spark] manuzhang opened a new pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

manuzhang opened a new pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983
 
 
   ### What changes were proposed in this pull request?
   
   Set `isFinalPlan=true` before posting the final AdaptiveSparkPlan event (`SparkListenerSQLAdaptiveExecutionUpdate`)
   
   ### Why are the changes needed?
   
   Otherwise, any attempt to listen on the final event by pattern matching `isFinalPlan=true` would fail
   
   ### Does this PR introduce any user-facing change?
   
   No.
   
   
   ### How was this patch tested?
   
   All tests in `AdaptiveQueryExecSuite` are exteneded with a verification that a `SparkListenerSQLAdaptiveExecutionUpdate` event with `isFinalPlan=True` exists
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568677399
 
 
   **[Test build #115682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115682/testReport)** for PR 26983 at commit [`66b2ddc`](https://github.com/apache/spark/commit/66b2ddceef7871a5e726a26f4a03dea5cac03aed).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568693377
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20507/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640215
 
 
   **[Test build #115682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115682/testReport)** for PR 26983 at commit [`66b2ddc`](https://github.com/apache/spark/commit/66b2ddceef7871a5e726a26f4a03dea5cac03aed).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568576092
 
 
   Can one of the admins verify this patch?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568747314
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568693377
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20507/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568693371
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568677776
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r361081996
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 ##########
 @@ -39,6 +40,20 @@ class AdaptiveQueryExecSuite
   setupTestData()
 
   private def runAdaptiveAndVerifyResult(query: String): (SparkPlan, SparkPlan) = {
+    var finalPlanExists = false
 
 Review comment:
   now we check the final plan only exists once

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568575467
 
 
   Can one of the admins verify this patch?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r361042667
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 ##########
 @@ -39,6 +40,20 @@ class AdaptiveQueryExecSuite
   setupTestData()
 
   private def runAdaptiveAndVerifyResult(query: String): (SparkPlan, SparkPlan) = {
+    var finalPlanExists = false
+    val listener = new SparkListener {
+      override def onOtherEvent(event: SparkListenerEvent): Unit = {
+        event match {
+          case SparkListenerSQLAdaptiveExecutionUpdate(_, _, sparkPlanInfo) =>
+            if (sparkPlanInfo.simpleString.startsWith("AdaptiveSparkPlan(isFinalPlan=true)")) {
 
 Review comment:
   `SparkPlanInfo` has lost the type info. Any other way to check ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568576092
 
 
   Can one of the admins verify this patch?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568747322
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115716/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568694963
 
 
   **[Test build #115716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115716/testReport)** for PR 26983 at commit [`ae33f82`](https://github.com/apache/spark/commit/ae33f82e0c5c452f7a5191914942bbaf534637ec).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640488
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568677778
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115682/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568886865
 
 
   thanks, merging to master!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640215
 
 
   **[Test build #115682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115682/testReport)** for PR 26983 at commit [`66b2ddc`](https://github.com/apache/spark/commit/66b2ddceef7871a5e726a26f4a03dea5cac03aed).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568677778
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115682/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568747314
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568677776
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568693371
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r360857528
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 ##########
 @@ -39,6 +40,20 @@ class AdaptiveQueryExecSuite
   setupTestData()
 
   private def runAdaptiveAndVerifyResult(query: String): (SparkPlan, SparkPlan) = {
+    var finalPlanExists = false
+    val listener = new SparkListener {
+      override def onOtherEvent(event: SparkListenerEvent): Unit = {
+        event match {
+          case SparkListenerSQLAdaptiveExecutionUpdate(_, _, sparkPlanInfo) =>
+            if (sparkPlanInfo.simpleString.startsWith("AdaptiveSparkPlan(isFinalPlan=true)")) {
 
 Review comment:
   can we make it more type safe? e.g. check if it's a `AdaptiveSparkPlanExec`, and check the `isFinalPlan` flag.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640495
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20479/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640488
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r361103015
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 ##########
 @@ -39,6 +40,25 @@ class AdaptiveQueryExecSuite
   setupTestData()
 
   private def runAdaptiveAndVerifyResult(query: String): (SparkPlan, SparkPlan) = {
+    var finalPlanExistsOnce: Option[Boolean] = None
+    val listener = new SparkListener {
+      override def onOtherEvent(event: SparkListenerEvent): Unit = {
+        event match {
+          case SparkListenerSQLAdaptiveExecutionUpdate(_, _, sparkPlanInfo) =>
+            if (sparkPlanInfo.simpleString.startsWith(
+              "AdaptiveSparkPlan(isFinalPlan=true)")) {
+              if (finalPlanExistsOnce.isDefined) {
+                finalPlanExistsOnce = Some(false)
 
 Review comment:
   exactly

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568575467
 
 
   Can one of the admins verify this patch?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568638822
 
 
   @maryannxue FYI

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640026
 
 
   ok to test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r360857625
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 ##########
 @@ -39,6 +40,20 @@ class AdaptiveQueryExecSuite
   setupTestData()
 
   private def runAdaptiveAndVerifyResult(query: String): (SparkPlan, SparkPlan) = {
+    var finalPlanExists = false
 
 Review comment:
   should we count it and make sure the event appears once and only once?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568640495
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20479/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r361085709
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 ##########
 @@ -39,6 +40,25 @@ class AdaptiveQueryExecSuite
   setupTestData()
 
   private def runAdaptiveAndVerifyResult(query: String): (SparkPlan, SparkPlan) = {
+    var finalPlanExistsOnce: Option[Boolean] = None
+    val listener = new SparkListener {
+      override def onOtherEvent(event: SparkListenerEvent): Unit = {
+        event match {
+          case SparkListenerSQLAdaptiveExecutionUpdate(_, _, sparkPlanInfo) =>
+            if (sparkPlanInfo.simpleString.startsWith(
+              "AdaptiveSparkPlan(isFinalPlan=true)")) {
+              if (finalPlanExistsOnce.isDefined) {
+                finalPlanExistsOnce = Some(false)
 
 Review comment:
   isn't it much simpler to use an int count?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568747322
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115716/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568747041
 
 
   **[Test build #115716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115716/testReport)** for PR 26983 at commit [`ae33f82`](https://github.com/apache/spark/commit/ae33f82e0c5c452f7a5191914942bbaf534637ec).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26983: [SPARK-30331][SQL]  Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#issuecomment-568694963
 
 
   **[Test build #115716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115716/testReport)** for PR 26983 at commit [`ae33f82`](https://github.com/apache/spark/commit/ae33f82e0c5c452f7a5191914942bbaf534637ec).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org