You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/05 19:14:40 UTC

[GitHub] [spark] ueshin opened a new pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

ueshin opened a new pull request #28740:
URL: https://github.com/apache/spark/pull/28740


   ### What changes were proposed in this pull request?
   
   This is a backport of #28730.
   
   In `Dataset.collectAsArrowToPython`, since the code block for `serveToStream` is run in the separate thread, `withAction` finishes as soon as it starts the thread. As a result, it doesn't collect the metrics of the actual action and Query UI shows the plan graph without metrics.
   
   We should call `serveToStream` first, then `withAction` in it.
   
   ### Why are the changes needed?
   
   When calling toPandas, usually Query UI shows each plan node's metric:
   
   ```py
   >>> df = spark.createDataFrame([(1, 10, 'abc'), (2, 20, 'def')], schema=['x', 'y', 'z'])
   >>> df.toPandas()
      x   y    z
   0  1  10  abc
   1  2  20  def
   ```
   
   ![Screen Shot 2020-06-05 at 10 58 30 AM](https://user-images.githubusercontent.com/506656/83914110-6f3b3080-a725-11ea-88c0-de83a833b05c.png)
   
   but if Arrow execution is enabled, it shows only plan nodes and the duration is not correct:
   
   ```py
   >>> spark.conf.set('spark.sql.execution.arrow.enabled', True)
   >>> df.toPandas()
      x   y    z
   0  1  10  abc
   1  2  20  def
   ```
   
   ![Screen Shot 2020-06-05 at 10 58 42 AM](https://user-images.githubusercontent.com/506656/83914127-782c0200-a725-11ea-84e4-74d861d5c20a.png)
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, the Query UI will show the plan with the correct metrics.
   
   ### How was this patch tested?
   
   I checked it manually in my local.
   
   ![Screen Shot 2020-06-05 at 11 29 48 AM](https://user-images.githubusercontent.com/506656/83914142-7e21e300-a725-11ea-8925-edc22df16388.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #28740:
URL: https://github.com/apache/spark/pull/28740


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639866391






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639865831


   **[Test build #123579 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123579/testReport)** for PR 28740 at commit [`753a3b7`](https://github.com/apache/spark/commit/753a3b7e1e0697b7eadcfe1ed3eb4b6628ac918a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639739435






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-640006886


   Merged to branch-2.4.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-640007031


   cc @holdenk FYI


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639738076


   **[Test build #123579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123579/testReport)** for PR 28740 at commit [`753a3b7`](https://github.com/apache/spark/commit/753a3b7e1e0697b7eadcfe1ed3eb4b6628ac918a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639738076


   **[Test build #123579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123579/testReport)** for PR 28740 at commit [`753a3b7`](https://github.com/apache/spark/commit/753a3b7e1e0697b7eadcfe1ed3eb4b6628ac918a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639866391






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28740: [SPARK-31903][SQL][PYSPARK][2.4] Fix toPandas with Arrow enabled to show metrics in Query UI.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28740:
URL: https://github.com/apache/spark/pull/28740#issuecomment-639739435






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org