You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/03 01:16:53 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

HyukjinKwon opened a new pull request #28711:
URL: https://github.com/apache/spark/pull/28711


   ### What changes were proposed in this pull request?
   
   Scala:
   
   ```scala
   scala> spark.range(10).explain("cost")
   ```
   ```
   == Optimized Logical Plan ==
   Range (0, 10, step=1, splits=Some(12)), Statistics(sizeInBytes=80.0 B)
   
   == Physical Plan ==
   *(1) Range (0, 10, step=1, splits=12)
   ```
   
   PySpark:
   
   ```python
   >>> spark.range(10).explain("cost")
   ```
   ```
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/.../spark/python/pyspark/sql/dataframe.py", line 333, in explain
       raise TypeError(err_msg)
   TypeError: extended (optional) should be provided as bool, got <class 'str'>
   ```
   
   This is also consistent with `DataFrame.sample` case.
   
   ### Why are the changes needed?
   
   To provide the consistent API support across APIs.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Nope, it's only changes in unreleased branches.
   If this lands to master only, yes, users will be able to set `mode` as `df.explain("...")` in Spark 3.1.
   
   ### How was this patch tested?
   
   Unittest was added and manually tested as well to make sure:
   
   ```python
   spark.range(10).explain(True)
   spark.range(10).explain(False)
   spark.range(10).explain("cost")
   spark.range(10).explain(extended="cost")
   spark.range(10).explain(mode="cost")
   spark.range(10).explain()
   spark.range(10).explain(True, "cost")
   spark.range(10).explain(1.0)
   ```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637898119


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123453/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637899870






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637898071






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637907212


   **[Test build #123454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123454/testReport)** for PR 28711 at commit [`984f33b`](https://github.com/apache/spark/commit/984f33b16b1ecd38cfc746ededa1b4e3601c0ee0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637907495






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637908447


   btw, typo in the description?
   ```
   This is also consistent with DataFrame.sample case.
                                          ^^^^^
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637907495






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637899539


   **[Test build #123454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123454/testReport)** for PR 28711 at commit [`984f33b`](https://github.com/apache/spark/commit/984f33b16b1ecd38cfc746ededa1b4e3601c0ee0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637924733


   Oh, I meant this is also consistent with DataFrame.sample in PySpark and Scala side - supporting `DataFrame.sample(1.0)` and `DataFrame.sample(False)`. Let me clarify.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637897830


   **[Test build #123453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123453/testReport)** for PR 28711 at commit [`8d73c4b`](https://github.com/apache/spark/commit/8d73c4b0fd1a2d41a4a8ec10de60077cec2de82f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637898104


   **[Test build #123453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123453/testReport)** for PR 28711 at commit [`8d73c4b`](https://github.com/apache/spark/commit/8d73c4b0fd1a2d41a4a8ec10de60077cec2de82f).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637899870






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637924733


   Oh, I meant this is also consistent with `DataFrame.sample` in PySpark and Scala side - supporting `DataFrame.sample(1.0)` and `DataFrame.sample(False)`. Let me clarify.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #28711:
URL: https://github.com/apache/spark/pull/28711


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637930643


   hahaha, I see.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637899539


   **[Test build #123454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123454/testReport)** for PR 28711 at commit [`984f33b`](https://github.com/apache/spark/commit/984f33b16b1ecd38cfc746ededa1b4e3601c0ee0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637897830


   **[Test build #123453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123453/testReport)** for PR 28711 at commit [`8d73c4b`](https://github.com/apache/spark/commit/8d73c4b0fd1a2d41a4a8ec10de60077cec2de82f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637898071






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637898114






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28711: [SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28711:
URL: https://github.com/apache/spark/pull/28711#issuecomment-637925187


   Thanks @maropu. Merged to master and branch-3.0!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org