You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/18 09:05:54 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

HyukjinKwon opened a new pull request #28569:
URL: https://github.com/apache/spark/pull/28569


   ### What changes were proposed in this pull request?
   
   This PR proposes:
   
   1.. Document the new `pyspark.resource` module introduced at https://github.com/apache/spark/commit/95aec091e4d8a45e648ce84d32d912f585eeb151#diff-23eeeb4347bdd26bfc6b7ee9a3b755dd
   
   2.. Move classes into fewer modules
   
   Before:
   
   ```
   pyspark
   ├── resource
   │   ├── executorrequests.py
   │   │   ├── class ExecutorResourceRequest 
   │   │   └── class ExecutorResourceRequests
   │   ├── taskrequests.py
   │   │   ├── class TaskResourceRequest 
   │   │   └── class TaskResourceRequests
   │   ├── resourceprofilebuilder.py
   │   │   └── class ResourceProfileBuilder
   │   ├── resourceprofile.py
   │   │   └── class ResourceProfile
   └── resourceinformation
       └── class ResourceInformation
   ```
   
   After:
   
   ```
   pyspark
   └── resource
       ├── requests.py
       │   ├── class ExecutorResourceRequest 
       │   ├── class ExecutorResourceRequests
       │   ├── class TaskResourceRequest 
       │   └── class TaskResourceRequests
       ├── profile.py
       │   ├── class ResourceProfileBuilder
       │   └── class ResourceProfile
       └── information.py
           └── class ResourceInformation
   ```
   
   3.. Minor docstring fix e.g.:
   
   ```diff
   -     @param name the name of the resource
   -     @param addresses an array of strings describing the addresses of the resource
   +     :param name: the name of the resource
   +     :param addresses: an array of strings describing the addresses of the resource
   +     .. versionadded:: 3.0.0
   ```
   
   ### Why are the changes needed?
   
   To document APIs, and move Python modules to fewer and simpler modules.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No, the changes are in unreleased branches.
   
   ### How was this patch tested?
   
   Manually tested via:
   
   ```bash
   cd python
   ./run-tests --python-executables=python3 --modules=pyspark-core
   ./run-tests --python-executables=python3 --modules=pyspark-resource
   ```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630543560






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28569:
URL: https://github.com/apache/spark/pull/28569#discussion_r426478018



##########
File path: python/pyspark/resource/information.py
##########
@@ -26,8 +26,10 @@ class ResourceInformation(object):
 
     One example is GPUs, where the addresses would be the indices of the GPUs
 
-    @param name the name of the resource
-    @param addresses an array of strings describing the addresses of the resource
+    :param name: the name of the resource
+    :param addresses: an array of strings describing the addresses of the resource
+
+    .. versionadded:: 3.0.0

Review comment:
       Will open a PR to backport for this change specifically for branch-3.0. This PR itself should go to master only.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630080038






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630054655






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630522970


   **[Test build #122824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122824/testReport)** for PR 28569 at commit [`6beeca7`](https://github.com/apache/spark/commit/6beeca79435cbfe6c1acecf8eb898efe84faf1f4).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630079086


   **[Test build #122798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122798/testReport)** for PR 28569 at commit [`6fa1ab4`](https://github.com/apache/spark/commit/6fa1ab470be6f6851aaf8f7acf7437cd930c172a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630530241






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630523152






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630057408






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28569:
URL: https://github.com/apache/spark/pull/28569#discussion_r426478018



##########
File path: python/pyspark/resource/information.py
##########
@@ -26,8 +26,10 @@ class ResourceInformation(object):
 
     One example is GPUs, where the addresses would be the indices of the GPUs
 
-    @param name the name of the resource
-    @param addresses an array of strings describing the addresses of the resource
+    :param name: the name of the resource
+    :param addresses: an array of strings describing the addresses of the resource
+
+    .. versionadded:: 3.0.0

Review comment:
       Will add a PR to backport this change specifically for branch-3.0. This PR itself should go to master only.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630080038






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630052816


   **[Test build #122797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122797/testReport)** for PR 28569 at commit [`83de0db`](https://github.com/apache/spark/commit/83de0db8ffc49c21c08ed0291e09ff35b02768b9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #28569: [SPARK-31748][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #28569:
URL: https://github.com/apache/spark/pull/28569


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28569: [SPARK-31748][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28569:
URL: https://github.com/apache/spark/pull/28569#discussion_r427669935



##########
File path: python/pyspark/resource/information.py
##########
@@ -26,8 +26,10 @@ class ResourceInformation(object):
 
     One example is GPUs, where the addresses would be the indices of the GPUs
 
-    @param name the name of the resource
-    @param addresses an array of strings describing the addresses of the resource
+    :param name: the name of the resource
+    :param addresses: an array of strings describing the addresses of the resource
+
+    .. versionadded:: 3.0.0

Review comment:
       Please make a PR to branch-3.0 for this part.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630513352






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630512946


   **[Test build #122824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122824/testReport)** for PR 28569 at commit [`6beeca7`](https://github.com/apache/spark/commit/6beeca79435cbfe6c1acecf8eb898efe84faf1f4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630529795


   **[Test build #122825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122825/testReport)** for PR 28569 at commit [`6beeca7`](https://github.com/apache/spark/commit/6beeca79435cbfe6c1acecf8eb898efe84faf1f4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630054655


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630056634


   **[Test build #122798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122798/testReport)** for PR 28569 at commit [`6fa1ab4`](https://github.com/apache/spark/commit/6fa1ab470be6f6851aaf8f7acf7437cd930c172a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630530241






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630054638


   **[Test build #122797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122797/testReport)** for PR 28569 at commit [`83de0db`](https://github.com/apache/spark/commit/83de0db8ffc49c21c08ed0291e09ff35b02768b9).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630054672


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122797/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630523156


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122824/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630053374






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630512946


   **[Test build #122824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122824/testReport)** for PR 28569 at commit [`6beeca7`](https://github.com/apache/spark/commit/6beeca79435cbfe6c1acecf8eb898efe84faf1f4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630523152


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630513352






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630543560






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630056634


   **[Test build #122798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122798/testReport)** for PR 28569 at commit [`6fa1ab4`](https://github.com/apache/spark/commit/6fa1ab470be6f6851aaf8f7acf7437cd930c172a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630527564


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630053374






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630529795


   **[Test build #122825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122825/testReport)** for PR 28569 at commit [`6beeca7`](https://github.com/apache/spark/commit/6beeca79435cbfe6c1acecf8eb898efe84faf1f4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630057408






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630049584


   cc @holdenk too - seems I can't add you as a reviewer.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28569: [SPARK-31748][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-631165795


   Thank you all!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28569:
URL: https://github.com/apache/spark/pull/28569#discussion_r426952356



##########
File path: python/pyspark/resource/requests.py
##########
@@ -167,3 +166,89 @@ def requests(self):
             return result
         else:
             return self._executor_resources
+
+
+class TaskResourceRequest(object):
+    """
+    .. note:: Evolving
+
+    A task resource request. This is used in conjuntion with the
+    :class:`pyspark.resource.ResourceProfile` to programmatically specify the resources
+    needed for an RDD that will be applied at the stage level. The amount is specified
+    as a Double to allow for saying you want more then 1 task per resource. Valid values

Review comment:
       typo: more then -> more than




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630543079


   **[Test build #122825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122825/testReport)** for PR 28569 at commit [`6beeca7`](https://github.com/apache/spark/commit/6beeca79435cbfe6c1acecf8eb898efe84faf1f4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28569:
URL: https://github.com/apache/spark/pull/28569#discussion_r426952135



##########
File path: python/pyspark/resource/profile.py
##########
@@ -15,10 +15,61 @@
 # limitations under the License.
 #
 
-from pyspark.resource.executorrequests import ExecutorResourceRequest,\
-    ExecutorResourceRequests
-from pyspark.resource.resourceprofile import ResourceProfile
-from pyspark.resource.taskrequests import TaskResourceRequest, TaskResourceRequests
+from pyspark.resource.requests import TaskResourceRequest, TaskResourceRequests, \
+    ExecutorResourceRequests, ExecutorResourceRequest
+
+
+class ResourceProfile(object):
+
+    """
+    .. note:: Evolving
+
+    Resource profile to associate with an RDD. A :class:`pyspark.resource.ResourceProfile`
+    allows the user to specify executor and task requirements for an RDD that will get
+    applied during a stage. This allows the user to change the resource requirements between
+    stages. This is meant to be immutable so user doesn't change it after building.

Review comment:
       nit: doesn't change -> cannot change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28569: [SPARK-31748][CORE][PYTHON] Document resource module in PySpark doc and rename/move classes

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28569:
URL: https://github.com/apache/spark/pull/28569#issuecomment-630052816


   **[Test build #122797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122797/testReport)** for PR 28569 at commit [`83de0db`](https://github.com/apache/spark/commit/83de0db8ffc49c21c08ed0291e09ff35b02768b9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org