You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/16 17:38:15 UTC

[GitHub] [spark] nchammas opened a new pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

nchammas opened a new pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928
 
 
   ### What changes were proposed in this pull request?
   
   This PR refactors how we track various Python test and build dependencies so they are pinned and internally consistent.
   
   ### Why are the changes needed?
   
   This should make it easier to bump dependencies (since they are specified in fewer places) and promote more consistent build behavior across Docker, Jenkins, GitHub, and local development environments.
   
   ### Does this PR introduce any user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   I built the Docker image just far enough to confirm that the dependencies are installed correctly:
   
   ```
   Step 16/31 : RUN pip install -r docs-requirements.txt
    ---> Running in b0c66cb2a543
   Collecting sphinx==2.3.1
     Downloading Sphinx-2.3.1-py3-none-any.whl (2.7 MB)
   Collecting mkdocs==1.0.4
     Downloading mkdocs-1.0.4-py2.py3-none-any.whl (1.2 MB)
   Collecting numpy==1.18.1
     Downloading numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
   ...
   ```
   
   The GitHub workflow will be tested as part of this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599792019
 
 
   **[Test build #119896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119896/testReport)** for PR 27928 at commit [`9c3a7af`](https://github.com/apache/spark/commit/9c3a7af3ce469e375a5141d779d288442fc5675a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601880695
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24830/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601939323
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615304827
 
 
   I manually copied the pinned requirements to the required location and kicked off the Docker build as follows (basically, trying to mimic parts of `do-release-docker.sh`):
   
   ```
   cp .../requirements-pinned.txt spark-rm/dev-requirements-pinned.txt
   docker build --build-arg UID=$UID ./spark-rm/
   ```
   
   It ran all the way through successfully on my laptop.
   
   @HyukjinKwon and @cloud-fan - Are you still interested in this PR? Would either of you be able to test `do-release-docker.sh`?
   
   If you'd prefer, I can back out any changes to the release machinery, just to be conservative. It's not ideal, but if that would help get this PR over the line, then I'll do it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607412613
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395133004
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   It will. I am good with using the virtualenv in CI and docker image. I use Conda too in my local.
   
   My concern is that pinning versions will either force the dev people to use virtualenv (or its friends) or potentially mess their installed packages.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601877338
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607569137
 
 
   **[Test build #120698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120698/testReport)** for PR 27928 at commit [`b088d15`](https://github.com/apache/spark/commit/b088d151c6102e13f124318d8b14e9279da3287a).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599792386
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600260552
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24674/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395130948
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   What about just using a virtualenv to isolate the effect on installed libraries - wouldn't that solve that part?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600348307
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599792019
 
 
   **[Test build #119896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119896/testReport)** for PR 27928 at commit [`9c3a7af`](https://github.com/apache/spark/commit/9c3a7af3ce469e375a5141d779d288442fc5675a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600348307
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599792391
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24622/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600348311
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119950/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395852390
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   Hmm, on second thought, perhaps the README should be left to a separate PR.  We already have advice on setting up a development environment in at least a couple of places, like [Useful Developer Tools](http://spark.apache.org/developer-tools.html) and [`docs/README.md`](https://github.com/apache/spark/blob/fae981e5f32e0ddb86591a616829423dfafb4ed0/docs/README.md#using-rbenv-and-pyenv).
   
   Perhaps we should consolidate that advice over on the Useful Developer Tools page, since it fits it with the information already there.
   
   Either that, or let's agree on some other approach to take. But I think we can defer any dev documentation changes to a follow-up PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing fixed in the upstream. It will still mess developer's environment. Arguably there are not so many Python-dedicated dev people in Spark community who fluently use pyenv, virtualenv or conda. _I think_ most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found 6 instances.
   
   - 3 of them were not quite pinned.
   - 2 of them were partially pinned
   - 1 of them was completely pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615285850
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26097/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600181974
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599792386
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607930617
 
 
   Following a common pattern in Python projects (e.g. Warehouse, Trio, my own Flintrock), I'm specifying broad development dependencies and then using pip-tools to compile them down to a pinned list of requirements.
   
   The broad requirements are for contributors to use locally, and the pinned requirements are for use by CI and our release tooling.
   
   I believe this should make everyone happy. 😄

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393721061
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   @HyukjinKwon - When looking at project dependencies, there is an important distinction between projects that are used as libraries and projects that are used as stand-alone applications.
   
   If your project is a library, then you know others are importing you alongside other dependencies too. To minimize the chance of transitive dependency conflicts, you want to be flexible in how you specify your dependencies.
   
   When your project is a stand-alone application, you don't have to worry about such things. You can pin every dependency to a specific version to get the most predictable and reliable build and runtime behavior.
   
   In our case, the Spark build environment is more akin to a stand-alone application than a library. We don't need to worry about downstream users struggling with dependency conflicts. We can get the most stable build behavior by pinning everything, and there is no downside as far as I can tell.
   
   I'll use [Trio](https://github.com/python-trio/trio) as an example again to illustrate my point:
   * Trio is a library that others will typically import alongside many other dependencies. So in [Trio's setup.py](https://github.com/python-trio/trio/blob/4d956a4ba51241ca5d22800fb1e0e5c36ba9bb47/setup.py#L81-L92) they are very flexible in how they specify their dependencies.
   * Trio's test environment, on the other hand, is only used by Trio contributors. So Trio [locks down](https://github.com/python-trio/trio/blob/4d956a4ba51241ca5d22800fb1e0e5c36ba9bb47/test-requirements.txt) every test requirement using [pip-tools](https://github.com/jazzband/pip-tools).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601880057
 
 
   **[Test build #120116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120116/testReport)** for PR 27928 at commit [`75682a6`](https://github.com/apache/spark/commit/75682a6b0023f22f4a030d4143913db2a0021597).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607537436
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600113207
 
 
   I'm still pretty OK with pinning, but, sure, allow taking later maintenance releases?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-605660296
 
 
   @BryanCutler @HyukjinKwon I'm not sure where the consensus lands here. I think part of your argument is you _do_ want to take some breakage that comes with automatic version updates. That seems a bit less valuable than relative stability, to me. I suppose I end up neutral and we have some slight preferences for and against then? Is there any change here with broader support?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599829073
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601880695
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24830/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607412622
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25384/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600260545
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r396197611
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,10 @@
-flake8==3.5.0
+flake8==3.7.*
 
 Review comment:
   I mentioned it elsewhere but I'll mention it again here: Linters like flake8 and pycodestyle introduce new checks in minor/feature releases. There is very high chance that every new check they introduce will flag new problems and fail the build. (In fact, we saw exactly that behavior with pydocstyle just before we removed it. And I experienced this with flake8 in Flintrock before [pinning the version](https://github.com/nchammas/flintrock/blob/52c6c84c9a1845b0ce89ca138172a6ec4cf0d632/requirements/developer.in#L4).)
   
   I don't understand the point of waiting for the build to break before pinning or severely limiting the versions for libraries like these.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607627444
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120706/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600096154
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24658/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601938672
 
 
   **[Test build #120116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120116/testReport)** for PR 27928 at commit [`75682a6`](https://github.com/apache/spark/commit/75682a6b0023f22f4a030d4143913db2a0021597).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393417977
 
 

 ##########
 File path: docs/requirements.txt
 ##########
 @@ -0,0 +1,3 @@
+sphinx==2.3.1
+mkdocs==1.0.4
+numpy==1.18.1
 
 Review comment:
   I think we can just merge at the moment given that there are not so many dependencies at this moment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing fixed in the upstream. It will still mess developer's environment. Arguably there are not so many Python-dedicated dev people in Spark community who fluently use pyenv, virtualenv or conda. _I think_ most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found 6 instances.
   
   - One was completely pinned.
   - Two were partially pinned
   - Three were not quite pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607412622
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25384/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607569409
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395198153
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   Would it help then if we expanded `dev/README.md` to show how to setup a virtual environment? I'm willing to do that.
   
   If we don't want to ask devs to use virtual environments at all, then perhaps we need to fork `dev/requirements.txt` and have a version that pins everything, for use in CI and releases, and a version that pins nothing, for use by devs who don't use virtual environments.
   
   Another alternative is the compromise currently standing in this PR, with some versions specified as `major.minor.*`.
   
   And yet another alternative (which I personally wouldn't favor, but I know it's common) is to Dockerize the whole development environment, but that's a lot of work.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601880057
 
 
   **[Test build #120116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120116/testReport)** for PR 27928 at commit [`75682a6`](https://github.com/apache/spark/commit/75682a6b0023f22f4a030d4143913db2a0021597).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615380135
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599790993
 
 
   Jenkins, retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600181313
 
 
   **[Test build #119934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119934/testReport)** for PR 27928 at commit [`cd05e25`](https://github.com/apache/spark/commit/cd05e25843671c7d17d3da621af12b431af6ad16).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615285850
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26097/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599738899
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119884/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607589450
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601939323
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607473468
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393200457
 
 

 ##########
 File path: docs/requirements.txt
 ##########
 @@ -0,0 +1,3 @@
+sphinx==2.3.1
+mkdocs==1.0.4
+numpy==1.18.1
 
 Review comment:
   We could also just fold this into `dev/requirements.txt` and have a single set of Python requirements that cover all development dependencies.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607473476
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120685/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-614761302
 
 
   I'm still generally OK with pinning things "lightly"; I am not sure how to test it though as I'm also not an RM.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599738041
 
 
   **[Test build #119884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119884/testReport)** for PR 27928 at commit [`9c3a7af`](https://github.com/apache/spark/commit/9c3a7af3ce469e375a5141d779d288442fc5675a).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607589044
 
 
   **[Test build #120706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120706/testReport)** for PR 27928 at commit [`b088d15`](https://github.com/apache/spark/commit/b088d151c6102e13f124318d8b14e9279da3287a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393504505
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   I understand the trade-off mentioned by @HyukjinKwon . Actually, I tried to pin once before and dropped my PR due to that. +1 for @HyukjinKwon 's advice, `I would only pin the version when an issue difficult to fix is found.`.
   
   The only exception is `spark-rm/Dockerfile`. We need to use specific version and to manager explicitly there.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607589044
 
 
   **[Test build #120706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120706/testReport)** for PR 27928 at commit [`b088d15`](https://github.com/apache/spark/commit/b088d151c6102e13f124318d8b14e9279da3287a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600260009
 
 
   **[Test build #119950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119950/testReport)** for PR 27928 at commit [`cd05e25`](https://github.com/apache/spark/commit/cd05e25843671c7d17d3da621af12b431af6ad16).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599672111
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24612/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393421338
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   cc somewhat arbitrary committers who might be interested in here: @srowen, @holdenk, @dongjoon-hyun, @BryanCutler. WDYT about this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599829077
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119896/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599677152
 
 
   cc @HyukjinKwon. I moved the requirements changes from #27912 to this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607589458
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25404/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600260545
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615285173
 
 
   **[Test build #121414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121414/testReport)** for PR 27928 at commit [`9704c60`](https://github.com/apache/spark/commit/9704c606983a4cf6bdd5bec41be8208fb5db63bc).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] BryanCutler commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r396770964
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   I also would prefer not to pin specific versions and agree with https://github.com/apache/spark/pull/27928#discussion_r395137725. It is good to have the community try with the latest to surface any issues, but we should be very clear what versions have been used in our CI, which could be from a virtualenv or just in a readme, so there is always an obvious fallback version.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607537445
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25396/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607537040
 
 
   **[Test build #120698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120698/testReport)** for PR 27928 at commit [`b088d15`](https://github.com/apache/spark/commit/b088d151c6102e13f124318d8b14e9279da3287a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r401802338
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -88,7 +88,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
+pip install -r ./dev/requirements.txt
 
 Review comment:
   Following from the comments by @cloud-fan [here](https://github.com/apache/spark/pull/27534#issuecomment-607011028) and [here](https://github.com/apache/spark/pull/28088#discussion_r401342910), and taking into account your request as well, I think we need to maintain two versions of the requirements file, one without pinning for dev purposes, and one with pinning for CI and the release machinery.
   
   I will implement this using pip-tools, following the examples set by Warehouse and Trio which I referenced earlier in this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599672102
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600257816
 
 
   Jenkinmensch, retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607537445
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25396/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615379157
 
 
   **[Test build #121414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121414/testReport)** for PR 27928 at commit [`9704c60`](https://github.com/apache/spark/commit/9704c606983a4cf6bdd5bec41be8208fb5db63bc).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `public class JavaUserDefinedScalar `
     * `public class SparkAvroKeyOutputFormat extends AvroKeyOutputFormat<GenericRecord> `
     * `  static class SparkRecordWriterFactory extends RecordWriterFactory<GenericRecord> `
     * `  class FMClassifierWrapperWriter(instance: FMClassifierWrapper) extends MLWriter `
     * `  class FMClassifierWrapperReader extends MLReader[FMClassifierWrapper] `
     * `  class FMRegressorWrapperWriter(instance: FMRegressorWrapper) extends MLWriter `
     * `  class FMRegressorWrapperReader extends MLReader[FMRegressorWrapper] `
     * `  class LinearRegressionWrapperWriter(instance: LinearRegressionWrapper) extends MLWriter `
     * `  class LinearRegressionWrapperReader extends MLReader[LinearRegressionWrapper] `
     * `case class LengthOfJsonArray(child: Expression) extends UnaryExpression`
     * `case class JsonObjectKeys(child: Expression) extends UnaryExpression with CodegenFallback`
     * `case class ShowViews(`
     * `case class ShowViewsCommand(`
     * `  class JDBCConfiguration(`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607473468
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-611578163
 
 
   What does everyone think of the current approach taken in this PR?
   
   Are we good to go now (pending a committer test of the release script to avoid a repeat of #27534)?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599672111
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24612/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601877346
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24829/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607627435
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599738885
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing fixed in the upstream. It will still mess developer's environment. Arguably there are not so many Python-dedicated dev people in Spark community who fluently use pyenv, virtualenv or conda. Most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found 6 requirements.txt for dev. I found 6 instances:
   
   - One was completely pinned.
   - Two were partially pinned
   - Three were not quite pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599792391
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24622/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600348311
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119950/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600181996
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119934/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615285173
 
 
   **[Test build #121414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121414/testReport)** for PR 27928 at commit [`9704c60`](https://github.com/apache/spark/commit/9704c606983a4cf6bdd5bec41be8208fb5db63bc).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600095348
 
 
   **[Test build #119934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119934/testReport)** for PR 27928 at commit [`cd05e25`](https://github.com/apache/spark/commit/cd05e25843671c7d17d3da621af12b431af6ad16).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600260009
 
 
   **[Test build #119950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119950/testReport)** for PR 27928 at commit [`cd05e25`](https://github.com/apache/spark/commit/cd05e25843671c7d17d3da621af12b431af6ad16).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600181996
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119934/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing. It will still mess developer's environment. Arguably there are not so many Python dedicated dev people in Spark community who fluently uses pyenv, virtualenv or conda. Most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found 6 requirements.txt for dev. I found 6 instances:
   
   - One was completely pinned.
   - Two were partially pinned
   - Three were not quite pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601939325
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120116/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599738885
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600347429
 
 
   **[Test build #119950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119950/testReport)** for PR 27928 at commit [`cd05e25`](https://github.com/apache/spark/commit/cd05e25843671c7d17d3da621af12b431af6ad16).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601877338
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615380145
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121414/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-602313639
 
 
   @srowen, With this change, we will have to maintain and keep `dev/requirements.txt` up to date. We will have to fix that file to use the latest version from now on. Why don't we try to don't pin first and see if we face related issues by using the latest versions? We can start to pin one by one when we face some issues difficult to fix.
   
   For the dependencies below, we have been testing via Github Actions. So far, I couldn't find any related issues related to the versions.
   
   - `pycodestyle` and `flake8` are Python linters.
   - `mkdocs` is for SQL documentation
   - `sphinx` is for Python documentation.
   
   We shouldn't pin `numpy` to encourage people to test the highest versions. It should ideally be `numpy>=1.7` according to `setup.py`.
   
   - `numpy` is an explicit dependency for ML/MLlib in PySpark.
   
   The dependencies below are release-specific and tricky to test. I suspect it's better for new dev people to test them out and pin the versions later when it's needed?
   
   - `PyGithub` and `Unidecode` are release-specific dependencies.
   - `jira` is for release, JIRA <> PR sync, and PR merge. (Did we test BTW?)
   
   FYI, there look some more occurrences such as `pandas` and `numpy` in PIP sanity check at `dev/run-pip-tests` as well from my cursory look:
   
   ```bash
   conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
   ```
   
   ```bash
   pip install --upgrade pip wheel numpy
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615380145
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121414/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395133004
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   It will. I am good with using the virtualenv in CI and docker image. I use Conda too in my local.
   
   My concern is that pinning versions will either force the dev people to use virtualenv (or its friends) or mess their installed packages.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393725207
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,10 @@
-flake8==3.5.0
+flake8==3.7.*
 
 Review comment:
   While I still think we should pin every version here, perhaps this approach is a compromise we can agree on.
   
   `==3.7.*` means pip will install the latest bugfix release on 3.7. If you already have any 3.7 version installed, even if it's not the latest one, pip will consider the requirement satisfied and won't do anything. To force pip to upgrade to the latest bugfix release of 3.7 when you already have a compatible version installed, simply request it via `pip install --upgrade`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r393420994
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   I quickly googled and skimmed other projects about the requirements [here](https://data-flair.training/blogs/python-open-source-projects/) and [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I still think it's more usual to specify the range, rather than the specific version.
   
   I am still not sure if it's right to pin the version yet. There's trade-off on pinning. I think Spark community is big enough to handle these issue from using the latest versions too. I would only pin the version when an issue difficult to fix is found.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-602375329
 
 
   > With this change, we will have to maintain and keep `dev/requirements.txt` up to date. 
   
   Maybe this is the disconnect between our points of view, because so far I haven't really been following your objections to pinning. Assuming we pin every library, why do we have to keep `dev/requirements.txt` up-to-date?
   
   As long as we can build the docs, run tests, and do whatever else we need to do as part of regular development, that file can remain frozen as-is for years.
   
   It's only when we specifically want to use some new feature of, say, Sphinx, that we need to bump versions. But that will happen very rarely, I imagine not more than once every couple of years.
   
   Does that address your concern? Why do you think we'd need to touch that file more often than once in a long while?
   
   > We shouldn't pin `numpy` to encourage people to test the highest versions. It should ideally be `numpy>=1.7` according to `setup.py`.
   > 
   > * `numpy` is an explicit dependency for ML/MLlib in PySpark.
   
   But the specification of numpy in `dev/requirements.txt` is so that we can build our docs. (It seems strange, but yes, numpy is a requirement to build our Python API docs.)
   
   Maybe we can improve this by replacing numpy in `dev/requirements.txt` with a reference to `setup.py`. That way we can track PySpark dependencies (whether for building the docs or for general execution) in one place. This will also pick up the Pandas requirement. How does that sound?
   
   A separate issue I raised earlier is that, if we want to not pin our build/test dependencies, we need to  figure out what to do about the Spark Docker image and CI. Either those will also source the unpinned requirements from the same file, or we go back to having the requirements specified in duplicate--with pinned versions for Docker and CI, and without pinned versions for developers.
   
   Obviously, I'd prefer to pin everything and keep it in one place, but if you want to go one of these routes I guess I'll do that. I just want to understand and try to address your objections before going there.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607587328
 
 
   Jenkins, you are very good looking today. Retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607589450
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601880685
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600096144
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599828606
 
 
   **[Test build #119896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119896/testReport)** for PR 27928 at commit [`9c3a7af`](https://github.com/apache/spark/commit/9c3a7af3ce469e375a5141d779d288442fc5675a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599671584
 
 
   **[Test build #119884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119884/testReport)** for PR 27928 at commit [`9c3a7af`](https://github.com/apache/spark/commit/9c3a7af3ce469e375a5141d779d288442fc5675a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601939325
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120116/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-614749903
 
 
   Checking in again here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607569413
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120698/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600096144
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r399965053
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -88,7 +88,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
+pip install -r ./dev/requirements.txt
 
 Review comment:
   and TBH, I would like to don't pin the version here as well.
   
   ```diff
   - pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
   + pip install sphinx mkdocs numpy
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing fixed in the upstream. It will still mess developer's environment. Arguably there are not so many Python dedicated dev people in Spark community who fluently uses pyenv, virtualenv or conda. Most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found 6 requirements.txt for dev. I found 6 instances:
   
   - One was completely pinned.
   - Two were partially pinned
   - Three were not quite pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607537436
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607473476
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120685/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r396177554
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,10 @@
-flake8==3.5.0
+flake8==3.7.*
 
 Review comment:
   If we can assume these dependencies follow SemVer, it we should better use wildcards on minor versions ...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600095348
 
 
   **[Test build #119934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119934/testReport)** for PR 27928 at commit [`cd05e25`](https://github.com/apache/spark/commit/cd05e25843671c7d17d3da621af12b431af6ad16).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600096154
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24658/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607412613
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607411876
 
 
   **[Test build #120685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120685/testReport)** for PR 27928 at commit [`bb83262`](https://github.com/apache/spark/commit/bb8326237f2d8a680038497a6f03c926e8194bd4).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607473007
 
 
   **[Test build #120685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120685/testReport)** for PR 27928 at commit [`bb83262`](https://github.com/apache/spark/commit/bb8326237f2d8a680038497a6f03c926e8194bd4).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas edited a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas edited a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607587328
 
 
   Jenkins, unconquerable lord of CI, retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601877346
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24829/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615285838
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607569413
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120698/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395137725
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   And also ideally using latest versions will bring some bug fixes. I assume we'll start to update this file to use latest pinned versions if this PR is merged, which is virtually equivalent with not pinning.
   
   Not pinning take the risk about unexpected build failures in Github Actions - not in Jenkins since the packages are installed once, which I don't think is very critical.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r396197611
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,10 @@
-flake8==3.5.0
+flake8==3.7.*
 
 Review comment:
   I mentioned it elsewhere but I'll mention it again here: Linters like flake8 and pycodestyle introduce new checks in minor/feature releases. There is very high chance that every new check they introduce will flag new problems and fail the build. (In fact, we saw exactly that behavior with pydocstyle just before we removed it.)
   
   I don't understand the point of waiting for the build to break before pinning or severely limiting the versions for libraries like these.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599829077
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119896/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607589458
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25404/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607569409
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600181974
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-602313639
 
 
   @srowen, With this change, we will have to maintain and keep `dev/requirements.txt` up to date. We will have to fix that file to use the latest version from now on. Why don't we try to don't pin first and see if we face related issues by using the latest versions? We can start to pin one by one when we face some issues difficult to fix.
   
   For the dependencies below, we have been testing via Github Actions. So far, I couldn't find any related issues related to the versions.
   
   - `pycodestyle` and `flake8` are Python linters.
   - `mkdocs` is for SQL documentation
   - `sphinx` is for Python documentation.
   
   We shouldn't pin `numpy` to encourage people to test the highest versions. It should ideally be `numpy>=1.7` according to `setup.py`.
   
   - `numpy` is an explicit dependency for ML/MLlib in PySpark.
   
   The dependencies below are release-specific and tricky to test. I suspect it's better for new dev people to test them out and pin the versions later when it's needed?
   
   - `PyGithub` and `Unidecode` are release-specific dependencies.
   - `jira` is for both release, JIRA <> PR sync, and PR merge. (Did we test BTW?)
   
   FYI, there look some more occurrences such as `pandas` and `numpy` in PIP sanity check at `dev/run-pip-tests` as well from my cursory look:
   
   ```bash
   conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
   ```
   
   ```bash
   pip install --upgrade pip wheel numpy
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599671584
 
 
   **[Test build #119884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119884/testReport)** for PR 27928 at commit [`9c3a7af`](https://github.com/apache/spark/commit/9c3a7af3ce469e375a5141d779d288442fc5675a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395715232
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   I would suggest a note in README to ensure people know how they can isolate this env if needed, and pinning minor version only? is that close enough for consensus?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-605817831
 
 
   Yes. I want to take some breakages that comes with automatic version updates, also given that we haven't faced many issues by doing that. Yes, seems like there's a bit of different preferences.
   
   Can we touch versions here and just keep the refactoring alone in this PR for now? I like that part of putting dependencies into `requirements.txt`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r401347559
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -88,7 +88,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
+pip install -r ./dev/requirements.txt
 
 Review comment:
   Yeah, I want. I want the central `requirements.txt` without pinning the version.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing fixed in the upstream. It will still mess developer's environment. Arguably there are not so many Python-dedicated dev people in Spark community who fluently use pyenv, virtualenv or conda. _I think_ most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found instances.
   
   - One was completely pinned.
   - Two were partially pinned
   - Three were not quite pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599738899
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119884/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r396197611
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,10 @@
-flake8==3.5.0
+flake8==3.7.*
 
 Review comment:
   I mentioned it elsewhere but I'll mention it again here: Linters like flake8 and pycodestyle introduce new checks in minor/feature releases. There is very high chance that every new check they introduce will flag new problems and fail the build.
   
   In fact, we saw exactly that behavior with pydocstyle just before we removed it. And I [experienced this](https://github.com/nchammas/flintrock/commit/9157b25d735ff6ef690cfdfb761f336bf999fc82) with pycodestyle in Flintrock before [pinning the version](https://github.com/nchammas/flintrock/commit/6c6d9562673b430d142d9062a99e9ac1c87366b8).
   
   I don't understand the point of waiting for the build to break before pinning or severely limiting the versions for libraries like these.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-601880685
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615380135
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-615285838
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r401207348
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -88,7 +88,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
+pip install -r ./dev/requirements.txt
 
 Review comment:
   I'm confused. I thought you wanted to keep using a central `requirements.txt`, but you just wanted to not pin the versions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r394842917
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   There's the trade-off here as well - it's the most stable but there might be multiple bugs existing fixed in the upstream. It will still mess developer's environment. Arguably there are not so many Python-dedicated dev people in Spark community who fluently use pyenv, virtualenv or conda. _I think_ most of them just use pip and local installation.
   
   I quickly skimmed requirements.txt **for dev or docs** at [here](https://hackernoon.com/50-popular-python-open-source-projects-on-github-in-2018-c750f9bf56a0). I skimmed top 20 projects, and found 6 requirements.txt for dev. I found 6 instances:
   
   - One was completely pinned.
   - Two were partially pinned
   - Three were not quite pinned.
   
   I agree the dev envs more tend to specify the versions; however, I think it's still prevailing to don't pin.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607537040
 
 
   **[Test build #120698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120698/testReport)** for PR 27928 at commit [`b088d15`](https://github.com/apache/spark/commit/b088d151c6102e13f124318d8b14e9279da3287a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599829073
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-600260552
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24674/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395852390
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   Hmm, on second thought, perhaps the README should be left to a separate PR.  We already have advice on setting up a development environment in at least a couple of places, like [Useful Developer Tools](http://spark.apache.org/developer-tools.html) and [`docs/README.md`](https://github.com/apache/spark/blob/fae981e5f32e0ddb86591a616829423dfafb4ed0/docs/README.md#using-rbenv-and-pyenv).
   
   Perhaps we should consolidate that advice over on the Useful Developer Tools page, since it fits in with the information already there.
   
   Either that, or let's agree on some other approach to take. But I think we can defer any dev documentation changes to a follow-up PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#discussion_r395729377
 
 

 ##########
 File path: dev/requirements.txt
 ##########
 @@ -1,5 +1,7 @@
-flake8==3.5.0
+pycodestyle==2.5.0
+flake8==3.7.9
 jira==1.0.3
 PyGithub==1.26.0
 Unidecode==0.04.19
-sphinx
+sphinx==2.3.1
+numpy==1.18.1
 
 Review comment:
   OK, I'll do that. I'll recommend that users create a virtual environment in `dev/README.md` and demonstrate how to do that. I'll also update the version specifiers to all be in the form of `major.minor.*`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas edited a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
nchammas edited a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-614749903
 
 
   Checking in again here. Is there something I can do to move this along?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-599672102
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607627435
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607627444
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120706/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607411876
 
 
   **[Test build #120685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120685/testReport)** for PR 27928 at commit [`bb83262`](https://github.com/apache/spark/commit/bb8326237f2d8a680038497a6f03c926e8194bd4).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies
URL: https://github.com/apache/spark/pull/27928#issuecomment-607626894
 
 
   **[Test build #120706 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120706/testReport)** for PR 27928 at commit [`b088d15`](https://github.com/apache/spark/commit/b088d151c6102e13f124318d8b14e9279da3287a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org