You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/10 23:05:31 UTC

[GitHub] [spark] nchammas opened a new pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

nchammas opened a new pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534

### What changes were proposed in this pull request?

This PR makes the following refinements to the workflow for building docs:
* Install Python and Ruby consistently using pyenv and rbenv across both the docs README and the release Dockerfile.
* Pin the Python and Ruby versions we use.
* Pin all direct Python and Ruby dependency versions.
* Eliminate any use of `sudo pip` or `sudo gem`, which the Python community discourages.
* Remove unused cruft in the Python docs Makefiles.
* Update a deprecated mkdocs config.

### Why are the changes needed?

This PR should increase the consistency and reproducibility of the doc-building process by managing Python and Ruby in a more consistent way, and by eliminating unused or outdated code.

Here's a possible example of an issue building the docs that would be addressed by the changes in this PR: https://github.com/apache/spark/pull/27459#discussion_r376135719

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Manual tests:
* I was able to build the Docker image successfully, minus the final part about `RUN useradd`.
* I am unable to run `do-release-docker.sh` because I am not a committer and don't have the required GPG key.
* I built the docs locally and viewed them in the browser.

I think I need a committer to more fully test out these changes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595971844
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119478/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r387158036
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   I guess I'd default to pinning the version in the absence of a strong argument either way. I do think we've had inconsistencies in the spark-website project due to differing jekyll versions, for example. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584402671
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595570469
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584444965
 
 
   **[Test build #118187 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118187/testReport)** for PR 27534 at commit [`bc15b86`](https://github.com/apache/spark/commit/bc15b8652efe1d500c801023ce3d102f7a7953fe).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587945770
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118653/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r382040514
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   I think it's OK to pin. We have to maintain this across docs vs script, but, probably better this way.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595498091
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588433227
 
 
   Oh, in that case, please use different IDs from the beginning next time especially.
   > nobody asked me to split up the PR! 
   
   Thanks, @nchammas .

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588507887
 
 
   Will do. Apologies to @dongjoon-hyun and everyone else watching this PR for the noise.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas edited a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas edited a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-585596381
 
 
   Should I break up this PR to make it easier to review? e.g.
   1. Makefile trim (split to #27625)
   2. pages -> nav (split to #27626)
   3. pyenv and rbenv + dependency pinning
   
   Asking just in case the silence from other reviewers is because this PR is too big.
   
   If y'all just haven't gotten around to taking a look, no worries. I'm being proactive.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r386124976
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   I'm not following your concern about pinning versions. Is there some scenario or overhead from pinning versions that you are concerned about?
   
   It seems inevitable to me that contributors will waste time debugging issues that stem from them running different versions of these libraries. Instead of waiting for those issues to arise, it seems safe and simple enough to avoid them by pinning what we use. In fact, what prompted me to open this PR is an interaction I had with you on another PR where we were seeing different behavior when trying to build the docs. So I believe we've already seen at least one example of this problem. 😄 (I linked to that interaction from this PR's description.)
   
   Pinning our doc build dependencies is the core idea of this PR. If we don't want to do that, then I'm happy to close this PR and leave things as-is.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HeartSaVioR commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-589399811
 
 
   Personally I'm feeling that setting Affected Version on improvement type is giving confusion while the value we get is relatively small, but let's discuss it in dev@ instead of here, as it's off-topic.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591470464
 
 
   FWIW, I don't think Affected Version really matters much for an improvement. You can set it to say it would be a valid improvement as of a version -- maybe it improves a thing that's only present from 2.4.0 onwards or something. But for most cases it probably 'affects' (could be applied to) lots of versions, and that isn't as interesting as in the case of a bug. Not required unless it's really important clarify, doesn't hurt in most cases either.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587926442
 
 
   **[Test build #118656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118656/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595498096
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24162/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587974850
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606505361
 
 
   Sure~ Of course for reverting this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595497604
 
 
   Jenkins retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588402239
 
 
   Hi, @HyukjinKwon .
   We had better avoid landing this improvement on `branch-3.0`. This changes the release stuff a lot and and technically give a big risk to the next release manager because this resets our previous efforts and validation during preview/preview2.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-607021040
 
 
   We can pin the version for release related ones; however, I doubt if we should do that for others e.g., CI, documentation. It's being discussed at https://github.com/apache/spark/pull/27928

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587905981
 
 
   **[Test build #118652 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118652/testReport)** for PR 27534 at commit [`25fdaef`](https://github.com/apache/spark/commit/25fdaef39f9e7d27c9d485e03eadad5b1934c4ab).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-607021040
 
 
   We can pin the version for release related ones; however, I doubt if we should do that for others e.g., CI, documentation. It's being discussed at https://github.com/apache/spark/pull/27928#pullrequestreview-385094894

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595570475
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119425/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587907892
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587730563
 
 
   **[Test build #118653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118653/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588416416

Got it. Then, it might be some misunderstanding. I thought you are aware of the community policies because you have been working in the community for a long time.
1. First of all, we don't backport `Improvement`. The decision is made per PR.
2. Due to (1), for improvement JIRAs, we had better use the version of master branch for `Affected Version`. In your case, you had better set `3.1.0` clearly when you register as `Improvement`.
3. For the same ID PRs, we use the same JIRA ID for the related follow-up PRs only. And, in that case, we use `[FOLLOWUP]` tag in the PR title.

Do you think your 3 PRs are follow-ups? For me, those are orthogonal. e.g., `Redefine workflow` is irrelevant to replace the deprecate `Mkdocs` option. The main reason why the committers ask to split is that you put orthogonal things into a single PR. It's bad not only for review, but also for understanding the commit logs.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r385479956

##########
File path: docs/README.md
##########
@@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
Python, R and SQL.

-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].

Review comment:
I am not strongly against this change, but not very fond of it as it forces to install unnecessary dependencies. Other people might use conda to manage Python versions as an example. At least I use conda. Given that there are many ways to control this, I am not sure about documenting one way. Why don't we just mention Ruby 2 and Python 3?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606524598
 
 
   I think my wording made some miscommunication here:
   
   > Let me revert this to make RC preparation easier.
   
   I simply meant future RCs :-).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591198791
 
 
   There's nothing confusing to "Affected Version" on improvement type. It just targets where you want that fix to be landed. The general guideline is bug fixes only for backporting but it's judged case-by-case in practice. For example, I just backport minor documentation improvement when possible to reduce conflicts.
   
   The only thing wrong here is I merged #27626 mistakenly with the same JIRA here and @nchammas used the same JIRA mistakenly. There was just a bit of miscommunication, which we should avoid next time.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377394628
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   @nchammas, btw why should we fix the version? I think it's better to promote the latest versions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587722115
 
 
   I've split off the Makefile and Mkdocs changes and updated my previous comment with links to the new PRs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587932015
 
 
   Is it possible for contributors to re-trigger the GitHub workflows for a certain commit? I'm not seeing how to do that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas edited a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas edited a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-585596381
 
 
   Should I break up this PR to make it easier to review? (Edit: Done.)
   
   e.g.
   1. Makefile trim (split to #27625)
   2. pages -> nav (split to #27626)
   3. pyenv and rbenv + dependency pinning
   
   Asking just in case the silence from other reviewers is because this PR is too big.
   
   If y'all just haven't gotten around to taking a look, no worries. I'm being proactive.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HeartSaVioR commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591210981
 
 
   > There's nothing confusing to "Affected Version" on improvement type. It just targets where you want that fix to be landed.
   
   In fact we don't guide like that. For example, suppose someone found a point to improve in Spark 2.4.x - they would set "Affected Version" to be 2.4.x as it may be the only version they know about, but as I see some guides on the fly we have been guiding to the latest "unreleased" minor version which they never know about.
   
   While I don't think it represents the intention clearly on marking Affected Version on improvement, would we call it simply "the version which master branch targets to", and add it to the contribution guide? In contribution guide we only mention about bugs, no other types.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587923992
 
 
   Retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r387406314

Review comment:
I think docker changes are fine. Can we just remove rbenv and pyenv in this doc if we're good with that?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587945770
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118653/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r381522628

Review comment:
If the committers are not comfortable with this change, a more conservative change we can make for now is simply to pin the versions of the various Python and Ruby libraries that we install. We can revisit the rbenv/pyenv idea at a later time.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588413484
 
 
   > Do you think we should have Fixed Version for 2.4.6 because of 6e0c116 ?
   
   I don't know why you are asking me this question. If you are implying that I somehow asked for 6e0c116 to be part of the next 2.4.x release, then there is some gross misunderstanding here. I have no opinions on the releases or where each patch should land, and I did not mention 2.4 in my JIRA ticket or any of the 3 PRs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584445333
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118187/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587907934
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118652/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377433240
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   We should pin dependencies for the usual reason: It guarantees a stable environment that gives reproducible results.
   
   New releases may have breaking behavior, so you don't always want to install the latest version. And different users building the docs may have outdated versions installed, which will yield them a broken build. (e.g. If you ran `pip install sphinx` when the latest release was 1.1.0, then pip will not update your version of sphinx as new releases are made unless you explicitly request it via `pip install --upgrade`.)
   
   These categories of build problem are addressed by pinning versions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377394406
 
 

 ##########
 File path: dev/create-release/spark-rm/Dockerfile
 ##########
 @@ -20,9 +20,9 @@
 # Includes:
 
 Review comment:
   @dongjoon-hyun and @wangyum might better to review.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587945765
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587721454
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23405/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595971844
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119478/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591198791
 
 
   There's nothing confusing to "Affected Version" on improvement type. It just targets where you want that fix to be landed. The general guide line is bug fixes only for backporting but it's judged case-by-case in practice. For example, I just backport minor documentation improvement when possible to reduce conflicts.
   
   The only thing wrong here is I merged #27626 mistakenly with the same JIRA here and @nchammas used the same JIRA mistakenly. There was just a bit of miscommunication, which we should avoid next time.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588385183
 
 
   @dongjoon-hyun 
   
   > Please don't reuse the same JIRA ID for different PR
   
   The 3 PRs were initially 1 PR. I split up this PR into 3 to make it easier to review. The JIRA ticket captures what all 3 PRs are addressing, which is basically a set of refinements to the doc-building workflow.
   
   As far as I know, there doesn't have to be a 1-to-1 relationship between PRs and JIRA tickets. Is that not the case?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595569601
 
 
   **[Test build #119425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119425/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r389079482

Review comment:
I moved them to their own sub-section. If we think it's distracting / not helpful, I can remove it.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587721454
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23405/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377394873
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
 The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
 Python, R and SQL.
 
-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].
 
 Review comment:
   @nchammas, what about we keep the default way, and simply mention such pyend, virtualenv or conda?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588402239
 
 
   Hi, @HyukjinKwon .
   We had better avoid landing this improvement on `branch-3.0`. This changes the release stuff a lot and and technically gives a big risk to the next release manager because this resets our previous efforts and validation during preview/preview2.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588416416

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588396903

@nchammas . No, you had better make it umbrella JIRA.
> The 3 PRs were initially 1 PR. I split up this PR into 3 to make it easier to review. The JIRA ticket captures what all 3 PRs are addressing, which is basically a set of refinements to the doc-building workflow.

First of all, after merging your three PRs, do you think what is going to be the `Fixed Version` of SPARK-30731? Do you think we should have `Fixed Version` for `2.4.6` because of https://github.com/apache/spark/commit/6e0c1162bf95938081e93c77a39b2826fec9c84e ?

Second, you created SPARK-30731 as `Improvement` JIRA. Did you aim to land it `branch-2.4` from the beginning? Then, you must understand Apache Spark community policy. We don't backport the improvement.

Since you are reusing the same JIRA ID, @HyukjinKwon mistakenly landed SPARK-30731 to `branch-2.4`.
- https://github.com/apache/spark/commit/6e0c1162bf95938081e93c77a39b2826fec9c84e

It seems that you are intentional because you don't understand the community policy, let me fix the situation first.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595909684
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24208/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595971100
 
 
   **[Test build #119478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119478/testReport)** for PR 27534 at commit [`ebba668`](https://github.com/apache/spark/commit/ebba66868f9833cbf2771476b609bfefaf8e946c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588413484
 
 
   > Do you think we should have Fixed Version for 2.4.6 because of 6e0c116 ?
   
   I don't know why you are asking me this question. If you are implying that I somehow asked for there to be a new 2.4.x release because of 6e0c116, then there is some gross misunderstanding here. I have no opinions on the releases or where each patch should land, and I did not mention 2.4 in my JIRA ticket or any of the 3 PRs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588424478
 
 
   > I thought you are aware of the community policies because you have been working in the community for a long time.
   
   I thought I was familiar with the policies, too. 😄
   
   > 1. First of all, we don't backport `Improvement`. The decision is made per PR.
   
   Understood. And again, just to be clear, I didn't knowingly ask to backport any improvement.
   
   > 2. Due to (1), for improvement JIRAs, we had better use the version of master branch for `Affected Version`. In your case, you had better set `3.1.0` clearly when you register as `Improvement`.
   
   Perhaps this is part of why I am confused. By setting the "affects version" to 3.0.0 on my ticket, was I technically asking for the improvement to be backported to branch-3.0?
   
   This still doesn't explain how branch-2.4 entered the picture, by the way. Is a backport to branch-2.4 implied because 3.0 hasn't been released yet?
   
   > 3. For the same ID PRs, we use the same JIRA ID for the related follow-up PRs only. And, in that case, we use `[FOLLOWUP]` tag in the PR title.
   
   I wasn't familiar with this convention, but: Duly noted.
   
   > Do you think your 3 PRs are follow-ups? For me, those are orthogonal. e.g., `Redefine workflow` is irrelevant to replace the deprecate `Mkdocs` option. The main reason why the committers ask to split is that you put orthogonal things into a single PR. It's bad not only for review, but also for understanding the commit logs.
   
   I agree. But just for the record, nobody asked me to split up the PR! I split it up on my own initiative to facilitate review. And in the process of doing that, as we've just seen, I unintentionally made a mess of things... 😅

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595497772
 
 
   **[Test build #119425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119425/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584402671
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591214680
 
 
   This is written in the guide:
   
   > Affects Version. For Bugs, assign at least one version that is known to exhibit the problem or need the change
   
   Ideally they should search JIRAs and see if there are duplicates. If there are duplicate, we should resolve it as a duplicate. We can clarify more, of course. However, I don't see that this is confusing or causes any bad pattern or overhead yet.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377394288
 
 

 ##########
 File path: python/docs/Makefile
 ##########
 @@ -4,7 +4,6 @@
 # You can set these variables from the command line.
 
 Review comment:
   +1 for this change too although I didn't check the details. This Makefile was from the old sphinx example.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587719427
 
 
   **[Test build #118652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118652/testReport)** for PR 27534 at commit [`25fdaef`](https://github.com/apache/spark/commit/25fdaef39f9e7d27c9d485e03eadad5b1934c4ab).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377394186
 
 

 ##########
 File path: sql/mkdocs.yml
 ##########
 @@ -15,5 +15,5 @@
 
 site_name: Spark SQL, Built-in Functions
 theme: readthedocs
-pages:
+nav:
 
 Review comment:
   Yeap, I saw the warning. +1

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584445333
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118187/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584402678
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22949/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587719427
 
 
   **[Test build #118652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118652/testReport)** for PR 27534 at commit [`25fdaef`](https://github.com/apache/spark/commit/25fdaef39f9e7d27c9d485e03eadad5b1934c4ab).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r388686114

Review comment:
Can we revert these doc changes in the documentation? Otherwise, I am good with other changes.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r385479036
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   Hm, there's tradeoff to pin the version (promote to test and fix it compatible first vs prevent to break at the latest versions). I think we haven't faced any notable problem so far to maintain those implicit compatibility across versions. What about we pin the version when actually we see some overhead to maintain this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595570475
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119425/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606444361
 
 
   Reverted at https://github.com/apache/spark/commit/4d4c3e76f6d1d5ede511c3ff4036b0c458a0a4e3.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588358204
 
 
   cc @vanzin for `spark-rm` changes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r401088457
 
 

 ##########
 File path: dev/create-release/spark-rm/Dockerfile
 ##########
 @@ -50,36 +46,43 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
   rm -rf /var/lib/apt/lists/* && \
   apt-get clean && \
   apt-get update && \
-  $APT_INSTALL software-properties-common && \
-  apt-add-repository -y ppa:brightbox/ruby-ng && \
-  apt-get update && \
   # Install openjdk 8.
   $APT_INSTALL openjdk-8-jdk && \
   update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java && \
   # Install build / source control tools
   $APT_INSTALL curl wget git maven ivy subversion make gcc lsof libffi-dev \
-    pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev && \
+    pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev
+
+ENV PATH "$PATH:/root/.pyenv/bin:/root/.pyenv/shims"
+RUN curl -L https://github.com/pyenv/pyenv-installer/raw/dd3f7d0914c5b4a416ca71ffabdf2954f2021596/bin/pyenv-installer | bash
 
 Review comment:
   Hmm, that's strange, because I was able to build enough of the Dockerfile to get past this point, and I did not hit this problem.
   
   What I couldn't test was to run everything via `do-release-docker.sh`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587974858
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118656/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588364796
 
 
   Since https://github.com/apache/spark/pull/27626 is merged to `branch-2.4` as SPARK-30731, you had better update SPARK-30731 accordingly. And, this PR should have different new JIRA ID.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595498096
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24162/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591198791
 
 
   There's nothing confusing to "Affected Version" on improvement type. It just targets where you want that fix to be landed. The general guide line is bug fixes only but it's judged case-by-case in practice. For example, I just backport minor documentation improvement when possible to reduce conflicts.
   
   The only thing wrong here is I merged #27626 mistakenly with the same JIRA here and @nchammas used the same JIRA mistakenly. There was just a bit of miscommunication, which we should avoid next time.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587927030
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23408/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587974858
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118656/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595909097
 
 
   **[Test build #119478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119478/testReport)** for PR 27534 at commit [`ebba668`](https://github.com/apache/spark/commit/ebba66868f9833cbf2771476b609bfefaf8e946c).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HeartSaVioR edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591217692
 
 
   It only says for bugs, while we're saying improvements. Anyway I will go forward - let's avoid polluting the original topic.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588403216

> Second, you created SPARK-30731 as Improvement JIRA. Did you aim to land it branch-2.4 from the beginning? Then, you must understand Apache Spark community policy. We don't backport the improvement.

No, I didn't aim to land it in branch-2.4. As I noted over on https://github.com/apache/spark/pull/27626#issuecomment-588391223, I don't know why the MkDocs patch was landed there. I created the JIRA with an "affects version" of 3.0.0 and didn't mention 2.4 anywhere. I didn't ask to backport any improvements, and I don't know why branch-2.4 became a point of discussion in any of these 3 PRs.

> It seems that you are intentional because you don't understand the community policy, let me fix the situation first.

I'm not sure what you mean by this, but I think you've imputed some motives to me that I don't have. All I wanted to do was split up 1 PR into 3 PRs. They are all refinements to the doc-building workflow. They are all improvements that don't need to go anywhere other than to master and maybe also branch-3.0.

I wasn't trying to circumvent community policy in any way. I'm happy to create any new JIRAs required, but it looks like you've gone ahead and done that yourself.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595971834
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587927018
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595909684
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24208/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591183763
 
 
   The change looks reasonable. Updating docs and pinning versions sounds good. It sounds like you were able to test most of it, and probably the key part -- the actual doc generation. I am not sure I personally can even test the last bit without accidentally pushing something. I think that, if nobody else sees an issue with it and we believe it's working, it can go into master for 3.1.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587721422
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen closed pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen closed pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HeartSaVioR edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591210981

> There's nothing confusing to "Affected Version" on improvement type. It just targets where you want that fix to be landed.

In fact we don't guide like that. For example, suppose someone found a point to improve in Spark 2.4.x - they would set "Affected Version" to be 2.4.x (or try to set it to 2.5.0 and realize it doesn't work) as it may be the only version they know about, but as I see some guides on the fly we have been guiding to the latest "unreleased" minor version which they never know about.

While I don't think it represents the intention clearly on marking Affected Version on improvement, would we call it simply "the version which master branch targets to", and add it to the contribution guide? In contribution guide we only mention about bugs, no other types.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606780868
 
 
   > We can manually test it by changing the `ENTRYPOINT` in `Dockerfile` to `/bin/bash` and check if the python and ruby are installed correctly.
   > 
   > The release script doesn't work anymore after this patch because we use a non-standard way to install python/ruby (download a bash script and run it):
   > 
   > 1. The actual work is done by user `spark-rm`, which can't access `/root/.pyenv`
   > 2. Even if we fix the permission issue, the installed python doesn't work well with the system libraries, and have errors like `ImportError: No module named threading`
   > 3. It's wrong to append the `PATH`, and we should prepend it. Otherwise `python` still points to the system default which is 2.7.
   
   This is strange since I tested this successfully via `docker build .` from within the folder containing the Dockerfile.
   
   What I couldn't do was test building the image via `do-release-docker.sh`, which I called out in the PR description and again [in the comments](https://github.com/apache/spark/pull/27534#issuecomment-592144594).
   
   I think the reason my test didn't see these issues is because the `do-release-docker.sh` script builds the image with some options that change the executing user from `root`.
   
   ```
   run_silent "Building spark-rm image with tag $IMGTAG..." "docker-build.log" \
     docker build --no-cache -t "spark-rm:$IMGTAG" --build-arg UID=$UID "$SELF/spark-rm"
   ```
   
   I guess the moral of the story is that I couldn't test `do-release-docker.sh` directly, and my substitute test of `docker build .` was not a good test since it didn't account for the `root` vs. `spark-rm` user change.
   
   I should have more carefully tried to duplicate the `docker build` command as it appeared in `do-release-docker.sh`, and insisted more forcefully that we not merge this PR in until a committer had had a chance to try out `do-release-docker.sh` directly.
   
   Sorry.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606494157
 
 
   BTW, are we using the release script in `master` branch?
   We had better use the release script in `branch-3.0` to avoid `Apache Spark 2.4.2` situation.
   
   cc @rxin since he is a release manager.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587945765
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-607011028
 
 
   This PR does have a good point to fix the dependency versions so that the script is more robust. I'm happy to see a working version of it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584445329
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587927018
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588418529
 
 
   I asked that questions because you insisted to have the same JIRA ID. You cannot share the same JIRA ID for three different your contributions because we will review per PRs.
   > I don't know why you are asking me this question.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r386124063
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
 The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
 Python, R and SQL.
 
-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].
 
 Review comment:
   We can just tell people to install Ruby 2 and Python 3, but without specifying specific versions of them we risk having contributors and committers waste time dealing with library versions that are incompatible with the version of the language/runtime they have installed, or that somehow don't work with our configuration.
   
   It's a small risk, but I figured if we're going to use rbenv and pyenv in building the Docker image, it would make sense to recommend them for the interactive/local workflow as well. The experience is consistent, which is what's nice about using these tools, and they are "lower level" than things like Conda in that they are relatively simple shell scripts that only manage installations.
   
   Are you OK with using rbenv and pyenv in the Docker image?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377434717
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
 The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
 Python, R and SQL.
 
-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].
 
 Review comment:
   We can do that, but I am promoting pyenv and rbenv because they give us a consistent way to install any version of Python and Ruby across Docker and the typical developer environment, without depending on package repositories that typically lag a year or two behind the latest release. pyenv and rbenv also give and simpler and more consistent workflow for getting and using specific versions of Python and Ruby.
   
   With the way things are right now, we don't specify any version of Python or Ruby in the docs README, and in the Dockerfile we use a [4 year old version of Ruby](https://www.ruby-lang.org/en/news/2015/12/25/ruby-2-3-0-released/) and [3 year old version of Python](https://www.python.org/downloads/release/python-360/). Furthermore, to upgrade from these versions to newer ones, developers have to figure out the upgrade path themselves for their local workstation, and for Docker they have to hunt through package repositories that, again, often lag behind what's available.
   
   pyenv and rbenv give us a consistent way to install and manage what versions of Ruby and Python we're using to build docs -- whether on the local machine or in a Docker image -- and they make it dead simple to change the versions we want to use.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584445329
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587721422
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r377434717
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
 The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
 Python, R and SQL.
 
-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].
 
 Review comment:
   We can do that, but I am promoting pyenv and rbenv because they give us a consistent way to install any version of Python and Ruby across Docker and the typical developer environment, without depending on package repositories that typically lag a year or two behind the latest release. pyenv and rbenv also give a simpler and more consistent workflow for getting and using specific versions of Python and Ruby.
   
   With the way things are right now, we don't specify any version of Python or Ruby in the docs README, and in the Dockerfile we use a [4 year old version of Ruby](https://www.ruby-lang.org/en/news/2015/12/25/ruby-2-3-0-released/) and [3 year old version of Python](https://www.python.org/downloads/release/python-360/). Furthermore, to upgrade from these versions to newer ones, developers have to figure out the upgrade path themselves for their local workstation, and for Docker they have to hunt through package repositories that, again, often lag behind what's available.
   
   pyenv and rbenv give us a consistent way to install and manage what versions of Ruby and Python we're using to build docs -- whether on the local machine or in a Docker image -- and they make it dead simple to change the versions we want to use.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584402199
 
 
   **[Test build #118187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118187/testReport)** for PR 27534 at commit [`bc15b86`](https://github.com/apache/spark/commit/bc15b8652efe1d500c801023ce3d102f7a7953fe).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587974850
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-592144594
 
 
   I don't think any committers have tried working with the proposed build instructions and/or Docker definition. Perhaps a committer could try things out to make sure it works for them?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r400686586
 
 

 ##########
 File path: dev/create-release/spark-rm/Dockerfile
 ##########
 @@ -50,36 +46,43 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
   rm -rf /var/lib/apt/lists/* && \
   apt-get clean && \
   apt-get update && \
-  $APT_INSTALL software-properties-common && \
-  apt-add-repository -y ppa:brightbox/ruby-ng && \
-  apt-get update && \
   # Install openjdk 8.
   $APT_INSTALL openjdk-8-jdk && \
   update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java && \
   # Install build / source control tools
   $APT_INSTALL curl wget git maven ivy subversion make gcc lsof libffi-dev \
-    pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev && \
+    pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev
+
+ENV PATH "$PATH:/root/.pyenv/bin:/root/.pyenv/shims"
+RUN curl -L https://github.com/pyenv/pyenv-installer/raw/dd3f7d0914c5b4a416ca71ffabdf2954f2021596/bin/pyenv-installer | bash
 
 Review comment:
   Okay, this part seems not tested:
   
   ```bash
   Generating SQL API Markdown files.
   20/03/31 06:41:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   Traceback (most recent call last):
     File "/opt/spark-rm/output/spark/sql/gen-sql-api-docs.py", line 21, in <module>
       from pyspark.java_gateway import launch_gateway
     File "/opt/spark-rm/output/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 51, in <module>
     File "/opt/spark-rm/output/spark/python/lib/pyspark.zip/pyspark/context.py", line 22, in <module>
   ImportError: No module named threading
   log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
   log4j:WARN Please initialize the log4j system properly.
   log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
   ```
   
   Seems the installed Python is weird. [`threading`]( https://docs.python.org/3.7/library/threading.html) is the standard Python library that has existed from Python 2 to Python 3, but seems not existent with the Python installed here.
   
   I am going to revert this to unblock RC preparations.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606498086
 
 
   Yea we should use the script in branch-3.0 so this doesn't block the RC. But we should still revert this PR as it doesn't work.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HeartSaVioR commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591217692
 
 
   It only says for bugs, while we're saying improvements. Anyway I will go forward - let's not pollute the original topic.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HeartSaVioR edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591210981

> There's nothing confusing to "Affected Version" on improvement type. It just targets where you want that fix to be landed.

In fact we don't seem to guide like that. For example, suppose someone found a point to improve in Spark 2.4.x - they would set "Affected Version" to be 2.4.x (or try to set it to 2.5.0 and realize it doesn't work) as it may be the only version they know about, but as I see some guides on the fly we have been guiding to the latest "unreleased" minor version which they never know about.

Anyway let's not continue discussion on this in here. I'll initiate the discussion thread on dev@ soon.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587907892
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r387406314
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
 The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
 Python, R and SQL.
 
-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].
 
 Review comment:
   I think docker changes are fine.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r387406386
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   Okie. I am good with this change.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595570469
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r382040064

Review comment:
I don't know much about these tools, but updating the docs to point to anything newer, if it sounds good to someone looking closely at this part, seems OK.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on a change in pull request #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r381436287
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -31,15 +31,41 @@ whichever version of Spark you currently have checked out of revision control.
 The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
 Python, R and SQL.
 
-You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
-[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
-installed. Also install the following libraries:
+You need to have Ruby 2 and Python 3 installed. A handy way to install and manage various versions of Ruby and Python is with [`rbenv`] and [`pyenv`].
+
+[`rbenv`]: https://github.com/rbenv/rbenv
+[`pyenv`]: https://github.com/pyenv/pyenv
+
+On macOS you can install them with Homebrew:
+
+```sh
+brew install rbenv pyenv
+```
+
+To activate them, you'll need to run these commands or add them to the end of your `.bash_profile`:
+
+```sh
+eval "$(rbenv init -)"
+eval "$(pyenv init -)"
 
 Review comment:
   One alternative to running these `init` commands is to prefix the below commands so that they do the right thing without depending on any environment variables being set by `init`.
   
   So, for example, `pip install` would become `pyenv exec pip install` and `jekyll` would become `rbenv exec jekyll`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595497772
 
 
   **[Test build #119425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119425/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588403216

No, I didn't aim to land it in branch-2.4. As I noted over on https://github.com/apache/spark/pull/27626#issuecomment-588391223, I don't know why the MkDocs patch was landed there. I create the JIRA with an "affects version" of 3.0.0 and didn't mention 2.4 anywhere. I didn't ask to backport any improvements, and I don't know why branch-2.4 became a point of discussion in any of these 3 PRs.

> It seems that you are intentional because you don't understand the community policy, let me fix the situation first.

I wasn't trying to circumvent community policy in any way. I'm happy to create any new JIRAs required, but it looks like you've gone ahead and done that yourself.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

cloud-fan edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606443714
 
 
   We can manually test it by changing the `ENTRYPOINT` in `Dockerfile` to `/bin/bash` and check if the python and ruby are installed correctly.
   
   The release script doesn't work anymore after this patch because we use a non-standard way to install python/ruby (download a bash script and run it):
   1. The actual work is done by user `spark-rm`, which can't access `/root/.pyenv`
   2. Even if we fix the permission issue, the installed python doesn't work well with the system libraries, and have errors like `ImportError: No module named threading`
   3. It's wrong to append the `PATH`, and we should prepend it. Otherwise `python` still points to the system default which is 2.7.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606443714
 
 
   We can manually test it by changing the `ENTRYPOINT` in `Dockerfile` to `/bin/bash` and check if the python and ruby are installed correctly.
   
   The release script doesn't work anymore after this patch because we use a non-standard way to install python/ruby (download a bash script and run it):
   1. The actual work is done by user `spark-rm`, which can't access `/root/.pyenv`
   2. Even if we fix the permission issue, the installed python doesn't work well with the system libraries, and have errors like `ImportError: No module named threading`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584402678
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22949/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587944906
 
 
   **[Test build #118653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118653/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

dongjoon-hyun commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-588433227
 
 
   Oh, in that case, please use different IDs from the beginning next time especially.
   > nobody asked me to split up the PR! 
   Thanks, @nchammas .

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595971834
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-585596381
 
 
   Should I break up this PR to make it easier to review? e.g.
   1. Makefile trim
   2. pages -> nav
   3. pyenv and rbenv + dependency pinning
   
   Asking just in case the silence from other reviewers is because this PR is too big.
   
   If y'all just haven't gotten around to taking a look, no worries. I'm being proactive.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595909097
 
 
   **[Test build #119478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119478/testReport)** for PR 27534 at commit [`ebba668`](https://github.com/apache/spark/commit/ebba66868f9833cbf2771476b609bfefaf8e946c).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-584402199
 
 
   **[Test build #118187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118187/testReport)** for PR 27534 at commit [`bc15b86`](https://github.com/apache/spark/commit/bc15b8652efe1d500c801023ce3d102f7a7953fe).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591069010
 
 
   I'd like to move this PR forward. What path would y'all like to take here?
   
   Options are:
   1. Keep discussing the possibility of using rbenv and pyenv, and continue to refine the PR as it is.
   2. Scale back the PR to just pin the various Python and Ruby requirements to specific versions.
   3. Close this PR and leave the doc building process as-is.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595909677
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595498091
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r385479036
 
 

 ##########
 File path: docs/README.md
 ##########
 @@ -58,7 +84,7 @@ Note: Other versions of roxygen2 might work in SparkR documentation generation b
 To generate API docs for any language, you'll need to install these libraries:
 
 ```sh
-$ sudo pip install sphinx mkdocs numpy
+pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
 
 Review comment:
   Hm, there's tradeoff to pin the version (test and fix it compatible first vs easily breaking). I think we haven't faced any notable problem so far to maintain those implicit compatibility across versions. What about we pin the version when actually we see some overhead to maintain this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587927030
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23408/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r388686538

Review comment:
Or alternatively, we can have one section to describe use `rbenv` and `pyenv` separately .. that works to me too.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-591214680
 
 
   This is written in the guide:
   
   > Affects Version. For Bugs, assign at least one version that is known to exhibit the problem or need the change
   
   Ideally they should search JIRAs and see if there are duplicates. If there are duplicate, we should resolve it as a duplicate. We can clarify more, of course. However, I don't see that this is confusing or a bad pattern yet. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

srowen commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-592063962
 
 
   Just to be clear, any objections to merging?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

nchammas commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-606791725
 
 
   I can try taking another stab at this change for the future, now that I have a better idea of how to test it.
   
   However, given that this change was met with some resistance to begin with, I won't try to fix it unless there is explicit committer interest.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587926442
 
 
   **[Test build #118656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118656/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587730563
 
 
   **[Test build #118653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118653/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins removed a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-595909677
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

SparkQA commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587974613
 
 
   **[Test build #118656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118656/testReport)** for PR 27534 at commit [`2b1b42f`](https://github.com/apache/spark/commit/2b1b42f9fafc40e022eb5f6adf038c5678ce2173).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

AmplabJenkins commented on issue #27534: [SPARK-30731][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#issuecomment-587907934
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118652/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs

Posted by GitBox <gi...@apache.org>.

HyukjinKwon commented on a change in pull request #27534: [SPARK-30879][DOCS] Refine workflow for building docs
URL: https://github.com/apache/spark/pull/27534#discussion_r400686586
 
 

 ##########
 File path: dev/create-release/spark-rm/Dockerfile
 ##########
 @@ -50,36 +46,43 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
   rm -rf /var/lib/apt/lists/* && \
   apt-get clean && \
   apt-get update && \
-  $APT_INSTALL software-properties-common && \
-  apt-add-repository -y ppa:brightbox/ruby-ng && \
-  apt-get update && \
   # Install openjdk 8.
   $APT_INSTALL openjdk-8-jdk && \
   update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java && \
   # Install build / source control tools
   $APT_INSTALL curl wget git maven ivy subversion make gcc lsof libffi-dev \
-    pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev && \
+    pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev
+
+ENV PATH "$PATH:/root/.pyenv/bin:/root/.pyenv/shims"
+RUN curl -L https://github.com/pyenv/pyenv-installer/raw/dd3f7d0914c5b4a416ca71ffabdf2954f2021596/bin/pyenv-installer | bash
 
 Review comment:
   Okay, I suspect it was not tested due to the limitation described in the PR description:
   
   ```bash
   Generating SQL API Markdown files.
   20/03/31 06:41:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   Traceback (most recent call last):
     File "/opt/spark-rm/output/spark/sql/gen-sql-api-docs.py", line 21, in <module>
       from pyspark.java_gateway import launch_gateway
     File "/opt/spark-rm/output/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 51, in <module>
     File "/opt/spark-rm/output/spark/python/lib/pyspark.zip/pyspark/context.py", line 22, in <module>
   ImportError: No module named threading
   log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
   log4j:WARN Please initialize the log4j system properly.
   log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
   ```
   
   Seems the installed Python is weird. [`threading`]( https://docs.python.org/3.7/library/threading.html) is the standard Python library that has existed from Python 2 to Python 3, but seems not existent with the Python installed here.
   
   Let me revert this to make RC preparation easier.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org