You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "nchammas (via GitHub)" <gi...@apache.org> on 2026/07/05 14:12:56 UTC
[PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
nchammas opened a new pull request, #57010:
URL: https://github.com/apache/spark/pull/57010
### What changes were proposed in this pull request?
Add a couple of new `SKIP_` flags to the existing doc build script:
- `SKIP_SPARK_BUILD`
- `SKIP_UNIDOC`
### Why are the changes needed?
#48860 added `SPARK_DOCS_IS_BUILT_ON_HOST` but it doesn't really fit the existing interface of the doc build script, as clunky as it is. This PR tweaks the new skip flags so they are more consistent with the existing ones.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I ran the doc build as follows without issue:
```sh
SKIP_UNIDOC=1 SKIP_SPARK_BUILD=1 bundle exec jekyll build
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on code in PR #57010:
URL: https://github.com/apache/spark/pull/57010#discussion_r3559452405
##########
docs/_plugins/build_api_docs.rb:
##########
@@ -277,10 +281,12 @@ def build_spark_scala_and_java_docs_if_necessary
def build_scala_and_java_docs
build_spark_if_necessary
- print_header "Building Scala and Java API docs."
+ print_header "Building Scala and Java API docs"
cd(SPARK_PROJECT_ROOT)
- build_spark_scala_and_java_docs_if_necessary
+ if not (ENV['SKIP_UNIDOC'] == '1')
Review Comment:
@uros-b - I don't know how you feel about the broader discussion of skip flags, but for now I suggest we accept the inconsistency in how `SKIP_UNIDOC` is interpreted. In a follow-up PR I would like to resolve the various inconsistencies in how we use skip flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #57010:
URL: https://github.com/apache/spark/pull/57010#discussion_r3535264733
##########
docs/_plugins/build_api_docs.rb:
##########
@@ -277,10 +281,12 @@ def build_spark_scala_and_java_docs_if_necessary
def build_scala_and_java_docs
build_spark_if_necessary
- print_header "Building Scala and Java API docs."
+ print_header "Building Scala and Java API docs"
cd(SPARK_PROJECT_ROOT)
- build_spark_scala_and_java_docs_if_necessary
+ if not (ENV['SKIP_UNIDOC'] == '1')
Review Comment:
Please check possible latent inconsistency: SKIP_UNIDOC reuses a name that already exists cross-subsystem; dev/run-tests.py:307 reads it via truthiness and multiple non-doc CI jobs set SKIP_UNIDOC: true, whereas the doc script gates strictly on == '1'. No live conflict (the docs job at lines 1246-1577 sets neither new flag, so it builds both as before, and no CI path both exports SKIP_UNIDOC=true and runs the Jekyll build), but a user exporting SKIP_UNIDOC=true expecting the doc build to skip would be surprised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-4915650954
Part of what prompted the change to how the skip flags are handled is a combination of two factors:
1. The docs build script interface is informal.
2. In other places we are setting `SKIP_` flags to `true`, not `1`, and testing for any non-empty value.
Here are some examples:
https://github.com/apache/spark/blob/10e2aed582fb6c3b2aadf9ee260c3f16f35ecf5f/dev/run-tests.py#L307
https://github.com/apache/spark/blob/10e2aed582fb6c3b2aadf9ee260c3f16f35ecf5f/dev/run-tests.py#L653
https://github.com/apache/spark/blob/00fef6941f3f3958284aed9b50ad5d2798f3f997/dev/merge_spark_pr.py#L90-L92
This change does not break any existing users of the docs build script. Anyone setting `SKIP_{something}=1` and calling `jekyll` will get the same behavior as before.
In a follow-up PR, I would like to adjust the other uses of `SKIP_` flags so that we have a consistent convention of handling them as "truthy" values. i.e. Any non-empty value is interpreted as true.
https://github.com/apache/spark/blob/10e2aed582fb6c3b2aadf9ee260c3f16f35ecf5f/dev/run-tests.py#L647
https://github.com/apache/spark/blob/10e2aed582fb6c3b2aadf9ee260c3f16f35ecf5f/dev/run-tests.py#L657
I think having a consistent convention for how we use `SKIP_` flags will make the build infra a bit easier to understand.
Does my motivation make more sense now? I can move d61ae41 into a new PR if you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #57010:
URL: https://github.com/apache/spark/pull/57010#discussion_r3535271784
##########
docs/README.md:
##########
@@ -120,8 +120,11 @@ using [roxygen2](https://cran.r-project.org/web/packages/roxygen2/index.html) an
using [MkDocs](https://www.mkdocs.org/).
To control what API docs get built, you can set any combination of the following shell variables before you run `bundle exec jekyll build`:
-* `SKIP_API=1`: Skip building all the API docs.
-* `SKIP_SCALADOC=1`: Skip the Scala and Java API docs.
+* `SKIP_ERRORDOC=1`: Skip building the unified error documentation.
+* `SKIP_SPARK_BUILD=1`: Skip building Spark. A working Spark build is required for the docs (except the error docs) to build correctly, so set this only if you know you've already built Spark.
+* `SKIP_API=1`: Skip building all the API docs. This is basically the same as setting all of the skip flags below.
+* `SKIP_SCALADOC=1`: Skip the Scala and Java API docs, which includes building the Unidoc.
+* `SKIP_UNIDOC=1`: Skip building the Unidoc.
Review Comment:
Doc-wording nit: the SKIP_UNIDOC=1 entry ("Skip building the Unidoc") omits that the Scala/Java copy step (copy_and_update_scala_docs/copy_and_update_java_docs from target/scala-2.13/unidoc and target/javaunidoc) still runs, so the build fails there if no pre-built unidoc output exists. SKIP_SPARK_BUILD=1 carries the analogous "only set this if you've already built" caveat; SKIP_UNIDOC should too (use SKIP_SCALADOC=1 to skip the copy as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-4936045211
> For the following, not really true for anyone uses SKIP_{something}=1 or SKIP_{something}=0 before.
Anyone setting `SKIP_{something}=1` for docs will get the same behavior as before, and as far as I can tell our doc build infra doesn't explicitly set `SKIP_{something}=0` anywhere.
[1]: https://github.com/apache/spark/blob/79026ad70380d6e92503ea27b1f61f484c63dc8b/dev/spark-test-image-util/docs/build-docs#L52
For users running things interactively, the new `print_skip_flags` method prints a header that immediately makes it clear what flags are enabled for that run of the documentation build. So if someone is used to setting, for example, `SKIP_PYTHONDOC=0` during interactive runs, they'll get immediate feedback.
In other words, the project's use of skip flags is fairly limited and resolving the various inconsistencies is a very tractable, if minor, problem.
> I believe some part of this PR is not worthy enough to introduce those behavior changes as long as it works.
I would like in a follow-up PR to update the various usages of `SKIP_` across the project to make their interpretation consistent. That is, we either have them all work as enabled when set to a specific value like `1` or `true`, or we have them all work as enabled when set to any non-empty string.
It's a straightforward set of changes and will give us a convention to stick to going forward for build-related skip flags.
Are you against this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas closed pull request #57010: [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script
URL: https://github.com/apache/spark/pull/57010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-5169112611
I removed my previous change request on this PR although I'm still negative on this direction. It's because I don't want to block your contribution direction. Feel free to proceed in your way. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-5530947891
The `build-docs` script added in #48860 does not run as part of CI and is [currently broken](https://github.com/apache/spark/pull/58352#issuecomment-5455088866). This PR here refactors a skip flag added by that same PR.
I personally think it's best if we just remove `build-docs` or replace it with something that runs regularly on CI so we are not repeatedly discovering that it broke. But I don't know if others would support that. Because I am not sure about the future of `build-docs`, I will abandon this PR for now.
I will reintroduce `print_skip_flags` in a separate PR since that's useful by itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-4886400398
cc @panbingkun, who authored the referenced PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on code in PR #57010:
URL: https://github.com/apache/spark/pull/57010#discussion_r3873874691
##########
docs/_plugins/build_api_docs.rb:
##########
@@ -277,10 +281,12 @@ def build_spark_scala_and_java_docs_if_necessary
def build_scala_and_java_docs
build_spark_if_necessary
- print_header "Building Scala and Java API docs."
+ print_header "Building Scala and Java API docs"
cd(SPARK_PROJECT_ROOT)
- build_spark_scala_and_java_docs_if_necessary
+ if not (ENV['SKIP_UNIDOC'] == '1')
Review Comment:
The overloaded name here bothered me, so I updated the new skip flag for the doc build to `SKIP_UNIDOC_BUILD`. That way we don't have the inconsistent interpretation of value (truthy vs. `1`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on code in PR #57010:
URL: https://github.com/apache/spark/pull/57010#discussion_r3537031462
##########
docs/_plugins/build_api_docs.rb:
##########
@@ -277,10 +281,12 @@ def build_spark_scala_and_java_docs_if_necessary
def build_scala_and_java_docs
build_spark_if_necessary
- print_header "Building Scala and Java API docs."
+ print_header "Building Scala and Java API docs"
cd(SPARK_PROJECT_ROOT)
- build_spark_scala_and_java_docs_if_necessary
+ if not (ENV['SKIP_UNIDOC'] == '1')
Review Comment:
I've updated the doc build to accept any non-empty value for its skip flags. I've updated the PR description accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-5169284450
I have reverted the changes to the interpretation of doc build skip flags. They work as they did before, with `1` being the target value that activates the skip behavior.
I would prefer a common convention for skip flags, but I understand the preference to leave stable behavior alone. For now, I will set aside the proposal to cleanup the skip flags across the project. I appreciate your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57938][INFRA] Add new `SKIP_` flags to doc build script [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57010:
URL: https://github.com/apache/spark/pull/57010#issuecomment-4929238113
I'm already aware of the current environment usages are ad-hoc, @nchammas .
For the following, not really true for anyone uses `SKIP_{something}=1` or `SKIP_{something}=0` before.
> This change does not break any existing users of the docs build script. Anyone setting SKIP_{something}=1 and calling jekyll will get the same behavior as before.
I believe some part of this PR is not worthy enough to introduce those behavior changes as long as it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org