You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "yiaany (via GitHub)" <gi...@apache.org> on 2026/08/30 09:16:57 UTC
[PR] docs: add full-site version snapshots [datafusion]
yiaany opened a new pull request, #24792:
URL: https://github.com/apache/datafusion/pull/24792
## Which issue does this PR close?
- Closes #17071.
## Rationale for this change
The documentation published at https://datafusion.apache.org/ currently follows
`main`, so it can describe behavior that differs from released DataFusion
versions.
This PR introduces a complete immutable documentation snapshot for DataFusion
`55.0.0`, while keeping the root site on the latest `main` documentation. The
approach follows the model discussed in #17071: build the complete tagged site
into one version directory and use the PyData Sphinx Theme version switcher.
## What changes are included in this PR?
- Adds a site-wide version picker for `Development` and `55.0.0`.
- Builds the exact `55.0.0` tag into `/versions/55.0.0/`.
- Verifies that tag `55.0.0` resolves to commit
`d5552342012888b7d1a3ab88d92e3d292fc0cde0`.
- Preserves tagged documentation content, including the existing
"has not been released yet" sentence.
- Pins release links that would otherwise point to mutable `main` or `latest`
content.
- Preserves all existing immutable version directories during normal
documentation deployment.
- Deletes stale current-site output without deleting archived releases.
- Uses serialized, non-force deployment pushes.
- Adds generated-site validation and regression tests.
- Documents manual snapshot publication and the initial bootstrap sequence.
- Excludes Sphinx build caches and environment-specific metadata from the
published snapshot.
- Refuses to replace an existing assembly output directory.
This PR does not publish generated documentation, modify `asf-site`, backfill
older releases, add `latest` or `stable` aliases, or automate future release
publication.
## Are these changes tested?
Yes. I ran:
- `git diff --check`
- `./ci/scripts/doc_prettier_check.sh`
- `bash ci/scripts/markdown_link_check.sh`
- `./ci/scripts/typos_check.sh`
- `./ci/scripts/license_header.sh`
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./dev/rust_lint.sh`
- 17 versioned-documentation unit and integration tests
- current documentation build with warnings treated as errors
- exact-tag `55.0.0` full-site snapshot build
- final deployment assembly and generated-site validation
- browser smoke tests over HTTP for version switching, missing-page fallback,
mobile layout, and JavaScript-disabled release identification
The final snapshot contains no `.doctrees`, `.buildinfo`, temporary build paths,
or generated artifacts tracked in Git.
## Are there any user-facing changes?
Yes.
The current documentation remains at the site root. Users can switch between:
- `Development`: https://datafusion.apache.org/
- `55.0.0`: https://datafusion.apache.org/versions/55.0.0/
The initial `55.0.0` snapshot must be published manually.
## Bootstrap coordination
This PR intentionally remains a draft until maintainers choose the initial
publication sequence.
The current `asf-site` branch does not yet contain `versions/55.0.0/`, and
deployment validation intentionally fails closed until the snapshot exists.
Could a maintainer confirm which bootstrap procedure should be used?
1. Merge a preliminary retention-only workflow change, then publish the
snapshot manually.
2. Use a coordinated window: verify that no old documentation deployment is
running or queued, publish the snapshot, merge this PR, and verify that the
new deployment retains it.
I have not modified or published anything to `asf-site`.
I reviewed the implementation end-to-end and can explain the exact-tag build,
publication boundary, archive retention, validator, and deployment sequence
during review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add full-site version snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-5656079625
Thank you @yiaany -- I had not seen this -- I will review it tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add full-site version snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-5873102246
I am so sorry -- yes this is one of those important but not urgent PRs that I have let slip down. 😢
I will test it out day
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-6131101236
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24792?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.82%. Comparing base ([`6663e43`](https://app.codecov.io/gh/apache/datafusion/commit/6663e4348049cc7b70109c9e0f8e39049794ea23?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2bf479a`](https://app.codecov.io/gh/apache/datafusion/commit/2bf479acb6f4a9b374705db8c600599aabfdc31a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24792 +/- ##
==========================================
- Coverage 82.83% 82.82% -0.01%
==========================================
Files 1149 1149
Lines 452344 452344
Branches 452344 452344
==========================================
- Hits 374680 374670 -10
- Misses 54864 54871 +7
- Partials 22800 22803 +3
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24792?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-6446457683
> Thanks @alamb for pinging me, the PR makes sense to me!
Thank you @xudong963
I thought more about this last night and I would really like to reduce the manual steps for releasing. So what I plan to do is make a github action to rebuild the docs for old versions based on the contents of picker.json
That way to add a new version to the docs, we'll just make a PR to main to update the contents of the pick list.
Will work on that next
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-6467230380
> I thought more about this last night and I would really like to reduce the manual steps for releasing. So what I plan to do is make a github action to rebuild the docs for old versions based on the contents of picker.json
Here is a PR that sets up generating new doc versions when versions.json is updated:
- https://github.com/apache/datafusion/pull/26828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "yiaany (via GitHub)" <gi...@apache.org>.
yiaany commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-5895440362
> Thank you @yiaany
>
> I apologize for the delay in reviewing. I was struggling to find enough time to write a proper response.
>
> At a high level, I think this PR is too complicated for what it is doing-- which is probably my fault for not defining it more specifically. I apologize for not being clear
>
> I was hoping:
>
> 1. a static version list (you have it in versions.json I think)
> 2. documentation of how old versions are published as part of the release process
>
> I don't think we need to change the current CI for publishing / building documentation
>
> Then I was imagining we test it locally like:
>
> 1. manually build docs for a few versions (55.1.0 and 55.0.0 for example) and make a PR to the [asf-site](https://github.com/apache/datafusion/tree/asf-site) branch with the proposed layouts
> 2. Build the docs from this PR
> 3. Check out [asf-site](https://github.com/apache/datafusion/tree/asf-site) branch manually
> 4. move the manually built docs from the PR and put the versioned docs in the right plce
>
> If that looks good we could merge this main PR and I think the main docs site would be updated.
>
> I am not sure how much value all the various python test scripts add. If you think they are important, we should document clearly their intent and what types of regressions / breakages they protect again
>
> > Could a maintainer confirm which bootstrap procedure should be used?
>
> I suggest Build older site versions one statically / part of the release process (not via a CI action)
>
> Finally, this might be easier to test if you setup your fork so it published your forks `asf-site` branch as a github pages, and then test out the code / picker there. That would also make it easy for other reviewers to test it out / see what it looks like
Hi @alamb, thank you for the detailed review. You were right: I made the first version too complicated.
I’ve simplified #24792 to a static version list, the PyData version picker, and instructions for building and publishing release docs manually. I removed the custom build, deployment, validation, and test scripts. The existing docs CI and publishing workflow are unchanged in this PR.
I built the docs from the 55.0.0 and 55.1.0 tags and tested the version picker and missing-page fallback locally in a browser. The proposed generated layout is in #25881.
I also found that the current deployment’s rsync --delete would remove manually published release docs. I put the one-line fix in a separate PR, #25880. That change needs to land before the snapshots are published.
I haven’t set up a public GitHub Pages preview yet. Could you take another look at the smaller approach and let me know whether this matches what you had in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#discussion_r4228383620
##########
dev/release/README.md:
##########
@@ -313,6 +313,68 @@ git tag 55.2.0
git push apache 55.2.0
```
+#### Publish the versioned documentation
+
+**1. Build the documentation from the release tagm using the a configuration
Review Comment:
```suggestion
**1. Build the documentation from the release tag using the a configuration
```
##########
dev/release/README.md:
##########
@@ -313,6 +313,68 @@ git tag 55.2.0
git push apache 55.2.0
```
+#### Publish the versioned documentation
+
+**1. Build the documentation from the release tagm using the a configuration
+overlay (`docs/scripts/release/conf.py`) from the main checkout. For example,
+for `55.0.0`, from the repository root:
+
+```shell
+export DATAFUSION_DOCS_OVERLAY="$PWD/docs/scripts/release"
+git fetch apache tag 55.0.0
+git worktree add --detach /tmp/datafusion-55.0.0 55.0.0
+cd /tmp/datafusion-55.0.0/docs
+DATAFUSION_DOCS_SOURCE="$PWD/source" DATAFUSION_DOCS_VERSION=55.0.0 \
+ SPHINXOPTS="-W -c $DATAFUSION_DOCS_OVERLAY" \
+ uv run --package datafusion-docs --with sphinx-sitemap ./build.sh
+cd -
+```
+
+Notes:
+1. You need ot use an absolute path to the overlay in your current checkout.
+2. The `--with sphinx-sitemap` option supplies the sitemap extension for older tags.
+
+prepare a PR targeting `asf-site` that contains the
+documentation for this release. (For example
+[#25881](https://github.com/apache/datafusion/pull/25881)).
+
+```shell
+mkdir -p docs/build/html/versions/55.0.0
+rsync -a /tmp/datafusion-55.0.0/docs/build/html/ docs/build/html/versions/55.0.0/
+python3 -m http.server --directory docs/build/html 8000
+```
+
+**2. Add the built site to the `asf-site` branch.** Open a PR against
+`asf-site` that adds the release under `versions/<version>/`:
+
+```shell
+git fetch apache asf-site
+git worktree add -b docs/publish-55.0.0 /tmp/datafusion-asf-site apache/asf-site
+rsync -a --exclude '/.buildinfo' \
+ /tmp/datafusion-55.0.0/docs/build/html/ \
+ /tmp/datafusion-asf-site/versions/55.0.0/
+cd /tmp/datafusion-asf-site
+git add versions/55.0.0
+git commit -m "Publish documentation for 55.0.0"
+git push origin docs/publish-55.0.0
+```
+
+The release is live at `https://datafusion.apache.org/versions/55.0.0/` once
+the PR is merged.
+
+**3. Add the release to the version picker.** Open a PR against `main` that
+adds the release to `docs/source/_static/versions.json`. Put the new release
+first and move `"preferred": true` to it, so that it is the default shown.
Review Comment:
I see no `"preferred": true` currently at https://github.com/apache/datafusion/pull/24792/changes#diff-99ab87d9590045ea1e68507c78e72d2034bf0505d2b96edd222b3fe0e97fea1aR2-R6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #24792:
URL: https://github.com/apache/datafusion/pull/24792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-6427382400
Ok, thanks @martin-g and @yiaany
I would like to merge this PR in now so we can see how it works
Then, I think there are some follow ons like
1. adding a "show versions banner" warning (apparently you can do it with the theme option `show_version_warning_banner`)
2. Automating the creation / building of docs from a release tag
cc @xudong963 as I think you are planning to run the 56 release. I can help with the docs part of it, so while you don't need to do anything I wanted you to be aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add full-site version snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#discussion_r4126808375
##########
docs/scripts/validate_site.py:
##########
@@ -0,0 +1,420 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Validate current and immutable full-site documentation output."""
Review Comment:
What needs to be validated? I don't undertstand why we need a 400 line python script here
##########
docs/scripts/assemble_site.py:
##########
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Replace the current site while retaining immutable full-site snapshots."""
Review Comment:
Why do we need to replace the current site? This is pretty confusing to me
##########
.github/workflows/docs.yaml:
##########
@@ -20,12 +20,19 @@ on:
branches:
- main
paths:
+ - .gitattributes
Review Comment:
i think these changes are unrelated to this PR (to trigger on changes to pyproject/uv) -- can you please make a separate PR to add them to main (it will be easier to review / merge) along with the rationale?
##########
docs/scripts/snapshot_site.py:
##########
@@ -0,0 +1,286 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Build one complete documentation snapshot from an exact release tag."""
Review Comment:
don't we already have a build.sh script to do this? It seems like a lot of python code to call a few scripts 🤔
##########
docs/README.md:
##########
@@ -19,63 +19,142 @@
# DataFusion Documentation
-This folder contains the source content of the [User Guide](./source/user-guide)
-and [Contributor Guide](./source/contributor-guide). These are both published to
-https://datafusion.apache.org/ as part of the release process.
+This folder contains the sources for https://datafusion.apache.org/. The root
+site is built continuously from `main`. Complete, immutable release sites are
+published under `/versions/<version>/`.
## Dependencies
-Install build dependencies and build the documentation using
+From the repository root, install the documentation dependencies using
[uv](https://docs.astral.sh/uv/):
```sh
-uv sync
-uv run bash build.sh
+uv sync --package datafusion-docs
```
The docs build regenerates the workspace dependency graph via
`docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph`
(`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot`
(`brew install graphviz` or `sudo apt-get install -y graphviz`) are available.
+`.gitattributes` keeps documentation shell scripts LF-terminated so the same
Review Comment:
this seems like a somewhat irrelevant detail 🤔
##########
docs/README.md:
##########
@@ -19,63 +19,142 @@
# DataFusion Documentation
-This folder contains the source content of the [User Guide](./source/user-guide)
-and [Contributor Guide](./source/contributor-guide). These are both published to
-https://datafusion.apache.org/ as part of the release process.
+This folder contains the sources for https://datafusion.apache.org/. The root
+site is built continuously from `main`. Complete, immutable release sites are
+published under `/versions/<version>/`.
## Dependencies
-Install build dependencies and build the documentation using
+From the repository root, install the documentation dependencies using
[uv](https://docs.astral.sh/uv/):
```sh
-uv sync
-uv run bash build.sh
+uv sync --package datafusion-docs
```
The docs build regenerates the workspace dependency graph via
`docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph`
(`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot`
(`brew install graphviz` or `sudo apt-get install -y graphviz`) are available.
+`.gitattributes` keeps documentation shell scripts LF-terminated so the same
+scripts run from Linux and WSL checkouts.
-## Build & Preview
+## Build and Preview
-Run the provided script to build the HTML pages.
+Build the current complete site from the repository root:
```bash
-# If using venv, ensure you have activated it
-./build.sh
+uv run --package datafusion-docs docs/build.sh
```
-The HTML will be generated into a `build` directory. Open `build/html/index.html`
-in your preferred browser, e.g.
+The HTML is generated in `docs/build/html`. Serve it over HTTP because browsers
+do not load the version manifest from `file:` URLs:
-Preview the site on Linux by running this command.
+```bash
+python3 -m http.server --directory docs/build/html 8000
+```
+
+Then open http://localhost:8000/.
+
+The public and assembled layouts are:
+
+```text
+https://datafusion.apache.org/
+|-- user-guide/
+|-- library-user-guide/
+|-- contributor-guide/
+|-- _static/versions.json
+`-- versions/55.0.0/
+ |-- user-guide/
+ |-- library-user-guide/
+ |-- contributor-guide/
+ |-- download.html
+ |-- search.html
+ |-- _sources/
+ |-- _static/
+ `-- sitemap.xml
+```
+
+## Release Snapshots
+
+`docs/source/_static/versions.json` is both the PyData version-picker manifest
+and the release catalog. It contains `Development` at the site root and records
+each release's semantic version, exact tag, and exact 40-character commit.
+
+The first snapshot is the lightweight tag `55.0.0`, which peels to
Review Comment:
"peels to"?
##########
.gitattributes:
##########
@@ -1,4 +1,6 @@
.github/ export-ignore
+docs/*.sh text eol=lf
Review Comment:
why is this needed?
##########
docs/README.md:
##########
@@ -19,63 +19,142 @@
# DataFusion Documentation
-This folder contains the source content of the [User Guide](./source/user-guide)
-and [Contributor Guide](./source/contributor-guide). These are both published to
-https://datafusion.apache.org/ as part of the release process.
+This folder contains the sources for https://datafusion.apache.org/. The root
+site is built continuously from `main`. Complete, immutable release sites are
Review Comment:
what does 'immutable' mean in this context? Maybe we should just say "released versions"?
##########
docs/tests/test_versioned_docs.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
what does this test?
##########
docs/README.md:
##########
@@ -19,63 +19,142 @@
# DataFusion Documentation
-This folder contains the source content of the [User Guide](./source/user-guide)
-and [Contributor Guide](./source/contributor-guide). These are both published to
-https://datafusion.apache.org/ as part of the release process.
+This folder contains the sources for https://datafusion.apache.org/. The root
+site is built continuously from `main`. Complete, immutable release sites are
+published under `/versions/<version>/`.
## Dependencies
-Install build dependencies and build the documentation using
+From the repository root, install the documentation dependencies using
[uv](https://docs.astral.sh/uv/):
```sh
-uv sync
-uv run bash build.sh
+uv sync --package datafusion-docs
```
The docs build regenerates the workspace dependency graph via
`docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph`
(`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot`
(`brew install graphviz` or `sudo apt-get install -y graphviz`) are available.
+`.gitattributes` keeps documentation shell scripts LF-terminated so the same
+scripts run from Linux and WSL checkouts.
-## Build & Preview
+## Build and Preview
-Run the provided script to build the HTML pages.
+Build the current complete site from the repository root:
```bash
-# If using venv, ensure you have activated it
-./build.sh
+uv run --package datafusion-docs docs/build.sh
```
-The HTML will be generated into a `build` directory. Open `build/html/index.html`
-in your preferred browser, e.g.
+The HTML is generated in `docs/build/html`. Serve it over HTTP because browsers
+do not load the version manifest from `file:` URLs:
-Preview the site on Linux by running this command.
+```bash
+python3 -m http.server --directory docs/build/html 8000
+```
+
+Then open http://localhost:8000/.
+
+The public and assembled layouts are:
Review Comment:
is this list necessary? It seems like it will just get out of date over time
##########
docs/README.md:
##########
@@ -19,63 +19,142 @@
# DataFusion Documentation
-This folder contains the source content of the [User Guide](./source/user-guide)
-and [Contributor Guide](./source/contributor-guide). These are both published to
-https://datafusion.apache.org/ as part of the release process.
+This folder contains the sources for https://datafusion.apache.org/. The root
+site is built continuously from `main`. Complete, immutable release sites are
+published under `/versions/<version>/`.
## Dependencies
-Install build dependencies and build the documentation using
+From the repository root, install the documentation dependencies using
Review Comment:
if this needs changing, perhaps you can make another standalone PR to update the build docs
##########
docs/source/_static/versions.json:
##########
@@ -0,0 +1,14 @@
+[
Review Comment:
Yes, this looks good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-6069102799
Thank you @yiaany
I started going through this and made some changes / added documentation about how to test this locally.
I need to take one more pass through the instructions and then I think it will be good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add version picker and manual release snapshots [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-6449098535
We have a version picker! Thank you so much @martin-g and @yiaany -- this is so cool
https://datafusion.apache.org/
<img width="1693" height="500" alt="Screenshot 2026-10-27 at 1 35 19 PM" src="https://github.com/user-attachments/assets/a635f85e-cb2a-41f7-bdd0-d3882b207043" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add full-site version snapshots [datafusion]
Posted by "yiaany (via GitHub)" <gi...@apache.org>.
yiaany commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-5868379681
Hi @alamb, just checking in on this when you have a chance. It has been about two weeks since your last message. The checks are still green and I have not changed the PR since then. Please let me know if you would like any changes or if there is anything I should prepare for the review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add full-site version snapshots [datafusion]
Posted by "yiaany (via GitHub)" <gi...@apache.org>.
yiaany commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-5645942405
Hi @alamb and maintainers, just following up on this draft PR. It has been open since August 30, 2026, and the implementation and local validation are complete. The remaining question is the bootstrap sequence for publishing `versions/55.0.0/` to `asf-site`. Could someone advise whether option 1 or option 2 in the PR description is preferred? I am happy to make any changes needed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add full-site version snapshots [datafusion]
Posted by "yiaany (via GitHub)" <gi...@apache.org>.
yiaany commented on PR #24792:
URL: https://github.com/apache/datafusion/pull/24792#issuecomment-5645939583
Hi @alamb and maintainers, just following up on this draft PR. It has been open since August 30, 2026, and the implementation and local validation are complete. The remaining question is the bootstrap sequence for publishing ersions/55.0.0/ to sf-site. Could someone advise whether option 1 or option 2 in the PR description is preferred? I’m happy to make any changes needed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org