You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/05/11 18:56:59 UTC
[PR] Document how our documentation works [airflow]
potiuk opened a new pull request, #50464:
URL: https://github.com/apache/airflow/pull/50464
This is high-level description of the architecture and workflows of our documentation building and publishing.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084979151
##########
docs/README.md:
##########
@@ -37,3 +54,196 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it work
+
+# Diagrams of the documentation architecture
+
+This is the diagram of live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Note that this just publishes the documentation but does not update the "site" with version numbers or
+stable links to providers and airflow - if you release a new documentation version it will be available
+with direct URL (say https://apache.airflow.org/docs/apache-airflow/3.0.1/) but the main site will still
+point to previous version of the documentation as `stable` and the version drop-downs will not be updated.
+
+In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository. This will build the website and publish it to the `publish`
+branch of `airflow-site` repository, including refreshing of the version numbers in the drop-downs and
+stable links.
+
+
+
+
+Some time after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TODO: IMPLEMENT THIS, FOR NOW IT HAS
+TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
+workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
+snapshots of the `live` documentation.
+
+## Publishing changes to the website (including theme)
+
+The workflows in `apache-airflow` only update the documentation for the packages (Airflow, Helm chart,
+Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website
+itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.
+
+Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when
+the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered
+manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to
+`publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is
+published as the official website. This includes any changes to `.htaccess` of the website.
+
+Such a main build also publishes latest "sphinx-airflow-theme" package to GitHub so that the next build
+of documentation can automatically pick it up from there. This means that if you want to make changes to
+`javascript` or `css` that are part of the theme, you need to do it in `ariflow-site` repository and
+merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository
+and pick up the latest version of the theme.
+
+The version of sphinx theme is fixed in both repositories:
+
+* https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21
+* https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section
+
+In case of bigger changes to the theme, we
+can first iterate on the website and merge a new theme version, and only after that we can switch to the new
+version of the theme.
+
+
+# Fixing historical documentation
+
+Sometimes we need to update historical documentation (modify generated `html`) - for example when we find
+bad links or when we change some of the structure in the documentation. This can be done via the
+`airflow-site-archive` repository. The workflow is as follows:
+
+1. Get the latest version of the documentation from S3 to `airflow-site-archive` repository using
+ `Sync s3 to GitHub` workflow. This will download the latest version of the documentation from S3 to
+ `airflow-site-archive` repository (this should be normally not needed, if automated synchronization works).
+2. Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any
+ text editors, scripts etc. Those files are generated as `html` files and are not meant to be regenerated,
+ they should be modified as `html` files in-place
+3. Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.
+4. Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified
+ documentation to S3 bucket.
+5. You can choose, whether to sync the changes to `live` or `staging` bucket. The default is `live`.
+6. By default the workflow will synchronize all documentation modified in single - last commit pushed to
+ the branch you specified. You can also specify "full_sync" to synchronize all files in the repository.
+7. In case you specify "full_sync", you can also synchronize `all` docs or only selected documentation
+ packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify
+ more than one package separated by spaces.
+8. After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered
+ automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there
+ is no need to merge your changes to `main` branch of `airflow-site-archive` repository. You can safely
+ delete the branch you created in step 3.
+
+
+
+
+## Manually publishing documentation directly to S3
Review Comment:
Sure. We can add "Ask in #internal-airflow-ci-cd channel on slack"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084150164
##########
docs/README.md:
##########
@@ -37,3 +54,189 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible
+via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
Review Comment:
```suggestion
via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084968415
##########
dev/README_RELEASE_PROVIDERS.md:
##########
@@ -486,137 +485,6 @@ If you want to disable this behaviour, set the env **CLEAN_LOCAL_TAGS** to false
breeze release-management tag-providers
```
-## Prepare documentation
-
-Documentation is an essential part of the product and should be made available to users.
-In our cases, documentation for the released versions is published in a separate repository -
-[`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code
-and build tools are available in the `apache/airflow` repository, so you have to coordinate between
-the two repositories to be able to build the documentation.
-
-Documentation for providers can be found in the `/docs/apache-airflow-providers` directory
-and the `/docs/apache-airflow-providers-*/` directory. The first directory contains the package contents
-lists and should be updated every time a new version of Provider distributions is released.
-
-- First, copy the airflow-site repository and set the environment variable ``AIRFLOW_SITE_DIRECTORY``.
-
-```shell script
-git clone https://github.com/apache/airflow-site.git airflow-site
-cd airflow-site
-export AIRFLOW_SITE_DIRECTORY="$(pwd -P)"
-```
-
-Note if this is not the first time you clone the repo make sure main branch is rebased:
-
-```shell script
-cd "${AIRFLOW_SITE_DIRECTORY}"
-git checkout main
-git pull --rebase
-```
-
-- Then you can go to the directory and build the necessary documentation packages
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs --clean-build apache-airflow-providers all-providers --include-removed-providers --include-commits
-```
-
-Usually when we release packages we also build documentation for the "documentation-only" packages. This
-means that unless we release just few selected packages or if we need to deliberately skip some packages
-we should release documentation for all Provider distributions and the above command is the one to use.
-
-If we want to just release some providers you can release them using package names:
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs apache-airflow-providers cncf.kubernetes sftp --clean-build --include-commits
-```
-
-Alternatively, if you have set the environment variable: `DISTRIBUTIONS_LIST` above, just run the command:
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs --clean-build --include-commits
-```
-
-Or using `--distributions-list` argument:
-
-```shell script
-breeze build-docs --distributions-list PACKAGE1,PACKAGE2 --include-commits
-```
-
-- Now you can preview the documentation.
-
-```shell script
-./docs/start_doc_server.sh
-```
-
-If you encounter error like:
-
-```shell script
-airflow git:(main) ./docs/start_doc_server.sh
-./docs/start_doc_server.sh: line 22: cd: /Users/eladkal/PycharmProjects/airflow/docs/_build: No such file or directory
-```
-
-That probably means that the doc folder is empty thus it can not build the doc server.
-This indicates that previous step of building the docs did not work.
-
-- Copy the documentation to the ``airflow-site`` repository
-
-All providers (including overriding documentation for doc-only changes) - note that publishing is
-way faster on multi-cpu machines when you are publishing multiple providers:
-
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-
-breeze release-management publish-docs apache-airflow-providers all-providers --include-removed-providers \
- --override-versioned --run-in-parallel
-
-breeze release-management add-back-references all-providers
-```
-
-If you have providers as list of provider ids because you just released them you can build them with
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-
-breeze release-management publish-docs amazon apache.beam google ....
-breeze release-management add-back-references all-providers
-```
-
-Alternatively, if you have set the environment variable: `DISTRIBUTIONS_LIST` above, just run the command:
-
-```shell script
-breeze release-management publish-docs
-breeze release-management add-back-references all-providers
-```
-
-Or using `--distributions-list` argument:
-
-```shell script
-breeze release-management publish-docs --distributions-list PACKAGE1,PACKAGE2
-breeze release-management add-back-references all-providers
-```
-
-
-Review the state of removed, suspended, new packages in
-[the docs index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md):
-
-- If you publish a new package, you must add it to the list of packages in the index.
-- If there are changes to suspension or removal status of a package you must move it appropriate section.
-
-- Create the commit and push changes.
-
-```shell script
-cd "${AIRFLOW_SITE_DIRECTORY}"
-branch="add-documentation-$(date "+%Y-%m-%d%n")"
-git checkout -b "${branch}"
-git add .
-git commit -m "Add documentation for packages - $(date "+%Y-%m-%d%n")"
-git push --set-upstream origin "${branch}"
-```
Review Comment:
Right!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #50464:
URL: https://github.com/apache/airflow/pull/50464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084153505
##########
docs/README.md:
##########
@@ -37,3 +54,189 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible
+via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Right after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TO DO: IMPLEMENT THIS, FOR NOW IT HAS
+TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
+workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
+snapshots of the `live` documentation.
+
+Also another thing that is triggered automatically (TODO: IMPLEMENT THIS) is building the `airflow-site`
+and re-publishing it. This is needed in order to refresh version numbers of published documentation. After
+you publish new version of the documentation, the "stable" version of the documentation is automatically
+replaced with the new version, however building the `airflow-site` is needed to make sure that the drop-down
+version numbers are updated properly.
+
+
+## Publishing changes to the website (including theme)
+
+The workflows in `apache-airflow` only update the documentation for the documentation packages (Airflow, Helm chart,
Review Comment:
```suggestion
The workflows in `apache-airflow` only update the documentation for the packages (Airflow, Helm chart,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084950436
##########
docs/README.md:
##########
@@ -37,3 +54,196 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it work
+
+# Diagrams of the documentation architecture
+
+This is the diagram of live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Note that this just publishes the documentation but does not update the "site" with version numbers or
+stable links to providers and airflow - if you release a new documentation version it will be available
+with direct URL (say https://apache.airflow.org/docs/apache-airflow/3.0.1/) but the main site will still
+point to previous version of the documentation as `stable` and the version drop-downs will not be updated.
+
+In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository. This will build the website and publish it to the `publish`
+branch of `airflow-site` repository, including refreshing of the version numbers in the drop-downs and
+stable links.
+
+
+
+
+Some time after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TODO: IMPLEMENT THIS, FOR NOW IT HAS
+TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
+workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
+snapshots of the `live` documentation.
+
+## Publishing changes to the website (including theme)
+
+The workflows in `apache-airflow` only update the documentation for the packages (Airflow, Helm chart,
+Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website
+itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.
+
+Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when
+the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered
+manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to
+`publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is
+published as the official website. This includes any changes to `.htaccess` of the website.
+
+Such a main build also publishes latest "sphinx-airflow-theme" package to GitHub so that the next build
+of documentation can automatically pick it up from there. This means that if you want to make changes to
+`javascript` or `css` that are part of the theme, you need to do it in `ariflow-site` repository and
+merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository
+and pick up the latest version of the theme.
+
+The version of sphinx theme is fixed in both repositories:
+
+* https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21
+* https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section
+
+In case of bigger changes to the theme, we
+can first iterate on the website and merge a new theme version, and only after that we can switch to the new
+version of the theme.
+
+
+# Fixing historical documentation
+
+Sometimes we need to update historical documentation (modify generated `html`) - for example when we find
+bad links or when we change some of the structure in the documentation. This can be done via the
+`airflow-site-archive` repository. The workflow is as follows:
+
+1. Get the latest version of the documentation from S3 to `airflow-site-archive` repository using
+ `Sync s3 to GitHub` workflow. This will download the latest version of the documentation from S3 to
+ `airflow-site-archive` repository (this should be normally not needed, if automated synchronization works).
+2. Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any
+ text editors, scripts etc. Those files are generated as `html` files and are not meant to be regenerated,
+ they should be modified as `html` files in-place
+3. Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.
+4. Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified
+ documentation to S3 bucket.
+5. You can choose, whether to sync the changes to `live` or `staging` bucket. The default is `live`.
+6. By default the workflow will synchronize all documentation modified in single - last commit pushed to
+ the branch you specified. You can also specify "full_sync" to synchronize all files in the repository.
+7. In case you specify "full_sync", you can also synchronize `all` docs or only selected documentation
+ packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify
+ more than one package separated by spaces.
+8. After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered
+ automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there
+ is no need to merge your changes to `main` branch of `airflow-site-archive` repository. You can safely
+ delete the branch you created in step 3.
+
+
+
+
+## Manually publishing documentation directly to S3
Review Comment:
Should we have a section that explains how to get permission to the S3?
I assume only specific persons have access to the account
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084147683
##########
docs/README.md:
##########
@@ -37,3 +54,189 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
Review Comment:
```suggestion
This is the diagram of live documentation architecture:
```
##########
docs/README.md:
##########
@@ -37,3 +54,189 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
Review Comment:
```suggestion
* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it work
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084940243
##########
dev/README_RELEASE_PROVIDERS.md:
##########
@@ -1283,6 +1136,38 @@ If you want to disable this behaviour, set the env **CLEAN_LOCAL_TAGS** to false
breeze release-management tag-providers
```
+## Publish documentation
+
+Documentation is an essential part of the product and should be made available to users.
+In our cases, documentation for the released versions is published in S3 bucket, and the site is
+kept in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site),
+but the documentation source code and build tools are available in the `apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for
+emergency cases.
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+
+You can specify the tag to use to build the docs and list of providers to publish (separated by spaces) or
+``all-providers`` in case you want to publish all providers (optionally you can exclude
+some of those providers)
+
+After that step, the provider documentation should be available under the usual urls (same as in PyPI packages)
+but stable links and drop-down boxes should not be updated.
Review Comment:
We should describe on which S3 path we do that.
As far as i remember you asked to do it on stage, review the docs and only if looks good rerun on live?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084977656
##########
dev/README_RELEASE_PROVIDERS.md:
##########
@@ -1283,6 +1136,38 @@ If you want to disable this behaviour, set the env **CLEAN_LOCAL_TAGS** to false
breeze release-management tag-providers
```
+## Publish documentation
+
+Documentation is an essential part of the product and should be made available to users.
+In our cases, documentation for the released versions is published in S3 bucket, and the site is
+kept in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site),
+but the documentation source code and build tools are available in the `apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for
+emergency cases.
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+
+You can specify the tag to use to build the docs and list of providers to publish (separated by spaces) or
+``all-providers`` in case you want to publish all providers (optionally you can exclude
+some of those providers)
+
+After that step, the provider documentation should be available under the usual urls (same as in PyPI packages)
+but stable links and drop-down boxes should not be updated.
Review Comment:
We do not have yet "staging" workflow enabled. Live is the only one that works - we will update it later, but live should be used only for now. I will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50464:
URL: https://github.com/apache/airflow/pull/50464#issuecomment-2870183341
I also added description on how to manually do the same without GitHub Actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084151834
##########
docs/README.md:
##########
@@ -37,3 +54,189 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible
+via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Right after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TO DO: IMPLEMENT THIS, FOR NOW IT HAS
Review Comment:
```suggestion
repository is automatically synchronized with the live S3 bucket. TODO: IMPLEMENT THIS, FOR NOW IT HAS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2083628581
##########
docs/README.md:
##########
@@ -37,3 +53,162 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible
+via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Right after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TO DO: IMPLEMENT THIS, FOR NOW IT HAS
+TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
+workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
+snapshots of the `live` documentation.
+
+Also another thing that is triggered automatically (TODO: IMPLEMENT THIS) is building the `airflow-site`
+and re-publishing it. This is needed in order to refresh version numbers of published documentation. After
+you publish new version of the documentation, the "stable" version of the documentation is automatically
+replaced with the new version, however building the `airflow-site` is needed to make sure that the drop-down
+version numbers are updated properly.
+
+
+## Publishing changes to the website (including theme)
+
+The workflows in `apache-airflow` only update the documentation for the documentation packages (Airflow, Helm chart,
+Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website
+itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.
+
+Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when
+the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered
+manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to
+`publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is
+published as the official website. This includes any changes to `.htaccess` of the website.
+
+Such a main build also publishes latest "sphinx-airflow-theme" package to GitHub so that the next build
+of documentation can automatically pick it up from there. This means that if you want to make changes to
+`javascript` or `css` that are part of the theme, you need to do it in `ariflow-site` repository and
+merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository
+and pick up the latest version of the theme.
+
+The version of sphinx theme is fixed in both repositories:
+
+* https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21
+* https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section
+
+In case of bigger changes to the theme, we
+can first iterate on the website and merge a new theme version, and only after that we can switch to the new
+version of the theme.
+
+
+# Fixing historical documentation
+
+Sometimes we need to update historical documentation (modify generated `html`) - for example when we find
+bad links or when we change some of the structure in the documentation. This can be done via the
+`airflow-site-archive` repository. The workflow is as follows:
+
+* Get the latest version of the documentation from S3 to `airflow-site-archive` repository using
+ `Sync s3 to GitHub` workflow. This will download the latest version of the documentation from S3 to
+ `airflow-site-archive` repository (this should be normally not needed, if automated synchronization works).
+
+* Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any
+ text editors, scripts etc. Those files are generated as `html` files and are not meant to be regenerated,
+ they should be modified as `html` files in-place
+
+* Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.
+
+* Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified
+ documentation to S3 bucket.
+
+* You can choose, whether to sync the changes to `live` or `staging` bucket. The default is `live`.
+
+* By default the workflow will synchronize all documentation modified in single - last commit pushed to
+ the branch you specified. You can also specify "full_sync" to synchronize all files in the repository.
+
+* In case you specify "full_sync", you can also synchronize `all` docs or only selected documentation
+ packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify
+ more than one package separated by spaces.
+
+* After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered
+ automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there
+ is no need to merge your changes to `main` branch of `airflow-site-archive` repository.
+
+
+
+
+## Manually publishing documentation
+
+The regular publishing workflows involve running Github Actions workflow and they cover majority of cases,
+however sometimes some manual updates and cherry-picks are needed, when we discover problems with the
+publishing and doc building code - for example when we find that we need to fix extensions to sphinx.
+
+In such case, release manager or a committer can build and publish documentation locally - providing that
+they configure AWS credentials to be able to upload files to S3.
+
+You can checkout locally a version of airflow repo that you need and apply any cherry-picks you need before
+running publishing.
+
+This is done using breeze. You also need to have aws CLI installed and configured credentials to be able
+to upload files to S3. You can get credentials from one of the admins of Airflow's AWS account. The
+region to set for AWS is `us-east-2`.
+
Review Comment:
Added the second manual update option and mentioned `--dry-run`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2083615920
##########
docs/README.md:
##########
@@ -37,3 +53,162 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible
+via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Right after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TO DO: IMPLEMENT THIS, FOR NOW IT HAS
+TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
+workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
+snapshots of the `live` documentation.
+
+Also another thing that is triggered automatically (TODO: IMPLEMENT THIS) is building the `airflow-site`
+and re-publishing it. This is needed in order to refresh version numbers of published documentation. After
+you publish new version of the documentation, the "stable" version of the documentation is automatically
+replaced with the new version, however building the `airflow-site` is needed to make sure that the drop-down
+version numbers are updated properly.
+
+
+## Publishing changes to the website (including theme)
+
+The workflows in `apache-airflow` only update the documentation for the documentation packages (Airflow, Helm chart,
+Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website
+itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.
+
+Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when
+the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered
+manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to
+`publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is
+published as the official website. This includes any changes to `.htaccess` of the website.
+
+Such a main build also publishes latest "sphinx-airflow-theme" package to GitHub so that the next build
+of documentation can automatically pick it up from there. This means that if you want to make changes to
+`javascript` or `css` that are part of the theme, you need to do it in `ariflow-site` repository and
+merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository
+and pick up the latest version of the theme.
+
+The version of sphinx theme is fixed in both repositories:
+
+* https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21
+* https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section
+
+In case of bigger changes to the theme, we
+can first iterate on the website and merge a new theme version, and only after that we can switch to the new
+version of the theme.
+
+
+# Fixing historical documentation
+
+Sometimes we need to update historical documentation (modify generated `html`) - for example when we find
+bad links or when we change some of the structure in the documentation. This can be done via the
+`airflow-site-archive` repository. The workflow is as follows:
+
+* Get the latest version of the documentation from S3 to `airflow-site-archive` repository using
+ `Sync s3 to GitHub` workflow. This will download the latest version of the documentation from S3 to
+ `airflow-site-archive` repository (this should be normally not needed, if automated synchronization works).
+
+* Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any
+ text editors, scripts etc. Those files are generated as `html` files and are not meant to be regenerated,
+ they should be modified as `html` files in-place
+
+* Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.
+
+* Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified
+ documentation to S3 bucket.
+
+* You can choose, whether to sync the changes to `live` or `staging` bucket. The default is `live`.
+
+* By default the workflow will synchronize all documentation modified in single - last commit pushed to
+ the branch you specified. You can also specify "full_sync" to synchronize all files in the repository.
+
+* In case you specify "full_sync", you can also synchronize `all` docs or only selected documentation
+ packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify
+ more than one package separated by spaces.
+
+* After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered
+ automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there
+ is no need to merge your changes to `main` branch of `airflow-site-archive` repository.
+
+
+
+
+## Manually publishing documentation
+
+The regular publishing workflows involve running Github Actions workflow and they cover majority of cases,
+however sometimes some manual updates and cherry-picks are needed, when we discover problems with the
+publishing and doc building code - for example when we find that we need to fix extensions to sphinx.
+
+In such case, release manager or a committer can build and publish documentation locally - providing that
+they configure AWS credentials to be able to upload files to S3.
+
+You can checkout locally a version of airflow repo that you need and apply any cherry-picks you need before
+running publishing.
+
+This is done using breeze. You also need to have aws CLI installed and configured credentials to be able
+to upload files to S3. You can get credentials from one of the admins of Airflow's AWS account. The
+region to set for AWS is `us-east-2`.
+
Review Comment:
We can add a note about dry running a when using manual `breeze release-management publish-docs-to-s3` would better always.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50464:
URL: https://github.com/apache/airflow/pull/50464#issuecomment-2870056451
We still have few todos:
* complete staging setup - we have to setup staging website via .asf.yaml (but we need to know what's up wiht the proxy refreshing) - I do not want to mess-around with .asf.yaml until https://issues.apache.org/jira/browse/INFRA-26814 is solved.
* we need to automate running syncing of changes from live bucket to main of `apache-airflow-site-archive` - for now we manually sync those, but we need to either trigger them or schedule syncing in pull mode (The latter is likely a bit better as it will not need PAT to be stored as secret).
But this one generally describes the flows that we will have once this is complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2083625286
##########
docs/README.md:
##########
@@ -37,3 +53,162 @@ Documentation for general overview and summaries not connected with any specific
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
+
+# Architecture of documentation for Airflow
+
+Building documentation for Airflow is optimized for speed and for convenience workflows of the release
+managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
+multiple repositories and multiple sources of the documentation involved.
+
+There are few repositories under `apache` organization which are used to build the documentation for Airflow:
+
+* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
+ provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow)
+ from here we publish the documentation to S3 bucket where the documentation is hosted.
+* `airflow-site` - the repository with the website theme and content where we keep sources of the website
+ structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here
+ we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org)
+* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
+ of all the documentation packages that we keep on S3. This repository is automatically synchronized from
+ the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only
+ generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here.
+
+We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository:
+
+* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
+* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works
+
+# Diagrams of the documentation architecture
+
+This is the diagram od live documentation architecture:
+
+
+
+Staging documentation architecture is similar, but uses staging bucket and staging Apache Website.
+
+# Typical workflows
+
+There are a few typical workflows that we support:
+
+## Publishing the documentation by the release manager
+
+The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible
+via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation.
+
+The person who triggers the build (release manager) should specify the tag name of the docs to be published
+and the list of documentation packages to be published. Usually it is:
+
+* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`)
+* Helm chart: `helm-chart`
+* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.
+
+Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers,
+they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
+specified to be published without exclusions.
+
+You can also specify whether "live" or "staging" documentation should be published. The default is "live".
+
+Example screenshot of the workflow triggered from the GitHub UI:
+
+
+
+Right after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive`
+repository is automatically synchronized with the live S3 bucket. TO DO: IMPLEMENT THIS, FOR NOW IT HAS
+TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
+workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
+snapshots of the `live` documentation.
+
+Also another thing that is triggered automatically (TODO: IMPLEMENT THIS) is building the `airflow-site`
+and re-publishing it. This is needed in order to refresh version numbers of published documentation. After
+you publish new version of the documentation, the "stable" version of the documentation is automatically
+replaced with the new version, however building the `airflow-site` is needed to make sure that the drop-down
+version numbers are updated properly.
+
+
+## Publishing changes to the website (including theme)
+
+The workflows in `apache-airflow` only update the documentation for the documentation packages (Airflow, Helm chart,
+Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website
+itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.
+
+Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when
+the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered
+manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to
+`publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is
+published as the official website. This includes any changes to `.htaccess` of the website.
+
+Such a main build also publishes latest "sphinx-airflow-theme" package to GitHub so that the next build
+of documentation can automatically pick it up from there. This means that if you want to make changes to
+`javascript` or `css` that are part of the theme, you need to do it in `ariflow-site` repository and
+merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository
+and pick up the latest version of the theme.
+
+The version of sphinx theme is fixed in both repositories:
+
+* https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21
+* https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section
+
+In case of bigger changes to the theme, we
+can first iterate on the website and merge a new theme version, and only after that we can switch to the new
+version of the theme.
+
+
+# Fixing historical documentation
+
+Sometimes we need to update historical documentation (modify generated `html`) - for example when we find
+bad links or when we change some of the structure in the documentation. This can be done via the
+`airflow-site-archive` repository. The workflow is as follows:
+
+* Get the latest version of the documentation from S3 to `airflow-site-archive` repository using
+ `Sync s3 to GitHub` workflow. This will download the latest version of the documentation from S3 to
+ `airflow-site-archive` repository (this should be normally not needed, if automated synchronization works).
+
+* Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any
+ text editors, scripts etc. Those files are generated as `html` files and are not meant to be regenerated,
+ they should be modified as `html` files in-place
+
+* Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.
+
+* Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified
+ documentation to S3 bucket.
+
+* You can choose, whether to sync the changes to `live` or `staging` bucket. The default is `live`.
+
+* By default the workflow will synchronize all documentation modified in single - last commit pushed to
+ the branch you specified. You can also specify "full_sync" to synchronize all files in the repository.
+
+* In case you specify "full_sync", you can also synchronize `all` docs or only selected documentation
+ packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify
+ more than one package separated by spaces.
+
+* After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered
+ automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there
+ is no need to merge your changes to `main` branch of `airflow-site-archive` repository.
+
+
+
+
+## Manually publishing documentation
+
+The regular publishing workflows involve running Github Actions workflow and they cover majority of cases,
+however sometimes some manual updates and cherry-picks are needed, when we discover problems with the
+publishing and doc building code - for example when we find that we need to fix extensions to sphinx.
+
+In such case, release manager or a committer can build and publish documentation locally - providing that
+they configure AWS credentials to be able to upload files to S3.
+
+You can checkout locally a version of airflow repo that you need and apply any cherry-picks you need before
+running publishing.
+
+This is done using breeze. You also need to have aws CLI installed and configured credentials to be able
+to upload files to S3. You can get credentials from one of the admins of Airflow's AWS account. The
+region to set for AWS is `us-east-2`.
+
Review Comment:
yeah. I am also adding another manual method - without the need of S3 credentials - via archive repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #50464:
URL: https://github.com/apache/airflow/pull/50464#discussion_r2084937356
##########
dev/README_RELEASE_PROVIDERS.md:
##########
@@ -486,137 +485,6 @@ If you want to disable this behaviour, set the env **CLEAN_LOCAL_TAGS** to false
breeze release-management tag-providers
```
-## Prepare documentation
-
-Documentation is an essential part of the product and should be made available to users.
-In our cases, documentation for the released versions is published in a separate repository -
-[`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code
-and build tools are available in the `apache/airflow` repository, so you have to coordinate between
-the two repositories to be able to build the documentation.
-
-Documentation for providers can be found in the `/docs/apache-airflow-providers` directory
-and the `/docs/apache-airflow-providers-*/` directory. The first directory contains the package contents
-lists and should be updated every time a new version of Provider distributions is released.
-
-- First, copy the airflow-site repository and set the environment variable ``AIRFLOW_SITE_DIRECTORY``.
-
-```shell script
-git clone https://github.com/apache/airflow-site.git airflow-site
-cd airflow-site
-export AIRFLOW_SITE_DIRECTORY="$(pwd -P)"
-```
-
-Note if this is not the first time you clone the repo make sure main branch is rebased:
-
-```shell script
-cd "${AIRFLOW_SITE_DIRECTORY}"
-git checkout main
-git pull --rebase
-```
-
-- Then you can go to the directory and build the necessary documentation packages
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs --clean-build apache-airflow-providers all-providers --include-removed-providers --include-commits
-```
-
-Usually when we release packages we also build documentation for the "documentation-only" packages. This
-means that unless we release just few selected packages or if we need to deliberately skip some packages
-we should release documentation for all Provider distributions and the above command is the one to use.
-
-If we want to just release some providers you can release them using package names:
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs apache-airflow-providers cncf.kubernetes sftp --clean-build --include-commits
-```
-
-Alternatively, if you have set the environment variable: `DISTRIBUTIONS_LIST` above, just run the command:
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs --clean-build --include-commits
-```
-
-Or using `--distributions-list` argument:
-
-```shell script
-breeze build-docs --distributions-list PACKAGE1,PACKAGE2 --include-commits
-```
-
-- Now you can preview the documentation.
-
-```shell script
-./docs/start_doc_server.sh
-```
-
-If you encounter error like:
-
-```shell script
-airflow git:(main) ./docs/start_doc_server.sh
-./docs/start_doc_server.sh: line 22: cd: /Users/eladkal/PycharmProjects/airflow/docs/_build: No such file or directory
-```
-
-That probably means that the doc folder is empty thus it can not build the doc server.
-This indicates that previous step of building the docs did not work.
-
-- Copy the documentation to the ``airflow-site`` repository
-
-All providers (including overriding documentation for doc-only changes) - note that publishing is
-way faster on multi-cpu machines when you are publishing multiple providers:
-
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-
-breeze release-management publish-docs apache-airflow-providers all-providers --include-removed-providers \
- --override-versioned --run-in-parallel
-
-breeze release-management add-back-references all-providers
-```
-
-If you have providers as list of provider ids because you just released them you can build them with
-
-```shell script
-cd "${AIRFLOW_REPO_ROOT}"
-
-breeze release-management publish-docs amazon apache.beam google ....
-breeze release-management add-back-references all-providers
-```
-
-Alternatively, if you have set the environment variable: `DISTRIBUTIONS_LIST` above, just run the command:
-
-```shell script
-breeze release-management publish-docs
-breeze release-management add-back-references all-providers
-```
-
-Or using `--distributions-list` argument:
-
-```shell script
-breeze release-management publish-docs --distributions-list PACKAGE1,PACKAGE2
-breeze release-management add-back-references all-providers
-```
-
-
-Review the state of removed, suspended, new packages in
-[the docs index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md):
-
-- If you publish a new package, you must add it to the list of packages in the index.
-- If there are changes to suspension or removal status of a package you must move it appropriate section.
-
-- Create the commit and push changes.
-
-```shell script
-cd "${AIRFLOW_SITE_DIRECTORY}"
-branch="add-documentation-$(date "+%Y-%m-%d%n")"
-git checkout -b "${branch}"
-git add .
-git commit -m "Add documentation for packages - $(date "+%Y-%m-%d%n")"
-git push --set-upstream origin "${branch}"
-```
Review Comment:
We still need to do this if we add new provider or change his index place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how our documentation works [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50464:
URL: https://github.com/apache/airflow/pull/50464#issuecomment-2873096569
Updated latest comments @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org