You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "gopidesupavan (via GitHub)" <gi...@apache.org> on 2025/05/08 21:48:52 UTC
[PR] Add live site s3 docs bucket [airflow]
gopidesupavan opened a new pull request, #50375:
URL: https://github.com/apache/airflow/pull/50375
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2870250102
I think it was a bit "abstract" description so far but with https://github.com/apache/airflow/pull/50464 which describes how things work and show screenshots and workflows - it might be a little more digestible (albeit it's a longer read).
@eladkal @kaxil @ephraimbuddy -> the #50464 is for you to take a look and see if you are fine with the processes and workflows.
Also @eladkal -> I do not think it's worth to cover the case that happened this time (when we had to build docs for Amazon from main instead of tag) is worth automating. We should aim to be able to release provider docs directly from the tag, and in case we need to apply some extra fixes after the tag is already created, we always have manual publishing option - there are two ways described in #50464 and rather than trying to automate those cases, i'd keep the manual ways as "escape-hatch". We have enough ways to override the "GitHub Actions" regular process - and then it could even be done with help of others. And via `apache-airflow-site-archive` it does not even need S3 credentials - just building docs locally and pushing them to a branch in `apache-airflow-site-archive` and then pushing changes to S3 via GH -> S3 workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2866088397
Also there is another option - we could **always** publish latest main to staging when main succeeds. But that will go fine for "providers" but not too fine for airflow (we want to push airflow docs from v-* branch). And we already have the "main" docs published using the s3 main bucket which we use to pull inventories from, and I think what is at staging should be a deliberate act of release manager rather than what is "latest".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2873261000
> Two separate workflow (PROD/STAGING push?) is fine. So we can use it for testing things to when needed
One thing about single workflow that I thought about is that it could automatically push to staging when RC tag is used and that saves one decision to make to RM. We could parse the tag and when pre-release is detected, we would publish to staging instead of live.
That would also nicely correspond to building packages - because in pre-release builds in metadata we could point to that staging site rather than production one. Currently when you publish RC in PyPI - the documentation link points to non-existing RC version (and even some people reported that as a bug) but we could change it so that links are pointing to final version but the stagingon. and whenever you use RC/ALPHA/BETA tag, we could automatically push to staging.
And we could have a drop-down:
* auto (defaullt)
* live
* staging
That would "close" the loop on our RC packages preparation.
1. prepare RC packages -> staging link in metadata
2. publish docs using RC tag -> push to staging s3 while voting on RC
.... voting .....
4. final packages -> live link in metafataa
5. publish docs using final tag -> push to live s3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2871558675
Two separate workflow (PROD/STAGING push?) is fine. So we can use it for testing things to when needed
> What would be the preference for workflows of publishing airlfow/provider docs ?
>
> * two separate workflow (PROD/STAGING push?)
> * one workflow with checkbox? (what should be the default)
> * or maybe use the TAG and take away the freedom of choosing staging or not
>
> If we get answer to those questions - we can proceed with implementing staging the chosen way :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan merged PR #50375:
URL: https://github.com/apache/airflow/pull/50375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2865886776
> When would we use the stage option?
Yeah, we can use it for testing docs and @potiuk mentioned in dev channel we could publish RC provider docs to staging location and verify , thats a good idea :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2866074784
Yeah. I think we can work out the workflows for this step-by-step with @kaxil and @ephraimbuddy and others who would be the release managers.
Right after we improve and finalize the docs for regular workflows, we can work out how to do the "staging" ones. For now Kaxil for example publishes Airflow docs with surge.sh - which is fine but it would be better if we can do the same publishing for RC as for the regular releases.
For example when we are relasing the images we have the same workflow -> just "Release -> type version-> workflow will appropriately build the images with or without -`-pre` option and automatically using the rc versions of the providers (and handles all the complex cases) and with or without updating the "katest" image link (for example when we release an RC image we never update "latest" aliases in DockerHub.
Here, ti's a bit different story, because we have to first work out how we are publishing the staging site as well. Not only the documentation for airflow and providers.
And we do not have it automated yet, but we can - and we planned to have a dedicated branch "staging" that you can push in "airflow-site" and then the site will automatically be published at "staging.airflow.apache.org" - that will allow us to iterate on "site" itself. Then I imagine when you are releasing RC candidates, you could choose "staging" from the list (that might be even a bit simpler than now - just additional "staging" checkbox that might even be selected by default) or have a separate "Staging Publishing' workflow - and then the RC docs would be published at "staging.airflow.apache.org" as well. Also we could always select staging by default when tag contains pre-release version (rc/alpha etc.) and do not give freedom to the RM to select staging.
One thing to note there is a question of RC links in pypi - currently the links are nicely redirecting [Documentation](https://airflow.apache.org/docs/apache-airflow-providers-common-messaging/1.0.1rc1) for example - and ... it nicely redirects to 1.0.1 (I thiink it 's a side effect of the index.html fixing that @gopidesupavan implemented.
Probably we could do the same with future links that are for versions we have not released yet - but we could redirect them to staging - so for example: https://airflow.apache.org/docs/apache-airflow-providers-common-messaging/1.0.2rc1/ (when 1.0.2 is not yet released and docs not published) would redirect to https://staging.airflow.apache.org/docs/apache-airflow-providers-common-messaging/1.0.2/ -> and use the documentation from there.
In this case the workflows would look like this:
1) When you want to check a new "site" modification -> push to "staging" branch in "airflow-site" and it will be visible under "staging.airflow.apache.org". Later merging it to "main" branch will publish them to "airflow.apache.org" - so far, so good.
2) When you want to make a release candidate for providers, airlfow , docker-stack, all-providers -> you run the same workflow as for final publishing (choose rc* tag to use) -> they will also land at staging.airlfow.apache.org, links in PyPi packages will point to "apache.airflow.org/ .... x.y.zrc2" - but since documentation will not be published in "airflow.apache.org" - that documentation will get riderected to "staging.airflow.apache.org/.....x.y.z/" which will be already published by release manager.
3) once we publish the final release, the RM will re-run the docs push (but will select "no staging" option) and the docs will land in "airflow.apache.org" -> and the links from RC candidates in PyPI will no longer redirect to staging, they will point to the final release (of airflow or providers).
Does it sound like a good plan - @eladkal @kaxil @ephraimbuddy ?
What would be the preference for workflows of publishing airlfow/provider docs ?
* two separate workflow (PROD/STAGING push?)
* one workflow with checkbox? (what should be the default)
* or maybe use the TAG and take away the freedom of choosing staging or not
If we get answer to those questions - we can proceed with implementing staging the chosen way :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add live site s3 docs bucket [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #50375:
URL: https://github.com/apache/airflow/pull/50375#issuecomment-2869893741
Thanks Jarek for such a details process, this approach is fine.
But will leave it release managers, @eladkal @kaxil @ephraimbuddy WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org