You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2023/02/23 13:10:03 UTC
[URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Hello everyone,
We just ran out of space for the Airflow documentation build and our
documentation stopped being published - with the last provider's
releases.
The process of building requires all versions of Airflow to be present
when publishing new documentation.
As a stop-gap measure - I have prepared pr with removed docs for
<1.10.15 https://github.com/apache/airflow-site/pull/740. The old docs
are kept in git history and we can revert this change back asap. I
think not having information about < 1.10.14 is much better than not
having information about the latest providers (and airflow when we
release it soon).
Unless someone has a better idea (and PR), I see that as the fastest
way to temporarily fix the problem and I am going to merge it soon
unless we have some objection and counter-proposal.
However this is going to come back soon. I believe Ash mentioned that
someone from Astronomer **might** be able to help with rewriting the
docs build process so I think this is about the last moment to do so,
because if we continue, releasing new versions of both providers and
airflow without any change, the problem is going to get worse.
Next stop-gap measure is to remove docs for old providers' versions.
J,
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool :). Teamwork :D
On Fri, Feb 24, 2023 at 12:13 AM Jed Cunningham <je...@apache.org>
wrote:
> Here it is: https://github.com/apache/airflow-site/pull/742
>
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Jed Cunningham <je...@apache.org>.
Here it is: https://github.com/apache/airflow-site/pull/742
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Kaxil Naik <ka...@gmail.com>.
Cool stuff, now that it this one is merged, letβs revert the morning merge.
Thanks Jed.
On Thu, 23 Feb 2023 at 23:01, Jed Cunningham <je...@apache.org>
wrote:
> I came up with a stopgap that'll allow us to retain those old versions
> until we can work out the S3 archive. GitHub runners come with some stuff
> we don't actually use, so a little pre-cleanup and we've gained roughly
> 10Gb of extra disk π
>
> https://github.com/apache/airflow-site/pull/741
>
> If that looks good and we merge it, we can revert this morning's revert.
>
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Jed Cunningham <je...@apache.org>.
I came up with a stopgap that'll allow us to retain those old versions
until we can work out the S3 archive. GitHub runners come with some stuff
we don't actually use, so a little pre-cleanup and we've gained roughly
10Gb of extra disk π
https://github.com/apache/airflow-site/pull/741
If that looks good and we merge it, we can revert this morning's revert.
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Kaxil Naik <ka...@gmail.com>.
Yeah let's redirect them to s3 hosted site. We can just update .htacess
<https://github.com/apache/airflow-site/blob/gh-pages/.htaccess> file with
the redirect rules.
On Thu, 23 Feb 2023 at 19:48, Jarek Potiuk <ja...@potiuk.com> wrote:
> BTW. I thought about it and there are some approaches that might be done
> "quickly" - even without changing the build process. We could have a
> script to manually move the generated HTML code for some old versions of
> airflow/providers to a static HTML archive (s3 ?) and figure out how to
> redirect the old links there automatically (and add "archived version" link
> somewhere in the drop-down of versions. This would require mostly figuring
> out how to do it manually first, and then automating it eventually. Then we
> could even rewrite the whole "airflow-site" Git repo to make it way smaller
> after we clean it up.
>
> J.
>
> On Thu, Feb 23, 2023 at 8:21 PM Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
>
>> Thanks Jarek, for resolving this quickly.
>>
>> I agree that it is important to figure out a plan and next steps here
>> before the next release for sure.
>> Will sync and get back.
>>
>> Vikram
>>
>>
>> On Thu, Feb 23, 2023 at 6:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> The temporary fix worked - for now. The < 1.10.15 docs are no more on
>>> our pages:
>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html.
>>> The new provider docs are published.
>>>
>>> Likely in a few weeks the problem will happen again, and unfortunately
>>> - other than removing some other versions of some other docs - I
>>> believe we have no contingency plan and we might be left with
>>> inability to publish new docs, unless we have someone seriously
>>> looking and making the docs building process sustainable.
>>>
>>> J.
>>>
>>> On Thu, Feb 23, 2023 at 2:19 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >
>>> > I meant < 1.10.15 everywhere BTW
>>> >
>>> > On Thu, Feb 23, 2023 at 2:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> > >
>>> > > Hello everyone,
>>> > >
>>> > > We just ran out of space for the Airflow documentation build and our
>>> > > documentation stopped being published - with the last provider's
>>> > > releases.
>>> > >
>>> > > The process of building requires all versions of Airflow to be
>>> present
>>> > > when publishing new documentation.
>>> > >
>>> > > As a stop-gap measure - I have prepared pr with removed docs for
>>> > > <1.10.15 https://github.com/apache/airflow-site/pull/740. The old
>>> docs
>>> > > are kept in git history and we can revert this change back asap. I
>>> > > think not having information about < 1.10.14 is much better than not
>>> > > having information about the latest providers (and airflow when we
>>> > > release it soon).
>>> > >
>>> > > Unless someone has a better idea (and PR), I see that as the fastest
>>> > > way to temporarily fix the problem and I am going to merge it soon
>>> > > unless we have some objection and counter-proposal.
>>> > >
>>> > > However this is going to come back soon. I believe Ash mentioned that
>>> > > someone from Astronomer **might** be able to help with rewriting the
>>> > > docs build process so I think this is about the last moment to do so,
>>> > > because if we continue, releasing new versions of both providers and
>>> > > airflow without any change, the problem is going to get worse.
>>> > >
>>> > > Next stop-gap measure is to remove docs for old providers' versions.
>>> > >
>>> > > J,
>>>
>>
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Jarek Potiuk <ja...@potiuk.com>.
BTW. I thought about it and there are some approaches that might be done
"quickly" - even without changing the build process. We could have a
script to manually move the generated HTML code for some old versions of
airflow/providers to a static HTML archive (s3 ?) and figure out how to
redirect the old links there automatically (and add "archived version" link
somewhere in the drop-down of versions. This would require mostly figuring
out how to do it manually first, and then automating it eventually. Then we
could even rewrite the whole "airflow-site" Git repo to make it way smaller
after we clean it up.
J.
On Thu, Feb 23, 2023 at 8:21 PM Vikram Koka <vi...@astronomer.io.invalid>
wrote:
> Thanks Jarek, for resolving this quickly.
>
> I agree that it is important to figure out a plan and next steps here
> before the next release for sure.
> Will sync and get back.
>
> Vikram
>
>
> On Thu, Feb 23, 2023 at 6:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> The temporary fix worked - for now. The < 1.10.15 docs are no more on
>> our pages:
>> https://airflow.apache.org/docs/apache-airflow/stable/index.html.
>> The new provider docs are published.
>>
>> Likely in a few weeks the problem will happen again, and unfortunately
>> - other than removing some other versions of some other docs - I
>> believe we have no contingency plan and we might be left with
>> inability to publish new docs, unless we have someone seriously
>> looking and making the docs building process sustainable.
>>
>> J.
>>
>> On Thu, Feb 23, 2023 at 2:19 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > I meant < 1.10.15 everywhere BTW
>> >
>> > On Thu, Feb 23, 2023 at 2:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> > >
>> > > Hello everyone,
>> > >
>> > > We just ran out of space for the Airflow documentation build and our
>> > > documentation stopped being published - with the last provider's
>> > > releases.
>> > >
>> > > The process of building requires all versions of Airflow to be present
>> > > when publishing new documentation.
>> > >
>> > > As a stop-gap measure - I have prepared pr with removed docs for
>> > > <1.10.15 https://github.com/apache/airflow-site/pull/740. The old
>> docs
>> > > are kept in git history and we can revert this change back asap. I
>> > > think not having information about < 1.10.14 is much better than not
>> > > having information about the latest providers (and airflow when we
>> > > release it soon).
>> > >
>> > > Unless someone has a better idea (and PR), I see that as the fastest
>> > > way to temporarily fix the problem and I am going to merge it soon
>> > > unless we have some objection and counter-proposal.
>> > >
>> > > However this is going to come back soon. I believe Ash mentioned that
>> > > someone from Astronomer **might** be able to help with rewriting the
>> > > docs build process so I think this is about the last moment to do so,
>> > > because if we continue, releasing new versions of both providers and
>> > > airflow without any change, the problem is going to get worse.
>> > >
>> > > Next stop-gap measure is to remove docs for old providers' versions.
>> > >
>> > > J,
>>
>
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Thanks Jarek, for resolving this quickly.
I agree that it is important to figure out a plan and next steps here
before the next release for sure.
Will sync and get back.
Vikram
On Thu, Feb 23, 2023 at 6:28 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> The temporary fix worked - for now. The < 1.10.15 docs are no more on
> our pages:
> https://airflow.apache.org/docs/apache-airflow/stable/index.html.
> The new provider docs are published.
>
> Likely in a few weeks the problem will happen again, and unfortunately
> - other than removing some other versions of some other docs - I
> believe we have no contingency plan and we might be left with
> inability to publish new docs, unless we have someone seriously
> looking and making the docs building process sustainable.
>
> J.
>
> On Thu, Feb 23, 2023 at 2:19 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > I meant < 1.10.15 everywhere BTW
> >
> > On Thu, Feb 23, 2023 at 2:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > Hello everyone,
> > >
> > > We just ran out of space for the Airflow documentation build and our
> > > documentation stopped being published - with the last provider's
> > > releases.
> > >
> > > The process of building requires all versions of Airflow to be present
> > > when publishing new documentation.
> > >
> > > As a stop-gap measure - I have prepared pr with removed docs for
> > > <1.10.15 https://github.com/apache/airflow-site/pull/740. The old docs
> > > are kept in git history and we can revert this change back asap. I
> > > think not having information about < 1.10.14 is much better than not
> > > having information about the latest providers (and airflow when we
> > > release it soon).
> > >
> > > Unless someone has a better idea (and PR), I see that as the fastest
> > > way to temporarily fix the problem and I am going to merge it soon
> > > unless we have some objection and counter-proposal.
> > >
> > > However this is going to come back soon. I believe Ash mentioned that
> > > someone from Astronomer **might** be able to help with rewriting the
> > > docs build process so I think this is about the last moment to do so,
> > > because if we continue, releasing new versions of both providers and
> > > airflow without any change, the problem is going to get worse.
> > >
> > > Next stop-gap measure is to remove docs for old providers' versions.
> > >
> > > J,
>
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Jarek Potiuk <ja...@potiuk.com>.
The temporary fix worked - for now. The < 1.10.15 docs are no more on
our pages: https://airflow.apache.org/docs/apache-airflow/stable/index.html.
The new provider docs are published.
Likely in a few weeks the problem will happen again, and unfortunately
- other than removing some other versions of some other docs - I
believe we have no contingency plan and we might be left with
inability to publish new docs, unless we have someone seriously
looking and making the docs building process sustainable.
J.
On Thu, Feb 23, 2023 at 2:19 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> I meant < 1.10.15 everywhere BTW
>
> On Thu, Feb 23, 2023 at 2:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Hello everyone,
> >
> > We just ran out of space for the Airflow documentation build and our
> > documentation stopped being published - with the last provider's
> > releases.
> >
> > The process of building requires all versions of Airflow to be present
> > when publishing new documentation.
> >
> > As a stop-gap measure - I have prepared pr with removed docs for
> > <1.10.15 https://github.com/apache/airflow-site/pull/740. The old docs
> > are kept in git history and we can revert this change back asap. I
> > think not having information about < 1.10.14 is much better than not
> > having information about the latest providers (and airflow when we
> > release it soon).
> >
> > Unless someone has a better idea (and PR), I see that as the fastest
> > way to temporarily fix the problem and I am going to merge it soon
> > unless we have some objection and counter-proposal.
> >
> > However this is going to come back soon. I believe Ash mentioned that
> > someone from Astronomer **might** be able to help with rewriting the
> > docs build process so I think this is about the last moment to do so,
> > because if we continue, releasing new versions of both providers and
> > airflow without any change, the problem is going to get worse.
> >
> > Next stop-gap measure is to remove docs for old providers' versions.
> >
> > J,
Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds
Posted by Jarek Potiuk <ja...@potiuk.com>.
I meant < 1.10.15 everywhere BTW
On Thu, Feb 23, 2023 at 2:10 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Hello everyone,
>
> We just ran out of space for the Airflow documentation build and our
> documentation stopped being published - with the last provider's
> releases.
>
> The process of building requires all versions of Airflow to be present
> when publishing new documentation.
>
> As a stop-gap measure - I have prepared pr with removed docs for
> <1.10.15 https://github.com/apache/airflow-site/pull/740. The old docs
> are kept in git history and we can revert this change back asap. I
> think not having information about < 1.10.14 is much better than not
> having information about the latest providers (and airflow when we
> release it soon).
>
> Unless someone has a better idea (and PR), I see that as the fastest
> way to temporarily fix the problem and I am going to merge it soon
> unless we have some objection and counter-proposal.
>
> However this is going to come back soon. I believe Ash mentioned that
> someone from Astronomer **might** be able to help with rewriting the
> docs build process so I think this is about the last moment to do so,
> because if we continue, releasing new versions of both providers and
> airflow without any change, the problem is going to get worse.
>
> Next stop-gap measure is to remove docs for old providers' versions.
>
> J,