You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@gmail.com> on 2026/10/06 22:27:42 UTC
[CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Hi all,
More and more people write Dags with coding agents (Claude Code, Cursor,
Copilot, Codex), and those agents read our docs as HTML. A provider guide
page is about 150 KB of HTML for about 20 KB of content, and fetch tools
often summarise it down to a paragraph, code samples included. Pydantic AI,
Ray, Camel, Astronomer and most of the popular sites & project publish an
llms.txt index plus a Markdown copy of each page. We publish neither.
I've opened an issue with a plan and a working proof of concept, starting
with the Common AI provider docs:
https://github.com/apache/airflow/issues/74376
It has three parts:
1. Docs build: generate page.html.md next to every page, plus llms.txt and
llms-full.txt, using sphinx-llm and a small extension. The proof of concept
builds cleanly for Common AI with no change in build time.
2. Publishing: a couple of guards in breeze so stable/ always has the
Markdown copies.
3. Serving: Claude Code, Cursor, Copilot and OpenCode already send "Accept:
text/markdown" on every fetch. A short .htaccess rule in airflow-site can
answer those requests with the Markdown copy, so agents get it from the
URLs they already use. I tested this on airflow.staged.apache.org; ASF's
Fastly cache keeps the HTML and Markdown answers apart.
*I won't have time to build this or to review it, so it needs an owner who
can see it through, including reviews.* The issue lists what's done, what's
left, and the rollout constraints. It needs a committer, or a contributor
pairing with one: testing the airflow-site change on staging, publishing
docs and checking the live rollout all need committer access.
This would be a high impact change, so thanks in advance for whoever picks
it up.
Regards,
Kaxil
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Kaxil Naik <ka...@gmail.com>.
Awesome, sounds good
On Wed, 7 Oct 2026 at 00:03, Sameer Mesiah <sm...@gmail.com> wrote:
> I understand. I don't believe I should have much issue getting a committer
> to help me with all of that. In case I am not able to get a committer to
> assist me, I will inform you and we can hand it to someone who is a
> committer or has that assistance.
>
> Thanks,
> Sameer.
>
> On Tue, 6 Oct 2026 at 23:53, Kaxil Naik <ka...@gmail.com> wrote:
>
> > Absolutely happy for you to take it as long as you can pair with a
> commiter
> > because of the testing requirements I mentioned in the previous email and
> > the GitHub issue.
> >
> > *It needs a committer, or a contributor pairing with one: testing the
> > airflow-site change on staging, publishing docs and checking the live
> > rollout all need committer access.*
> >
> > On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
> >
> > > Hi Kaxil,
> > >
> > > I can pick this up if you are okay with it.
> > >
> > > On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > More and more people write Dags with coding agents (Claude Code,
> > Cursor,
> > > > Copilot, Codex), and those agents read our docs as HTML. A provider
> > guide
> > > > page is about 150 KB of HTML for about 20 KB of content, and fetch
> > tools
> > > > often summarise it down to a paragraph, code samples included.
> Pydantic
> > > AI,
> > > > Ray, Camel, Astronomer and most of the popular sites & project
> publish
> > an
> > > > llms.txt index plus a Markdown copy of each page. We publish neither.
> > > >
> > > > I've opened an issue with a plan and a working proof of concept,
> > starting
> > > > with the Common AI provider docs:
> > > >
> > > > https://github.com/apache/airflow/issues/74376
> > > >
> > > > It has three parts:
> > > >
> > > > 1. Docs build: generate page.html.md next to every page, plus
> llms.txt
> > > and
> > > > llms-full.txt, using sphinx-llm and a small extension. The proof of
> > > concept
> > > > builds cleanly for Common AI with no change in build time.
> > > > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > > > Markdown copies.
> > > > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> > > "Accept:
> > > > text/markdown" on every fetch. A short .htaccess rule in airflow-site
> > can
> > > > answer those requests with the Markdown copy, so agents get it from
> the
> > > > URLs they already use. I tested this on airflow.staged.apache.org;
> > ASF's
> > > > Fastly cache keeps the HTML and Markdown answers apart.
> > > >
> > > > *I won't have time to build this or to review it, so it needs an
> owner
> > > who
> > > > can see it through, including reviews.* The issue lists what's done,
> > > what's
> > > > left, and the rollout constraints. It needs a committer, or a
> > contributor
> > > > pairing with one: testing the airflow-site change on staging,
> > publishing
> > > > docs and checking the live rollout all need committer access.
> > > >
> > > > This would be a high impact change, so thanks in advance for whoever
> > > picks
> > > > it up.
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > >
> >
>
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by IONOS Support via dev <de...@airflow.apache.org>.
Dear Sender,
Thank you for contacting us. As we couldn't identify a specific request from your inquiry, please provide additional details so that we may assist you. Our team is here to help and can also be contacted by phone at 1-484-424-7392.
If you’re a customer, you also have access to additional support resources, including:
* Resend your request using the email address associated with your customer account for personalised support.
* Chat with our AI Assistant https://login.ionos.com directly in your IONOS account for instant answers.
* Log in https://my.ionos.com to your IONOS account to view available support options (Help & contact).
* For updating or verifying your contact information, please click here for instructions https://www.ionos.com/help/index.php?id=2574 .
If you have any complaints regarding content on websites hosted by IONOS, please find instructions and contact information here https://registrar.ionos.com/domains_raa/complaints?market=en_US .
Thank you again for reaching out. We're committed to providing you with the best possible support and look forward to helping you further.
Best regards,
IONOS Team
This email was created with the support of artificial intelligence (AI).
Contact: https://contact.ionos.com
Facebook: https://www.facebook.com/my.ionos
X/Twitter: https://twitter.com/IONOS_help_US
E-Mail: support@ionos.com
IONOS Inc.
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Shahar Epstein <sh...@apache.org>.
Thanks Sameer and David!
Please involve me for PMC support if needed.
Shahar
On Wed, Oct 7, 2026, 11:03 Blain David <da...@infrabel.be> wrote:
> Always willing to help with the review.
>
> ________________________________
> From: Sameer Mesiah <sm...@gmail.com>
> Sent: Wednesday, October 07, 2026 01:01
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for
> coding agents for Airflow site
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> abuse@infrabel.be>.
>
> I understand. I don't believe I should have much issue getting a committer
> to help me with all of that. In case I am not able to get a committer to
> assist me, I will inform you and we can hand it to someone who is a
> committer or has that assistance.
>
> Thanks,
> Sameer.
>
> On Tue, 6 Oct 2026 at 23:53, Kaxil Naik <ka...@gmail.com> wrote:
>
> > Absolutely happy for you to take it as long as you can pair with a
> commiter
> > because of the testing requirements I mentioned in the previous email and
> > the GitHub issue.
> >
> > *It needs a committer, or a contributor pairing with one: testing the
> > airflow-site change on staging, publishing docs and checking the live
> > rollout all need committer access.*
> >
> > On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
> >
> > > Hi Kaxil,
> > >
> > > I can pick this up if you are okay with it.
> > >
> > > On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > More and more people write Dags with coding agents (Claude Code,
> > Cursor,
> > > > Copilot, Codex), and those agents read our docs as HTML. A provider
> > guide
> > > > page is about 150 KB of HTML for about 20 KB of content, and fetch
> > tools
> > > > often summarise it down to a paragraph, code samples included.
> Pydantic
> > > AI,
> > > > Ray, Camel, Astronomer and most of the popular sites & project
> publish
> > an
> > > > llms.txt index plus a Markdown copy of each page. We publish neither.
> > > >
> > > > I've opened an issue with a plan and a working proof of concept,
> > starting
> > > > with the Common AI provider docs:
> > > >
> > > >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fissues%2F74376&data=05%7C02%7Cdavid.blain%40infrabel.be%7C8b25c7e9d32b4dfa939808df23fdc912%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639269245055449115%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rHwv1AaT41G789YkOXh7cINsukGuOc7fUn0%2FItMXHyo%3D&reserved=0
> <https://github.com/apache/airflow/issues/74376>
> > > >
> > > > It has three parts:
> > > >
> > > > 1. Docs build: generate page.html.md next to every page, plus
> llms.txt
> > > and
> > > > llms-full.txt, using sphinx-llm and a small extension. The proof of
> > > concept
> > > > builds cleanly for Common AI with no change in build time.
> > > > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > > > Markdown copies.
> > > > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> > > "Accept:
> > > > text/markdown" on every fetch. A short .htaccess rule in airflow-site
> > can
> > > > answer those requests with the Markdown copy, so agents get it from
> the
> > > > URLs they already use. I tested this on airflow.staged.apache.org;
> > ASF's
> > > > Fastly cache keeps the HTML and Markdown answers apart.
> > > >
> > > > *I won't have time to build this or to review it, so it needs an
> owner
> > > who
> > > > can see it through, including reviews.* The issue lists what's done,
> > > what's
> > > > left, and the rollout constraints. It needs a committer, or a
> > contributor
> > > > pairing with one: testing the airflow-site change on staging,
> > publishing
> > > > docs and checking the live rollout all need committer access.
> > > >
> > > > This would be a high impact change, so thanks in advance for whoever
> > > picks
> > > > it up.
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > >
> >
>
> General (Internal Property)
>
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Blain David <da...@infrabel.be>.
Always willing to help with the review.
________________________________
From: Sameer Mesiah <sm...@gmail.com>
Sent: Wednesday, October 07, 2026 01:01
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
I understand. I don't believe I should have much issue getting a committer
to help me with all of that. In case I am not able to get a committer to
assist me, I will inform you and we can hand it to someone who is a
committer or has that assistance.
Thanks,
Sameer.
On Tue, 6 Oct 2026 at 23:53, Kaxil Naik <ka...@gmail.com> wrote:
> Absolutely happy for you to take it as long as you can pair with a commiter
> because of the testing requirements I mentioned in the previous email and
> the GitHub issue.
>
> *It needs a committer, or a contributor pairing with one: testing the
> airflow-site change on staging, publishing docs and checking the live
> rollout all need committer access.*
>
> On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
>
> > Hi Kaxil,
> >
> > I can pick this up if you are okay with it.
> >
> > On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > More and more people write Dags with coding agents (Claude Code,
> Cursor,
> > > Copilot, Codex), and those agents read our docs as HTML. A provider
> guide
> > > page is about 150 KB of HTML for about 20 KB of content, and fetch
> tools
> > > often summarise it down to a paragraph, code samples included. Pydantic
> > AI,
> > > Ray, Camel, Astronomer and most of the popular sites & project publish
> an
> > > llms.txt index plus a Markdown copy of each page. We publish neither.
> > >
> > > I've opened an issue with a plan and a working proof of concept,
> starting
> > > with the Common AI provider docs:
> > >
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fissues%2F74376&data=05%7C02%7Cdavid.blain%40infrabel.be%7C8b25c7e9d32b4dfa939808df23fdc912%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639269245055449115%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rHwv1AaT41G789YkOXh7cINsukGuOc7fUn0%2FItMXHyo%3D&reserved=0<https://github.com/apache/airflow/issues/74376>
> > >
> > > It has three parts:
> > >
> > > 1. Docs build: generate page.html.md next to every page, plus llms.txt
> > and
> > > llms-full.txt, using sphinx-llm and a small extension. The proof of
> > concept
> > > builds cleanly for Common AI with no change in build time.
> > > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > > Markdown copies.
> > > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> > "Accept:
> > > text/markdown" on every fetch. A short .htaccess rule in airflow-site
> can
> > > answer those requests with the Markdown copy, so agents get it from the
> > > URLs they already use. I tested this on airflow.staged.apache.org;
> ASF's
> > > Fastly cache keeps the HTML and Markdown answers apart.
> > >
> > > *I won't have time to build this or to review it, so it needs an owner
> > who
> > > can see it through, including reviews.* The issue lists what's done,
> > what's
> > > left, and the rollout constraints. It needs a committer, or a
> contributor
> > > pairing with one: testing the airflow-site change on staging,
> publishing
> > > docs and checking the live rollout all need committer access.
> > >
> > > This would be a high impact change, so thanks in advance for whoever
> > picks
> > > it up.
> > >
> > > Regards,
> > > Kaxil
> > >
> >
>
General (Internal Property)
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Jarek Potiuk <ja...@potiuk.com>.
Nice!
On Wed, Oct 7, 2026 at 8:36 AM Blain David <da...@infrabel.be> wrote:
>
> Good initiative, this is getting more important these days for agents, so sites should indeed adapt to those as well.
>
> Funny think is I did exactly the same with our internal airflow documentation site so our agents can easily read it as well.
>
> General (Internal Property)
> ________________________________
> From: Sameer Mesiah <sm...@gmail.com>
> Sent: Wednesday, October 7, 2026 01:01
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
>
> I understand. I don't believe I should have much issue getting a committer
> to help me with all of that. In case I am not able to get a committer to
> assist me, I will inform you and we can hand it to someone who is a
> committer or has that assistance.
>
> Thanks,
> Sameer.
>
> On Tue, 6 Oct 2026 at 23:53, Kaxil Naik <ka...@gmail.com> wrote:
>
> > Absolutely happy for you to take it as long as you can pair with a commiter
> > because of the testing requirements I mentioned in the previous email and
> > the GitHub issue.
> >
> > *It needs a committer, or a contributor pairing with one: testing the
> > airflow-site change on staging, publishing docs and checking the live
> > rollout all need committer access.*
> >
> > On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
> >
> > > Hi Kaxil,
> > >
> > > I can pick this up if you are okay with it.
> > >
> > > On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > More and more people write Dags with coding agents (Claude Code,
> > Cursor,
> > > > Copilot, Codex), and those agents read our docs as HTML. A provider
> > guide
> > > > page is about 150 KB of HTML for about 20 KB of content, and fetch
> > tools
> > > > often summarise it down to a paragraph, code samples included. Pydantic
> > > AI,
> > > > Ray, Camel, Astronomer and most of the popular sites & project publish
> > an
> > > > llms.txt index plus a Markdown copy of each page. We publish neither.
> > > >
> > > > I've opened an issue with a plan and a working proof of concept,
> > starting
> > > > with the Common AI provider docs:
> > > >
> > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fissues%2F74376&data=05%7C02%7Cdavid.blain%40infrabel.be%7C8b25c7e9d32b4dfa939808df23fdc912%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639269245055449115%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rHwv1AaT41G789YkOXh7cINsukGuOc7fUn0%2FItMXHyo%3D&reserved=0<https://github.com/apache/airflow/issues/74376>
> > > >
> > > > It has three parts:
> > > >
> > > > 1. Docs build: generate page.html.md next to every page, plus llms.txt
> > > and
> > > > llms-full.txt, using sphinx-llm and a small extension. The proof of
> > > concept
> > > > builds cleanly for Common AI with no change in build time.
> > > > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > > > Markdown copies.
> > > > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> > > "Accept:
> > > > text/markdown" on every fetch. A short .htaccess rule in airflow-site
> > can
> > > > answer those requests with the Markdown copy, so agents get it from the
> > > > URLs they already use. I tested this on airflow.staged.apache.org;
> > ASF's
> > > > Fastly cache keeps the HTML and Markdown answers apart.
> > > >
> > > > *I won't have time to build this or to review it, so it needs an owner
> > > who
> > > > can see it through, including reviews.* The issue lists what's done,
> > > what's
> > > > left, and the rollout constraints. It needs a committer, or a
> > contributor
> > > > pairing with one: testing the airflow-site change on staging,
> > publishing
> > > > docs and checking the live rollout all need committer access.
> > > >
> > > > This would be a high impact change, so thanks in advance for whoever
> > > picks
> > > > it up.
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Blain David <da...@infrabel.be>.
Good initiative, this is getting more important these days for agents, so sites should indeed adapt to those as well.
Funny think is I did exactly the same with our internal airflow documentation site so our agents can easily read it as well.
General (Internal Property)
________________________________
From: Sameer Mesiah <sm...@gmail.com>
Sent: Wednesday, October 7, 2026 01:01
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
I understand. I don't believe I should have much issue getting a committer
to help me with all of that. In case I am not able to get a committer to
assist me, I will inform you and we can hand it to someone who is a
committer or has that assistance.
Thanks,
Sameer.
On Tue, 6 Oct 2026 at 23:53, Kaxil Naik <ka...@gmail.com> wrote:
> Absolutely happy for you to take it as long as you can pair with a commiter
> because of the testing requirements I mentioned in the previous email and
> the GitHub issue.
>
> *It needs a committer, or a contributor pairing with one: testing the
> airflow-site change on staging, publishing docs and checking the live
> rollout all need committer access.*
>
> On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
>
> > Hi Kaxil,
> >
> > I can pick this up if you are okay with it.
> >
> > On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > More and more people write Dags with coding agents (Claude Code,
> Cursor,
> > > Copilot, Codex), and those agents read our docs as HTML. A provider
> guide
> > > page is about 150 KB of HTML for about 20 KB of content, and fetch
> tools
> > > often summarise it down to a paragraph, code samples included. Pydantic
> > AI,
> > > Ray, Camel, Astronomer and most of the popular sites & project publish
> an
> > > llms.txt index plus a Markdown copy of each page. We publish neither.
> > >
> > > I've opened an issue with a plan and a working proof of concept,
> starting
> > > with the Common AI provider docs:
> > >
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fissues%2F74376&data=05%7C02%7Cdavid.blain%40infrabel.be%7C8b25c7e9d32b4dfa939808df23fdc912%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639269245055449115%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rHwv1AaT41G789YkOXh7cINsukGuOc7fUn0%2FItMXHyo%3D&reserved=0<https://github.com/apache/airflow/issues/74376>
> > >
> > > It has three parts:
> > >
> > > 1. Docs build: generate page.html.md next to every page, plus llms.txt
> > and
> > > llms-full.txt, using sphinx-llm and a small extension. The proof of
> > concept
> > > builds cleanly for Common AI with no change in build time.
> > > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > > Markdown copies.
> > > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> > "Accept:
> > > text/markdown" on every fetch. A short .htaccess rule in airflow-site
> can
> > > answer those requests with the Markdown copy, so agents get it from the
> > > URLs they already use. I tested this on airflow.staged.apache.org;
> ASF's
> > > Fastly cache keeps the HTML and Markdown answers apart.
> > >
> > > *I won't have time to build this or to review it, so it needs an owner
> > who
> > > can see it through, including reviews.* The issue lists what's done,
> > what's
> > > left, and the rollout constraints. It needs a committer, or a
> contributor
> > > pairing with one: testing the airflow-site change on staging,
> publishing
> > > docs and checking the live rollout all need committer access.
> > >
> > > This would be a high impact change, so thanks in advance for whoever
> > picks
> > > it up.
> > >
> > > Regards,
> > > Kaxil
> > >
> >
>
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Sameer Mesiah <sm...@gmail.com>.
I understand. I don't believe I should have much issue getting a committer
to help me with all of that. In case I am not able to get a committer to
assist me, I will inform you and we can hand it to someone who is a
committer or has that assistance.
Thanks,
Sameer.
On Tue, 6 Oct 2026 at 23:53, Kaxil Naik <ka...@gmail.com> wrote:
> Absolutely happy for you to take it as long as you can pair with a commiter
> because of the testing requirements I mentioned in the previous email and
> the GitHub issue.
>
> *It needs a committer, or a contributor pairing with one: testing the
> airflow-site change on staging, publishing docs and checking the live
> rollout all need committer access.*
>
> On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
>
> > Hi Kaxil,
> >
> > I can pick this up if you are okay with it.
> >
> > On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > More and more people write Dags with coding agents (Claude Code,
> Cursor,
> > > Copilot, Codex), and those agents read our docs as HTML. A provider
> guide
> > > page is about 150 KB of HTML for about 20 KB of content, and fetch
> tools
> > > often summarise it down to a paragraph, code samples included. Pydantic
> > AI,
> > > Ray, Camel, Astronomer and most of the popular sites & project publish
> an
> > > llms.txt index plus a Markdown copy of each page. We publish neither.
> > >
> > > I've opened an issue with a plan and a working proof of concept,
> starting
> > > with the Common AI provider docs:
> > >
> > > https://github.com/apache/airflow/issues/74376
> > >
> > > It has three parts:
> > >
> > > 1. Docs build: generate page.html.md next to every page, plus llms.txt
> > and
> > > llms-full.txt, using sphinx-llm and a small extension. The proof of
> > concept
> > > builds cleanly for Common AI with no change in build time.
> > > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > > Markdown copies.
> > > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> > "Accept:
> > > text/markdown" on every fetch. A short .htaccess rule in airflow-site
> can
> > > answer those requests with the Markdown copy, so agents get it from the
> > > URLs they already use. I tested this on airflow.staged.apache.org;
> ASF's
> > > Fastly cache keeps the HTML and Markdown answers apart.
> > >
> > > *I won't have time to build this or to review it, so it needs an owner
> > who
> > > can see it through, including reviews.* The issue lists what's done,
> > what's
> > > left, and the rollout constraints. It needs a committer, or a
> contributor
> > > pairing with one: testing the airflow-site change on staging,
> publishing
> > > docs and checking the live rollout all need committer access.
> > >
> > > This would be a high impact change, so thanks in advance for whoever
> > picks
> > > it up.
> > >
> > > Regards,
> > > Kaxil
> > >
> >
>
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Kaxil Naik <ka...@gmail.com>.
Absolutely happy for you to take it as long as you can pair with a commiter
because of the testing requirements I mentioned in the previous email and
the GitHub issue.
*It needs a committer, or a contributor pairing with one: testing the
airflow-site change on staging, publishing docs and checking the live
rollout all need committer access.*
On Tue, 6 Oct 2026 at 23:39, Sameer Mesiah <sm...@gmail.com> wrote:
> Hi Kaxil,
>
> I can pick this up if you are okay with it.
>
> On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
>
> > Hi all,
> >
> > More and more people write Dags with coding agents (Claude Code, Cursor,
> > Copilot, Codex), and those agents read our docs as HTML. A provider guide
> > page is about 150 KB of HTML for about 20 KB of content, and fetch tools
> > often summarise it down to a paragraph, code samples included. Pydantic
> AI,
> > Ray, Camel, Astronomer and most of the popular sites & project publish an
> > llms.txt index plus a Markdown copy of each page. We publish neither.
> >
> > I've opened an issue with a plan and a working proof of concept, starting
> > with the Common AI provider docs:
> >
> > https://github.com/apache/airflow/issues/74376
> >
> > It has three parts:
> >
> > 1. Docs build: generate page.html.md next to every page, plus llms.txt
> and
> > llms-full.txt, using sphinx-llm and a small extension. The proof of
> concept
> > builds cleanly for Common AI with no change in build time.
> > 2. Publishing: a couple of guards in breeze so stable/ always has the
> > Markdown copies.
> > 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send
> "Accept:
> > text/markdown" on every fetch. A short .htaccess rule in airflow-site can
> > answer those requests with the Markdown copy, so agents get it from the
> > URLs they already use. I tested this on airflow.staged.apache.org; ASF's
> > Fastly cache keeps the HTML and Markdown answers apart.
> >
> > *I won't have time to build this or to review it, so it needs an owner
> who
> > can see it through, including reviews.* The issue lists what's done,
> what's
> > left, and the rollout constraints. It needs a committer, or a contributor
> > pairing with one: testing the airflow-site change on staging, publishing
> > docs and checking the live rollout all need committer access.
> >
> > This would be a high impact change, so thanks in advance for whoever
> picks
> > it up.
> >
> > Regards,
> > Kaxil
> >
>
Re: [CONTRIBUTION OPPORTUNITY] llms.txt and Markdown docs for coding agents for Airflow site
Posted by Sameer Mesiah <sm...@gmail.com>.
Hi Kaxil,
I can pick this up if you are okay with it.
On Tue, 6 Oct 2026 at 23:28, Kaxil Naik <ka...@gmail.com> wrote:
> Hi all,
>
> More and more people write Dags with coding agents (Claude Code, Cursor,
> Copilot, Codex), and those agents read our docs as HTML. A provider guide
> page is about 150 KB of HTML for about 20 KB of content, and fetch tools
> often summarise it down to a paragraph, code samples included. Pydantic AI,
> Ray, Camel, Astronomer and most of the popular sites & project publish an
> llms.txt index plus a Markdown copy of each page. We publish neither.
>
> I've opened an issue with a plan and a working proof of concept, starting
> with the Common AI provider docs:
>
> https://github.com/apache/airflow/issues/74376
>
> It has three parts:
>
> 1. Docs build: generate page.html.md next to every page, plus llms.txt and
> llms-full.txt, using sphinx-llm and a small extension. The proof of concept
> builds cleanly for Common AI with no change in build time.
> 2. Publishing: a couple of guards in breeze so stable/ always has the
> Markdown copies.
> 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send "Accept:
> text/markdown" on every fetch. A short .htaccess rule in airflow-site can
> answer those requests with the Markdown copy, so agents get it from the
> URLs they already use. I tested this on airflow.staged.apache.org; ASF's
> Fastly cache keeps the HTML and Markdown answers apart.
>
> *I won't have time to build this or to review it, so it needs an owner who
> can see it through, including reviews.* The issue lists what's done, what's
> left, and the rollout constraints. It needs a committer, or a contributor
> pairing with one: testing the airflow-site change on staging, publishing
> docs and checking the live rollout all need committer access.
>
> This would be a high impact change, so thanks in advance for whoever picks
> it up.
>
> Regards,
> Kaxil
>