You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Srabasti Banerjee <sr...@gmail.com> on 2026/03/05 05:29:24 UTC
Re: [DISCUSS] Apache Airflow Provider Registry: a searchable catalog of all providers and modules
This is cool - all providers visually in one place - without scrolling
through pages of documentation - in dark mode - super helpful Kaxil:)
Warm Regards,
Srabasti Banerjee
On Tue, Feb 24, 2026 at 9:37 PM Rahul Vats <ra...@gmail.com> wrote:
> This is great, thank you Kaxil!
>
>
> Regards,
> Rahul Vats
>
> On Tue, 24 Feb 2026 at 19:32, Dheeraj Turaga <dh...@gmail.com>
> wrote:
>
> > Oh wow! Looks amazing! Great work Kaxil!!!
> >
> > On Mon, Feb 23, 2026 at 3:12 PM Buğra Öztürk <oz...@gmail.com>
> > wrote:
> >
> > > Looks really cool, thanks Kaxil! This is a great update, and it feels
> how
> > > it should exhibit in the scale of the Airflow providers :)
> > >
> > > On Mon, Feb 23, 2026 at 5:17 PM Vikram Koka via dev <
> > > dev@airflow.apache.org>
> > > wrote:
> > >
> > > > Excellent work Kaxil!
> > > > This is absolutely transformative for the Airflow user community and
> > the
> > > > integrations effort
> > > >
> > > > Vikram
> > > >
> > > > On Sun, Feb 22, 2026 at 10:47 PM Amogh Desai <am...@apache.org>
> > > > wrote:
> > > >
> > > > > Looks amazing, thanks for working on this, Kaxil.
> > > > >
> > > > > I already see it to be super useful instead of going into
> manhunting
> > > > > on the Airflow docs and google searches.
> > > > >
> > > > > Looking forward to see how this one will be adopted in the future.
> > > > >
> > > > > Thanks & Regards,
> > > > > Amogh Desai
> > > > >
> > > > >
> > > > > On Sun, Feb 22, 2026 at 1:34 AM Jens Scheffler <
> jscheffl@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 into "Very cool" category. The rest of Airflow site
> immediately
> > > > feels
> > > > > > like 1980 :-D
> > > > > >
> > > > > > On 20.02.26 23:37, Kaxil Naik wrote:
> > > > > > > PR: https://github.com/apache/airflow/pull/62261
> > > > > > >
> > > > > > >
> > > > > > > On Fri, 20 Feb 2026 at 21:50, Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Woohoo, what an update , thanks kaxil, it's really nice :)
> > > > > > >>
> > > > > > >> On Fri, Feb 20, 2026 at 9:12 PM Vincent Beck <
> > vincbeck@apache.org
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >>> Very cool indeed :) I can see it very useful for users!
> > > > > > >>>
> > > > > > >>> On 2026/02/20 20:04:58 Jarek Potiuk wrote:
> > > > > > >>>> Very cool :)
> > > > > > >>>>
> > > > > > >>>> J.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On Fri, Feb 20, 2026 at 6:22 PM Zhe-You Liu <
> > > zhu424.dev@gmail.com
> > > > >
> > > > > > >>> wrote:
> > > > > > >>>>> Hi Kaxil,
> > > > > > >>>>>
> > > > > > >>>>> I just checked out the staged site of the official Provider
> > > > > Registry,
> > > > > > >>> and
> > > > > > >>>>> it looks super impressive! The UX is excellent, and it’s
> > > > > definitely a
> > > > > > >>> big
> > > > > > >>>>> plus for users to find existing community integrations for
> > > their
> > > > > use
> > > > > > >>> cases
> > > > > > >>>>> and involve more Airflow users. I really appreciate your
> > > effort,
> > > > > and
> > > > > > >> I
> > > > > > >>> will
> > > > > > >>>>> check out the PR when it comes out.
> > > > > > >>>>>
> > > > > > >>>>> Best regards,
> > > > > > >>>>> Jason
> > > > > > >>>>>
> > > > > > >>>>> On Sat, Feb 21, 2026 at 1:09 AM Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>>>> Hey all,
> > > > > > >>>>>>
> > > > > > >>>>>> *tl;dr*: I'm proposing an official Provider Registry for
> the
> > > > > > >> Airflow
> > > > > > >>>>>> project, deployed at https://airflow.apache.org/registry/
> .
> > > > > > >>>>>>
> > > > > > >>>>>> Preview is up at
> > https://airflow.staged.apache.org/registry/
> > > --
> > > > > > >>> take a
> > > > > > >>>>>> look
> > > > > > >>>>>> and let me know what you think.
> > > > > > >>>>>>
> > > > > > >>>>>> PR to Airflow repo incoming in a couple of hours :)
> > > > > > >>>>>>
> > > > > > >>>>>> *Why now*
> > > > > > >>>>>> With AIP-95 approved, Airflow now has a formal provider
> > > > lifecycle:
> > > > > > >>>>>> incubation, production, mature, and deprecated. That opens
> > the
> > > > > door
> > > > > > >>> for
> > > > > > >>>>>> accepting more community-built providers and giving them
> an
> > > > > > >> official
> > > > > > >>>>> home,
> > > > > > >>>>>> while setting clear expectations about maturity and
> support.
> > > But
> > > > > > >>>>> lifecycle
> > > > > > >>>>>> stages only work if users can actually see them.
> > > > > > >>>>>>
> > > > > > >>>>>> Right now, there's no place on airflow.apache.org where
> > > someone
> > > > > > >> can
> > > > > > >>>>> browse
> > > > > > >>>>>> providers, check their lifecycle stage, or discover what
> > > modules
> > > > > > >> they
> > > > > > >>>>> ship.
> > > > > > >>>>>> This registry fills that gap. It gives the PMC a tool to
> > > > > > >> communicate
> > > > > > >>>>>> provider maturity to users, and it gives the community an
> > > > official
> > > > > > >>> way to
> > > > > > >>>>>> surface new providers -- clearly labelled with their
> > lifecycle
> > > > > > >> stage.
> > > > > > >>>>>> *What it does*
> > > > > > >>>>>> The registry currently catalogs 99 providers and 1,648
> > modules
> > > > > > >>> across all
> > > > > > >>>>>> 11 module types (operators, hooks, sensors, triggers,
> > > transfers,
> > > > > > >>>>> executors,
> > > > > > >>>>>> notifiers, secret backends, logging handlers, Dag bundles,
> > and
> > > > > > >>>>> decorators).
> > > > > > >>>>>> It's built with Eleventy <https://www.11ty.dev/> (thanks
> > Ash,
> > > > for
> > > > > > >>> the
> > > > > > >>>>>> suggestion and for prototyping an approach with it) and
> > > > > > >>> auto-generated
> > > > > > >>>>>> directly from the provider.yaml files in the repo -- no
> > > separate
> > > > > > >> data
> > > > > > >>>>>> pipeline, no manual curation. When a provider is added or
> > > > updated,
> > > > > > >>> the
> > > > > > >>>>> next
> > > > > > >>>>>> CI build picks it up automatically.
> > > > > > >>>>>>
> > > > > > >>>>>> The entire registry is a static site (HTML, CSS, JS): no
> > > server,
> > > > > no
> > > > > > >>>>>> database, same deployment model as the existing Airflow
> > docs.
> > > > It's
> > > > > > >>>>>> generated at build time from the provider.yaml files and
> > > served
> > > > > > >> from
> > > > > > >>> S3
> > > > > > >>>>> via
> > > > > > >>>>>> CloudFront.
> > > > > > >>>>>>
> > > > > > >>>>>> *A few things you can do with it*:
> > > > > > >>>>>>
> > > > > > >>>>>> - Search across all providers and modules (Cmd+K,
> > powered
> > > by
> > > > > > >>> Pagefind)
> > > > > > >>>>>> - Browse by category (Cloud, Databases, AI & ML, etc.)
> > > > > > >>>>>> - Filter/sort by lifecycle stage, downloads, module
> > count
> > > > > > >>>>>> - Explore provider detail pages with per-version
> module
> > > > > > >> listings,
> > > > > > >>>>>> connection types, parameters, and install commands
> > > > > > >>>>>> - Access JSON API endpoints (/api/providers.json,
> > > > > > >>> /api/modules.json)
> > > > > > >>>>> for
> > > > > > >>>>>> programmatic access -- useful for AI agents and
> tooling
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> The design is deliberately discovery-first: it links out
> to
> > > the
> > > > > API
> > > > > > >>>>>> reference docs and user guides rather than hosting
> > everything
> > > > > > >> itself.
> > > > > > >>>>> This
> > > > > > >>>>>> avoids duplicating content between provider docs and
> > registry
> > > > > > >>> entries.
> > > > > > >>>>>> CI/CD is integrated with our existing docs pipeline and
> > syncs
> > > to
> > > > > S3
> > > > > > >>>>>> automatically. Nothing in provider code, provider.yaml
> > > schemas,
> > > > > > >> core
> > > > > > >>>>>> Airflow, or the docs build is changed by this.
> > > > > > >>>>>>
> > > > > > >>>>>> *How it relates to the Astronomer Registry*
> > > > > > >>>>>> Many of you know the Astronomer Registry (
> > > > > > >>> https://registry.astronomer.io
> > > > > > >>>>> ),
> > > > > > >>>>>> which has been the go-to for discovering Airflow providers
> > for
> > > > > > >>> years. Big
> > > > > > >>>>>> thanks to Astronomer and Josh Fell for building and
> > > maintaining
> > > > > it.
> > > > > > >>>>>>
> > > > > > >>>>>> This new registry is designed to be a community-owned
> > > successor
> > > > on
> > > > > > >>>>>> airflow.apache.org, with the eventual goal of redirecting
> > > > > > >>>>>> registry.astronomer.io traffic here once it's stable.
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> *Remaining work*
> > > > > > >>>>>> Still to do after this lands:
> > > > > > >>>>>>
> > > > > > >>>>>> - apache/airflow-site PR for .htaccess rewrite and a
> > > > > "Registry"
> > > > > > >>> nav
> > > > > > >>>>> link
> > > > > > >>>>>> - Redirect registry.astronomer.io traffic once the
> > > official
> > > > > one
> > > > > > >>> is
> > > > > > >>>>>> stable
> > > > > > >>>>>> - A way to add third-party providers that are not in
> the
> > > > > Airflow
> > > > > > >>> repo,
> > > > > > >>>>>> like Great Expectations, Cosmos etc - I have a POC
> > working
> > > > on
> > > > > > >>> this.
> > > > > > >>>>>>
> > > > > > >>>>>> *Future ideas (will create GH issues)*
> > > > > > >>>>>>
> > > > > > >>>>>> - Explicit categories in provider.yaml (currently
> > > > > keyword-based
> > > > > > >>>>>> matching)
> > > > > > >>>>>> - LLM-friendly exports (llms.txt, "Copy for AI"
> buttons
> > > > etc.)
> > > > > > >>>>>> - Example DAGs for each Provider.
> > > > > > >>>>>> - and many more – but I think the current state is
> > > valuable
> > > > > > >> enough
> > > > > > >>>>>> already
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> I'd appreciate feedback and reviews!
> > > > > > >>>>>>
> > > > > > >>>>>> Regards,
> > > > > > >>>>>> Kaxil
> > > > > > >>>>>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > >>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >>>
> > > > > > >>>
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Bugra Ozturk
> > >
> >
>
Re: [DISCUSS] Apache Airflow Provider Registry: a searchable catalog of all providers and modules
Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool. This is a fantastic start to work on more tooling to help our
stewards to do their duties for providers they will "steward". We very much
need it :)
J.
On Sat, Mar 7, 2026 at 5:08 AM Kaxil Naik <ka...@gmail.com> wrote:
> Thank you for all the feedback, I really appreciate it.
>
> In the interest of ensuring reviews become manageable I merged the PR. If
> there are any issues in CI or around doc publishing, please let me know.
>
> Nothing would show up in the "live" (non-staging site) until I add
> `.htaccess`. I plan to work on incremental updates to handle some of the
> todos and review comments as separate PRs next week with a goal to get
> the *Airflow
> Registry* live around the same timing as Airflow 3.2 or by the week of 16th
> March.
>
> I hope to write some sort of blog post on the Airflow website about it at
> the time of launch.
>
> Regards,
> Kaxil
>
> On Thu, 5 Mar 2026 at 05:30, Srabasti Banerjee <
> srabasti.banerjee.c@gmail.com> wrote:
>
> > This is cool - all providers visually in one place - without scrolling
> > through pages of documentation - in dark mode - super helpful Kaxil:)
> >
> > Warm Regards,
> > Srabasti Banerjee
> >
> > On Tue, Feb 24, 2026 at 9:37 PM Rahul Vats <ra...@gmail.com>
> wrote:
> >
> > > This is great, thank you Kaxil!
> > >
> > >
> > > Regards,
> > > Rahul Vats
> > >
> > > On Tue, 24 Feb 2026 at 19:32, Dheeraj Turaga <dh...@gmail.com>
> > > wrote:
> > >
> > > > Oh wow! Looks amazing! Great work Kaxil!!!
> > > >
> > > > On Mon, Feb 23, 2026 at 3:12 PM Buğra Öztürk <
> ozturkbugra93@gmail.com>
> > > > wrote:
> > > >
> > > > > Looks really cool, thanks Kaxil! This is a great update, and it
> feels
> > > how
> > > > > it should exhibit in the scale of the Airflow providers :)
> > > > >
> > > > > On Mon, Feb 23, 2026 at 5:17 PM Vikram Koka via dev <
> > > > > dev@airflow.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Excellent work Kaxil!
> > > > > > This is absolutely transformative for the Airflow user community
> > and
> > > > the
> > > > > > integrations effort
> > > > > >
> > > > > > Vikram
> > > > > >
> > > > > > On Sun, Feb 22, 2026 at 10:47 PM Amogh Desai <
> > amoghdesai@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Looks amazing, thanks for working on this, Kaxil.
> > > > > > >
> > > > > > > I already see it to be super useful instead of going into
> > > manhunting
> > > > > > > on the Airflow docs and google searches.
> > > > > > >
> > > > > > > Looking forward to see how this one will be adopted in the
> > future.
> > > > > > >
> > > > > > > Thanks & Regards,
> > > > > > > Amogh Desai
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 22, 2026 at 1:34 AM Jens Scheffler <
> > > jscheffl@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 into "Very cool" category. The rest of Airflow site
> > > immediately
> > > > > > feels
> > > > > > > > like 1980 :-D
> > > > > > > >
> > > > > > > > On 20.02.26 23:37, Kaxil Naik wrote:
> > > > > > > > > PR: https://github.com/apache/airflow/pull/62261
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, 20 Feb 2026 at 21:50, Pavankumar Gopidesu <
> > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Woohoo, what an update , thanks kaxil, it's really nice :)
> > > > > > > > >>
> > > > > > > > >> On Fri, Feb 20, 2026 at 9:12 PM Vincent Beck <
> > > > vincbeck@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Very cool indeed :) I can see it very useful for users!
> > > > > > > > >>>
> > > > > > > > >>> On 2026/02/20 20:04:58 Jarek Potiuk wrote:
> > > > > > > > >>>> Very cool :)
> > > > > > > > >>>>
> > > > > > > > >>>> J.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Fri, Feb 20, 2026 at 6:22 PM Zhe-You Liu <
> > > > > zhu424.dev@gmail.com
> > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>>>> Hi Kaxil,
> > > > > > > > >>>>>
> > > > > > > > >>>>> I just checked out the staged site of the official
> > Provider
> > > > > > > Registry,
> > > > > > > > >>> and
> > > > > > > > >>>>> it looks super impressive! The UX is excellent, and
> it’s
> > > > > > > definitely a
> > > > > > > > >>> big
> > > > > > > > >>>>> plus for users to find existing community integrations
> > for
> > > > > their
> > > > > > > use
> > > > > > > > >>> cases
> > > > > > > > >>>>> and involve more Airflow users. I really appreciate
> your
> > > > > effort,
> > > > > > > and
> > > > > > > > >> I
> > > > > > > > >>> will
> > > > > > > > >>>>> check out the PR when it comes out.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Best regards,
> > > > > > > > >>>>> Jason
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Sat, Feb 21, 2026 at 1:09 AM Kaxil Naik <
> > > > > kaxilnaik@gmail.com>
> > > > > > > > >>> wrote:
> > > > > > > > >>>>>> Hey all,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *tl;dr*: I'm proposing an official Provider Registry
> for
> > > the
> > > > > > > > >> Airflow
> > > > > > > > >>>>>> project, deployed at
> > https://airflow.apache.org/registry/
> > > .
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Preview is up at
> > > > https://airflow.staged.apache.org/registry/
> > > > > --
> > > > > > > > >>> take a
> > > > > > > > >>>>>> look
> > > > > > > > >>>>>> and let me know what you think.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> PR to Airflow repo incoming in a couple of hours :)
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *Why now*
> > > > > > > > >>>>>> With AIP-95 approved, Airflow now has a formal
> provider
> > > > > > lifecycle:
> > > > > > > > >>>>>> incubation, production, mature, and deprecated. That
> > opens
> > > > the
> > > > > > > door
> > > > > > > > >>> for
> > > > > > > > >>>>>> accepting more community-built providers and giving
> them
> > > an
> > > > > > > > >> official
> > > > > > > > >>>>> home,
> > > > > > > > >>>>>> while setting clear expectations about maturity and
> > > support.
> > > > > But
> > > > > > > > >>>>> lifecycle
> > > > > > > > >>>>>> stages only work if users can actually see them.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Right now, there's no place on airflow.apache.org
> where
> > > > > someone
> > > > > > > > >> can
> > > > > > > > >>>>> browse
> > > > > > > > >>>>>> providers, check their lifecycle stage, or discover
> what
> > > > > modules
> > > > > > > > >> they
> > > > > > > > >>>>> ship.
> > > > > > > > >>>>>> This registry fills that gap. It gives the PMC a tool
> to
> > > > > > > > >> communicate
> > > > > > > > >>>>>> provider maturity to users, and it gives the community
> > an
> > > > > > official
> > > > > > > > >>> way to
> > > > > > > > >>>>>> surface new providers -- clearly labelled with their
> > > > lifecycle
> > > > > > > > >> stage.
> > > > > > > > >>>>>> *What it does*
> > > > > > > > >>>>>> The registry currently catalogs 99 providers and 1,648
> > > > modules
> > > > > > > > >>> across all
> > > > > > > > >>>>>> 11 module types (operators, hooks, sensors, triggers,
> > > > > transfers,
> > > > > > > > >>>>> executors,
> > > > > > > > >>>>>> notifiers, secret backends, logging handlers, Dag
> > bundles,
> > > > and
> > > > > > > > >>>>> decorators).
> > > > > > > > >>>>>> It's built with Eleventy <https://www.11ty.dev/>
> > (thanks
> > > > Ash,
> > > > > > for
> > > > > > > > >>> the
> > > > > > > > >>>>>> suggestion and for prototyping an approach with it)
> and
> > > > > > > > >>> auto-generated
> > > > > > > > >>>>>> directly from the provider.yaml files in the repo --
> no
> > > > > separate
> > > > > > > > >> data
> > > > > > > > >>>>>> pipeline, no manual curation. When a provider is added
> > or
> > > > > > updated,
> > > > > > > > >>> the
> > > > > > > > >>>>> next
> > > > > > > > >>>>>> CI build picks it up automatically.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> The entire registry is a static site (HTML, CSS, JS):
> no
> > > > > server,
> > > > > > > no
> > > > > > > > >>>>>> database, same deployment model as the existing
> Airflow
> > > > docs.
> > > > > > It's
> > > > > > > > >>>>>> generated at build time from the provider.yaml files
> and
> > > > > served
> > > > > > > > >> from
> > > > > > > > >>> S3
> > > > > > > > >>>>> via
> > > > > > > > >>>>>> CloudFront.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *A few things you can do with it*:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> - Search across all providers and modules (Cmd+K,
> > > > powered
> > > > > by
> > > > > > > > >>> Pagefind)
> > > > > > > > >>>>>> - Browse by category (Cloud, Databases, AI & ML,
> > etc.)
> > > > > > > > >>>>>> - Filter/sort by lifecycle stage, downloads,
> module
> > > > count
> > > > > > > > >>>>>> - Explore provider detail pages with per-version
> > > module
> > > > > > > > >> listings,
> > > > > > > > >>>>>> connection types, parameters, and install commands
> > > > > > > > >>>>>> - Access JSON API endpoints (/api/providers.json,
> > > > > > > > >>> /api/modules.json)
> > > > > > > > >>>>> for
> > > > > > > > >>>>>> programmatic access -- useful for AI agents and
> > > tooling
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> The design is deliberately discovery-first: it links
> out
> > > to
> > > > > the
> > > > > > > API
> > > > > > > > >>>>>> reference docs and user guides rather than hosting
> > > > everything
> > > > > > > > >> itself.
> > > > > > > > >>>>> This
> > > > > > > > >>>>>> avoids duplicating content between provider docs and
> > > > registry
> > > > > > > > >>> entries.
> > > > > > > > >>>>>> CI/CD is integrated with our existing docs pipeline
> and
> > > > syncs
> > > > > to
> > > > > > > S3
> > > > > > > > >>>>>> automatically. Nothing in provider code, provider.yaml
> > > > > schemas,
> > > > > > > > >> core
> > > > > > > > >>>>>> Airflow, or the docs build is changed by this.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *How it relates to the Astronomer Registry*
> > > > > > > > >>>>>> Many of you know the Astronomer Registry (
> > > > > > > > >>> https://registry.astronomer.io
> > > > > > > > >>>>> ),
> > > > > > > > >>>>>> which has been the go-to for discovering Airflow
> > providers
> > > > for
> > > > > > > > >>> years. Big
> > > > > > > > >>>>>> thanks to Astronomer and Josh Fell for building and
> > > > > maintaining
> > > > > > > it.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> This new registry is designed to be a community-owned
> > > > > successor
> > > > > > on
> > > > > > > > >>>>>> airflow.apache.org, with the eventual goal of
> > redirecting
> > > > > > > > >>>>>> registry.astronomer.io traffic here once it's stable.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *Remaining work*
> > > > > > > > >>>>>> Still to do after this lands:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> - apache/airflow-site PR for .htaccess rewrite
> and a
> > > > > > > "Registry"
> > > > > > > > >>> nav
> > > > > > > > >>>>> link
> > > > > > > > >>>>>> - Redirect registry.astronomer.io traffic once
> the
> > > > > official
> > > > > > > one
> > > > > > > > >>> is
> > > > > > > > >>>>>> stable
> > > > > > > > >>>>>> - A way to add third-party providers that are not
> in
> > > the
> > > > > > > Airflow
> > > > > > > > >>> repo,
> > > > > > > > >>>>>> like Great Expectations, Cosmos etc - I have a POC
> > > > working
> > > > > > on
> > > > > > > > >>> this.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *Future ideas (will create GH issues)*
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> - Explicit categories in provider.yaml (currently
> > > > > > > keyword-based
> > > > > > > > >>>>>> matching)
> > > > > > > > >>>>>> - LLM-friendly exports (llms.txt, "Copy for AI"
> > > buttons
> > > > > > etc.)
> > > > > > > > >>>>>> - Example DAGs for each Provider.
> > > > > > > > >>>>>> - and many more – but I think the current state is
> > > > > valuable
> > > > > > > > >> enough
> > > > > > > > >>>>>> already
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I'd appreciate feedback and reviews!
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Regards,
> > > > > > > > >>>>>> Kaxil
> > > > > > > > >>>>>>
> > > > > > > > >>>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >>> To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > > > > > >>> For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Bugra Ozturk
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Apache Airflow Provider Registry: a searchable catalog of all providers and modules
Posted by Kaxil Naik <ka...@gmail.com>.
Thank you for all the feedback, I really appreciate it.
In the interest of ensuring reviews become manageable I merged the PR. If
there are any issues in CI or around doc publishing, please let me know.
Nothing would show up in the "live" (non-staging site) until I add
`.htaccess`. I plan to work on incremental updates to handle some of the
todos and review comments as separate PRs next week with a goal to get
the *Airflow
Registry* live around the same timing as Airflow 3.2 or by the week of 16th
March.
I hope to write some sort of blog post on the Airflow website about it at
the time of launch.
Regards,
Kaxil
On Thu, 5 Mar 2026 at 05:30, Srabasti Banerjee <
srabasti.banerjee.c@gmail.com> wrote:
> This is cool - all providers visually in one place - without scrolling
> through pages of documentation - in dark mode - super helpful Kaxil:)
>
> Warm Regards,
> Srabasti Banerjee
>
> On Tue, Feb 24, 2026 at 9:37 PM Rahul Vats <ra...@gmail.com> wrote:
>
> > This is great, thank you Kaxil!
> >
> >
> > Regards,
> > Rahul Vats
> >
> > On Tue, 24 Feb 2026 at 19:32, Dheeraj Turaga <dh...@gmail.com>
> > wrote:
> >
> > > Oh wow! Looks amazing! Great work Kaxil!!!
> > >
> > > On Mon, Feb 23, 2026 at 3:12 PM Buğra Öztürk <oz...@gmail.com>
> > > wrote:
> > >
> > > > Looks really cool, thanks Kaxil! This is a great update, and it feels
> > how
> > > > it should exhibit in the scale of the Airflow providers :)
> > > >
> > > > On Mon, Feb 23, 2026 at 5:17 PM Vikram Koka via dev <
> > > > dev@airflow.apache.org>
> > > > wrote:
> > > >
> > > > > Excellent work Kaxil!
> > > > > This is absolutely transformative for the Airflow user community
> and
> > > the
> > > > > integrations effort
> > > > >
> > > > > Vikram
> > > > >
> > > > > On Sun, Feb 22, 2026 at 10:47 PM Amogh Desai <
> amoghdesai@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Looks amazing, thanks for working on this, Kaxil.
> > > > > >
> > > > > > I already see it to be super useful instead of going into
> > manhunting
> > > > > > on the Airflow docs and google searches.
> > > > > >
> > > > > > Looking forward to see how this one will be adopted in the
> future.
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Amogh Desai
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 22, 2026 at 1:34 AM Jens Scheffler <
> > jscheffl@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 into "Very cool" category. The rest of Airflow site
> > immediately
> > > > > feels
> > > > > > > like 1980 :-D
> > > > > > >
> > > > > > > On 20.02.26 23:37, Kaxil Naik wrote:
> > > > > > > > PR: https://github.com/apache/airflow/pull/62261
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, 20 Feb 2026 at 21:50, Pavankumar Gopidesu <
> > > > > > > gopidesupavan@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Woohoo, what an update , thanks kaxil, it's really nice :)
> > > > > > > >>
> > > > > > > >> On Fri, Feb 20, 2026 at 9:12 PM Vincent Beck <
> > > vincbeck@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Very cool indeed :) I can see it very useful for users!
> > > > > > > >>>
> > > > > > > >>> On 2026/02/20 20:04:58 Jarek Potiuk wrote:
> > > > > > > >>>> Very cool :)
> > > > > > > >>>>
> > > > > > > >>>> J.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> On Fri, Feb 20, 2026 at 6:22 PM Zhe-You Liu <
> > > > zhu424.dev@gmail.com
> > > > > >
> > > > > > > >>> wrote:
> > > > > > > >>>>> Hi Kaxil,
> > > > > > > >>>>>
> > > > > > > >>>>> I just checked out the staged site of the official
> Provider
> > > > > > Registry,
> > > > > > > >>> and
> > > > > > > >>>>> it looks super impressive! The UX is excellent, and it’s
> > > > > > definitely a
> > > > > > > >>> big
> > > > > > > >>>>> plus for users to find existing community integrations
> for
> > > > their
> > > > > > use
> > > > > > > >>> cases
> > > > > > > >>>>> and involve more Airflow users. I really appreciate your
> > > > effort,
> > > > > > and
> > > > > > > >> I
> > > > > > > >>> will
> > > > > > > >>>>> check out the PR when it comes out.
> > > > > > > >>>>>
> > > > > > > >>>>> Best regards,
> > > > > > > >>>>> Jason
> > > > > > > >>>>>
> > > > > > > >>>>> On Sat, Feb 21, 2026 at 1:09 AM Kaxil Naik <
> > > > kaxilnaik@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>>>> Hey all,
> > > > > > > >>>>>>
> > > > > > > >>>>>> *tl;dr*: I'm proposing an official Provider Registry for
> > the
> > > > > > > >> Airflow
> > > > > > > >>>>>> project, deployed at
> https://airflow.apache.org/registry/
> > .
> > > > > > > >>>>>>
> > > > > > > >>>>>> Preview is up at
> > > https://airflow.staged.apache.org/registry/
> > > > --
> > > > > > > >>> take a
> > > > > > > >>>>>> look
> > > > > > > >>>>>> and let me know what you think.
> > > > > > > >>>>>>
> > > > > > > >>>>>> PR to Airflow repo incoming in a couple of hours :)
> > > > > > > >>>>>>
> > > > > > > >>>>>> *Why now*
> > > > > > > >>>>>> With AIP-95 approved, Airflow now has a formal provider
> > > > > lifecycle:
> > > > > > > >>>>>> incubation, production, mature, and deprecated. That
> opens
> > > the
> > > > > > door
> > > > > > > >>> for
> > > > > > > >>>>>> accepting more community-built providers and giving them
> > an
> > > > > > > >> official
> > > > > > > >>>>> home,
> > > > > > > >>>>>> while setting clear expectations about maturity and
> > support.
> > > > But
> > > > > > > >>>>> lifecycle
> > > > > > > >>>>>> stages only work if users can actually see them.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Right now, there's no place on airflow.apache.org where
> > > > someone
> > > > > > > >> can
> > > > > > > >>>>> browse
> > > > > > > >>>>>> providers, check their lifecycle stage, or discover what
> > > > modules
> > > > > > > >> they
> > > > > > > >>>>> ship.
> > > > > > > >>>>>> This registry fills that gap. It gives the PMC a tool to
> > > > > > > >> communicate
> > > > > > > >>>>>> provider maturity to users, and it gives the community
> an
> > > > > official
> > > > > > > >>> way to
> > > > > > > >>>>>> surface new providers -- clearly labelled with their
> > > lifecycle
> > > > > > > >> stage.
> > > > > > > >>>>>> *What it does*
> > > > > > > >>>>>> The registry currently catalogs 99 providers and 1,648
> > > modules
> > > > > > > >>> across all
> > > > > > > >>>>>> 11 module types (operators, hooks, sensors, triggers,
> > > > transfers,
> > > > > > > >>>>> executors,
> > > > > > > >>>>>> notifiers, secret backends, logging handlers, Dag
> bundles,
> > > and
> > > > > > > >>>>> decorators).
> > > > > > > >>>>>> It's built with Eleventy <https://www.11ty.dev/>
> (thanks
> > > Ash,
> > > > > for
> > > > > > > >>> the
> > > > > > > >>>>>> suggestion and for prototyping an approach with it) and
> > > > > > > >>> auto-generated
> > > > > > > >>>>>> directly from the provider.yaml files in the repo -- no
> > > > separate
> > > > > > > >> data
> > > > > > > >>>>>> pipeline, no manual curation. When a provider is added
> or
> > > > > updated,
> > > > > > > >>> the
> > > > > > > >>>>> next
> > > > > > > >>>>>> CI build picks it up automatically.
> > > > > > > >>>>>>
> > > > > > > >>>>>> The entire registry is a static site (HTML, CSS, JS): no
> > > > server,
> > > > > > no
> > > > > > > >>>>>> database, same deployment model as the existing Airflow
> > > docs.
> > > > > It's
> > > > > > > >>>>>> generated at build time from the provider.yaml files and
> > > > served
> > > > > > > >> from
> > > > > > > >>> S3
> > > > > > > >>>>> via
> > > > > > > >>>>>> CloudFront.
> > > > > > > >>>>>>
> > > > > > > >>>>>> *A few things you can do with it*:
> > > > > > > >>>>>>
> > > > > > > >>>>>> - Search across all providers and modules (Cmd+K,
> > > powered
> > > > by
> > > > > > > >>> Pagefind)
> > > > > > > >>>>>> - Browse by category (Cloud, Databases, AI & ML,
> etc.)
> > > > > > > >>>>>> - Filter/sort by lifecycle stage, downloads, module
> > > count
> > > > > > > >>>>>> - Explore provider detail pages with per-version
> > module
> > > > > > > >> listings,
> > > > > > > >>>>>> connection types, parameters, and install commands
> > > > > > > >>>>>> - Access JSON API endpoints (/api/providers.json,
> > > > > > > >>> /api/modules.json)
> > > > > > > >>>>> for
> > > > > > > >>>>>> programmatic access -- useful for AI agents and
> > tooling
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> The design is deliberately discovery-first: it links out
> > to
> > > > the
> > > > > > API
> > > > > > > >>>>>> reference docs and user guides rather than hosting
> > > everything
> > > > > > > >> itself.
> > > > > > > >>>>> This
> > > > > > > >>>>>> avoids duplicating content between provider docs and
> > > registry
> > > > > > > >>> entries.
> > > > > > > >>>>>> CI/CD is integrated with our existing docs pipeline and
> > > syncs
> > > > to
> > > > > > S3
> > > > > > > >>>>>> automatically. Nothing in provider code, provider.yaml
> > > > schemas,
> > > > > > > >> core
> > > > > > > >>>>>> Airflow, or the docs build is changed by this.
> > > > > > > >>>>>>
> > > > > > > >>>>>> *How it relates to the Astronomer Registry*
> > > > > > > >>>>>> Many of you know the Astronomer Registry (
> > > > > > > >>> https://registry.astronomer.io
> > > > > > > >>>>> ),
> > > > > > > >>>>>> which has been the go-to for discovering Airflow
> providers
> > > for
> > > > > > > >>> years. Big
> > > > > > > >>>>>> thanks to Astronomer and Josh Fell for building and
> > > > maintaining
> > > > > > it.
> > > > > > > >>>>>>
> > > > > > > >>>>>> This new registry is designed to be a community-owned
> > > > successor
> > > > > on
> > > > > > > >>>>>> airflow.apache.org, with the eventual goal of
> redirecting
> > > > > > > >>>>>> registry.astronomer.io traffic here once it's stable.
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> *Remaining work*
> > > > > > > >>>>>> Still to do after this lands:
> > > > > > > >>>>>>
> > > > > > > >>>>>> - apache/airflow-site PR for .htaccess rewrite and a
> > > > > > "Registry"
> > > > > > > >>> nav
> > > > > > > >>>>> link
> > > > > > > >>>>>> - Redirect registry.astronomer.io traffic once the
> > > > official
> > > > > > one
> > > > > > > >>> is
> > > > > > > >>>>>> stable
> > > > > > > >>>>>> - A way to add third-party providers that are not in
> > the
> > > > > > Airflow
> > > > > > > >>> repo,
> > > > > > > >>>>>> like Great Expectations, Cosmos etc - I have a POC
> > > working
> > > > > on
> > > > > > > >>> this.
> > > > > > > >>>>>>
> > > > > > > >>>>>> *Future ideas (will create GH issues)*
> > > > > > > >>>>>>
> > > > > > > >>>>>> - Explicit categories in provider.yaml (currently
> > > > > > keyword-based
> > > > > > > >>>>>> matching)
> > > > > > > >>>>>> - LLM-friendly exports (llms.txt, "Copy for AI"
> > buttons
> > > > > etc.)
> > > > > > > >>>>>> - Example DAGs for each Provider.
> > > > > > > >>>>>> - and many more – but I think the current state is
> > > > valuable
> > > > > > > >> enough
> > > > > > > >>>>>> already
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> I'd appreciate feedback and reviews!
> > > > > > > >>>>>>
> > > > > > > >>>>>> Regards,
> > > > > > > >>>>>> Kaxil
> > > > > > > >>>>>>
> > > > > > > >>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > >>> For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > > > >>>
> > > > > > > >>>
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Bugra Ozturk
> > > >
> > >
> >
>