[DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Shahar Epstein <sh...@apache.org>]

Hi all,

This follows up on the earlier lazy-consensus
<https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> and
related discussions around deprecating AirflowException.

While working on a cleanup PR for the Google Cloud Run operator in the
Google provider (#67769), Amogh pointed out that replacing AirflowException
with built-in or custom exceptions is a breaking change. Users may rely on
AirflowException in callbacks (on_failure_callback, on_retry_callback) or
in custom wrappers that catch it explicitly.

Rather than making these changes incrementally across individual PRs, I'd
like to discuss a coordinated migration strategy.
Proposal

*1. Emit a deprecation warning from **AirflowException.__init__**.*

Although emitting warnings when exceptions are raised is uncommon, it is
justified here because users may depend on AirflowException in callbacks
and exception handlers. This provides advance notice before
package-specific migrations occur.

*2. Migrate per package in the next breaking release of that package.*

Require a single migration PR per package (airflow-core and each provider),
rather than spreading exception changes across multiple PRs and releases.
The migration would be included in that package's next breaking release and
documented accordingly.
Questions

   - Should we add a deprecation warning to AirflowException?
   - Is the per-package migration approach reasonable?
   - Should we define a project-wide exception taxonomy, or leave exception
   choices to individual provider teams?

If there's agreement, I'll follow up with a lazy-consensus proposal.

Thanks,


Shahar

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Blain David <da...@infrabel.be>]

Ok, I always thought of AirflowException as  base exception related to Airflow and then have specific AirflowExceptions inheriting from that one like AirflowOptionalProviderFeatureException or AirflowTimetableInvalid,
hence why it wasn't that clear to me.  But indeed raising AirflowException doesn't give any context to why the exception was being thrown.  Thx Amogh
________________________________
From: Amogh Desai <am...@apache.org>
Sent: Monday, June 1, 2026 14:24
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.

David - It is specificity. `AirflowException` is a catch-all exception and
whenever
it is raised, the caller has no information about what went wrong without
parsing
the message string. An exception such as `ConnectionTimeoutError` on the
other
hand tells you immediately what happened and lets you handle different
failure
modes differently in code.

You cannot write `except AirflowException` and meaningfully distinguish a
transient
network failure from a permanent config error without inspecting the message

On the contrary, builtins like ValueError, ConnectionError, TimeoutError
integrate
naturally with standard python tooling and are familiar to users who are
not deeply
involved Airflow users and to add, provider-specific exception types give
provider
maintainers a clean(er) surface to document resolution steps

The goal is not to do this overnight but to stop using it as a default when
something
more specific is appropriate and/or available.

Thanks & Regards,
Amogh Desai


On Mon, Jun 1, 2026 at 5:22 PM Blain David <da...@infrabel.be> wrote:

> Probably a stupid question from my side, but why do we want to migrate
> away from AirflowException?
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Monday, June 01, 2026 10:48
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [DISCUSS] Migrating from AirflowException to built-in /
> custom exceptions
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> abuse@infrabel.be>.
>
> - Where does the database live and who owns it? Is this a new repo, part of
> the docs, something in CLAUDE.md?
>
> Just generated docs in airflow.apache.org where the errors point at. No
> "database" per-se
>
> - What does the write path look like? Agents write during PR review, but
> what would be the trigger? any new exception type,
> any failing test, or something else?
>
> Just regular PR updates and AGENTS.md instructing to reconcile it whenever
> something about exceptions change.
>
> On Mon, Jun 1, 2026 at 7:56 AM Amogh Desai <am...@apache.org> wrote:
>
> > I like TP's dual-subclass approach. I see it as the most well-scoped and
> > the cleanest solution here and I think it
> > resolves most of the open questions. Raise a specific exception that
> > inherits from both AirflowException and the
> > target type, the existing catch based code will keep working, new code
> gets
> > specificity, no deprecation cycles or windows
> > to manage.
> >
> > On Jarek's suggestions database idea, I think that's worth a separate
> > thread. The exception hierarchy should not be designed
> > around a storage/retrieval system that does not exist yet. And the
> database
> > idea does not actually depend on a new exception
> > hierarchy either, it could work off existing error messages. Conflating
> the
> > two makes both harder to evaluate.
> >
> > Jarek, a couple of questions if you want to push the suggestions database
> > idea forward:
> >
> > - Where does the database live and who owns it? Is this a new repo, part
> of
> > the docs, something in CLAUDE.md?
> > - What does the write path look like? Agents write during PR review, but
> > what would be the trigger? any new exception type,
> > any failing test, or something else?
> >
> > Happy to ask further details and see it in a different proposal if we
> have
> > appetite at the moment.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Sun, May 31, 2026 at 10:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > Okay. That sounds great. It would be interesting to see what the
> system
> > >
> > > Well.. If others agree that is :). That is just one of the options :)
> > >
> > > On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <sm...@gmail.com>
> > > wrote:
> > >
> > > > Okay. That sounds great. It would be interesting to see what the
> system
> > > > looks like when it is live.
> > > >
> > > > Thanks,
> > > > Sameer Mesiah.
> > > >
> > > > On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > > Hi Sameer,
> > > > >
> > > > > Thanks for your thoughtful questions! Here are my thoughts on those
> > > > points:
> > > > >
> > > > > 1)  I see the value here mainly in terms of efficiency and
> > > > sustainability.
> > > > > By computing a solution once and storing it, we can save the time
> and
> > > > > tokens an LLM would otherwise use to solve the same problem over
> and
> > > > over.
> > > > > It's a nice way to be more environmentally friendly, too!
> > > > >
> > > > > 2)  Since many contributions are already agent-assisted, we can
> > address
> > > > > hallucinations by adding some guidance to AGENTS.md. Rather than
> > > relying
> > > > on
> > > > > periodic checks, we can ask agents to perform their own consistency
> > and
> > > > > hallucination checks during the PR process. This keeps a human
> > > naturally
> > > > in
> > > > > the loop during review, turning token usage into a great
> opportunity
> > > for
> > > > > automated quality control—which we ask our contributors' agents to
> > > > perform.
> > > > >
> > > > > This is actually a very interesting "twist" - because this also
> > nicely
> > > > > approaches the "assymetry" we have by having a lot more AI assisted
> > > > > contributions. We can ask - in our AGENTS.md those agents to do
> > **way**
> > > > mor
> > > > > things to make our projects better - all of them done on the local
> > > > machines
> > > > > of the contributors, with their tokens being spent.
> > > > >
> > > > > Best,
> > > > > Jarek Potiuk
> > > > >
> > > > >
> > > > > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <
> smesiah971@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jarek,
> > > > > >
> > > > > > I do have a few questions about the "suggestions database" point:
> > > > > >
> > > > > > 1) Previously, it was agreed that since LLMs work well with
> > > > unstructured
> > > > > > error messages, there was minimal benefit to an additional
> taxonomy
> > > > layer
> > > > > > on top of existing exceptions. I was just hoping you could
> > elaborate
> > > on
> > > > > the
> > > > > > marginal value of a structured data store over better exception
> > > > messages?
> > > > > >
> > > > > > 2) If we were to implement this "suggestions database", do you
> > > believe
> > > > > the
> > > > > > hallucination rate of the current agentic systems we have
> available
> > > is
> > > > > low
> > > > > > enough that it requires minimal manual intervention to the extent
> > > that
> > > > it
> > > > > > is worthwhile to introduce? If a maintainer/committer/contributor
> > has
> > > > to
> > > > > > skim through the database periodically to remove hallucinations,
> > > would
> > > > we
> > > > > > not be expending tokens in favour of negligible time savings?
> > > > > >
> > > > > > Granted, I could be overestimating the hallucination rate of
> these
> > > > > agentic
> > > > > > systems (I have never used them personally) so it might be the
> > ideal
> > > > > > solution. It's an interesting idea anyway so I would love to hear
> > > more
> > > > > > about it.
> > > > > >
> > > > > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > > > > >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Ff275xo3tjd4olsmrc8nggncs62fjnl0x&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cdfc5e56a73b5436493ed08debfd8f869%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159135797574631%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=U2S9D1TTDtsQhNfOpz6dbBvFcLTu4YMxw%2F9QnOIVGOQ%3D&reserved=0<https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x>
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Ff275xo3tjd4olsmrc8nggncs62fjnl0x&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cdfc5e56a73b5436493ed08debfd8f869%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159135797617081%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lPY9AQuVMo7mRqWb2raKWtrCQicn0g8oN2tPfXfh08o%3D&reserved=0<https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x>>
> > ).
> > > I
> > > > > see
> > > > > > > two main approaches:
> > > > > > >
> > > > > > > 1.  Generic Python exceptions: Simple and concise, but
> difficult
> > > for
> > > > > > > providing nuanced diagnostic or resolution guidance in logs.
> > > > > > > 2.  Custom exception hierarchy: Allows us to accumulate
> specific
> > > > > > > troubleshooting documentation for users. This also enables the
> > > > > > non-breaking
> > > > > > > compatibility TP mentioned.
> > > > > > >
> > > > > > > While I previously favored simplicity, I now lean toward a
> > > structured
> > > > > > > hierarchy. In an agentic context, we could automate the
> > maintenance
> > > > of
> > > > > a
> > > > > > > "suggestions database" by scanning past issues. This would
> > provide
> > > > > users
> > > > > > > (and agents) with clear, documented resolution steps,
> ultimately
> > > > saving
> > > > > > > time and tokens.
> > > > > > >
> > > > > > > Best,
> > > > > > > Jarek Potiuk
> > > > > > >
> > > > > > >
> > > > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > > > > > dev@airflow.apache.org> wrote:
> > > > > > >
> > > > > > > > I think we can make this non-breaking entirely? We can simply
> > > > raise a
> > > > > > > > subclass of both AirflowException and whatever we want to
> > migrate
> > > > to.
> > > > > > > >
> > > > > > > > If we want to, we can emit a deprecation warning if
> > > > AirflowException
> > > > > is
> > > > > > > > imported directly by the user, but I feel even that is
> > > particularly
> > > > > > > needed.
> > > > > > > >
> > > > > > > > Raising a custom exception that’s as specific as practical is
> > > > always
> > > > > > the
> > > > > > > > best practice. Using a custom exception that subclasses both
> > > > built-in
> > > > > > and
> > > > > > > > AirflowException (but not AirflowException itself) is a good
> > idea
> > > > on
> > > > > > its
> > > > > > > > own.
> > > > > > > >
> > > > > > > > TP
> > > > > > > >
> > > > > > > >
> > > > > > > > > On 31 May 2026, at 21:25, Jens Scheffler <
> > jscheffl@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I do not have a stron opinion on this, with the past
> decision
> > > we
> > > > > > mainly
> > > > > > > > wanted to ensure code is clean.
> > > > > > > > >
> > > > > > > > > For me also Option (3) would be viable: We leave existing
> > code
> > > as
> > > > > it
> > > > > > is
> > > > > > > > and just ensure that new code is clean. But I am also okay to
> > > join
> > > > > the
> > > > > > > > other majority opinion.
> > > > > > > > >
> > > > > > > > > Jens
> > > > > > > > >
> > > > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> Thanks for opening this thread. This was long overdue
> > > > considering
> > > > > > that
> > > > > > > > >> current migration efforts for AirflowException are more
> > > > scattered
> > > > > > > rather
> > > > > > > > >> than a consolidated pre-planned wave.
> > > > > > > > >>
> > > > > > > > >> In general, I agree with your overall approach. But there
> > are
> > > a
> > > > > few
> > > > > > > > things
> > > > > > > > >> I want to raise in response to your proposal:
> > > > > > > > >>
> > > > > > > > >> On 1)
> > > > > > > > >>
> > > > > > > > >> I think Amogh already expanded on thiss somewhat. But I
> > > believe
> > > > it
> > > > > > is
> > > > > > > > >> important to understand that AirflowException is not
> > > constrained
> > > > > to
> > > > > > > > >> the provider layer but often bubble up from Airflow
> core.For
> > > > > example
> > > > > > > > >> task_runner.py explicitly catches AirflowException as part
> > of
> > > > task
> > > > > > > > >> execution and retry handling, and there are also usages in
> > the
> > > > > hook
> > > > > > > and
> > > > > > > > >> connection layers.
> > > > > > > > >>
> > > > > > > > >> But depending on whether we scope this to core + providers
> > vs
> > > > > > > providers
> > > > > > > > >> only, the migration strategy looks quite different.  If
> the
> > > > > > intention
> > > > > > > is
> > > > > > > > >> provider-only migration, users will continue to encounter
> > > > > > > > AirflowException
> > > > > > > > >> from core Airflow components, which may make the overall
> > > > exception
> > > > > > > model
> > > > > > > > >> feel somewhat inconsistent. If the intention is
> project-wide
> > > > > > > > deprecation, I
> > > > > > > > >> think we need a broader plan than provider-level
> migrations
> > > > alone.
> > > > > > > > >>
> > > > > > > > >> Also, as Amogh already implied (he is free to correct me
> in
> > > > case I
> > > > > > > > >> misinterpreted his reservations), deprecation warnings may
> > not
> > > > be
> > > > > > > > >> worthwhile here. Unlike a deprecated API or method, this
> is
> > > not
> > > > > > > > something
> > > > > > > > >> users are actively choosing to call. In most cases,
> Airflow
> > > > itself
> > > > > > is
> > > > > > > > >> deciding which exception type to raise. I am leaning
> against
> > > > > > warnings
> > > > > > > > due
> > > > > > > > >> to the potential for log pollution and the relatively
> > limited
> > > > > value
> > > > > > > they
> > > > > > > > >> provide.
> > > > > > > > >>
> > > > > > > > >> On 2)
> > > > > > > > >>
> > > > > > > > >> I agree with your per-package migration proposal. But as
> > Amogh
> > > > > said,
> > > > > > > we
> > > > > > > > >> should ensure new exceptions inherit from AirflowException
> > to
> > > > > > prevent
> > > > > > > > >> breaking changes across the provider ecosystem. I must add
> > > that
> > > > > > there
> > > > > > > > will
> > > > > > > > >> inevitably be a period where users run multiple providers,
> > > some
> > > > > > > migrated
> > > > > > > > >> and some not. This may result in a mixture of exception
> > types
> > > > > > > appearing
> > > > > > > > in
> > > > > > > > >> logs and stack traces. While not ideal, this is probably
> > > > > unavoidable
> > > > > > > > given
> > > > > > > > >> independent provider versioning. As long as migrations are
> > > > clearly
> > > > > > > > >> documented, I do not see this as a major issue.
> > > > > > > > >>
> > > > > > > > >> My answers to your questions:
> > > > > > > > >>
> > > > > > > > >> *Should we add a deprecation warning to
> AirflowException?*
> > > > > > > > >>
> > > > > > > > >> I am +0. Not strongly opposed, but I am leaning no due to
> > the
> > > > risk
> > > > > > of
> > > > > > > > log
> > > > > > > > >> pollution and the fact that the warning would be emitted
> by
> > > > > Airflow
> > > > > > > > itself
> > > > > > > > >> rather than as a result of an explicit user action.
> > > > > > > > >>
> > > > > > > > >> *Is the per-package migration approach reasonable?*
> > > > > > > > >>
> > > > > > > > >> Yes. No blocking concerns from my side.
> > > > > > > > >>
> > > > > > > > >> *Should we define a project-wide exception taxonomy, or
> > leave
> > > > > > > exception
> > > > > > > > >> choices to individual provider teams?*
> > > > > > > > >>
> > > > > > > > >> I would leave this to individual provider teams. The
> > providers
> > > > > have
> > > > > > > very
> > > > > > > > >> different APIs, error models, and opinions around
> exception
> > > > > > handling.
> > > > > > > > >> Attempting to enforce a central taxonomy across all
> > providers
> > > > may
> > > > > be
> > > > > > > > >> impractical and may not provide much practical value over
> > what
> > > > we
> > > > > > > > currently
> > > > > > > > >> have in place.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Sameer Mesiah.
> > > > > > > > >>
> > > > > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <
> > > shahar@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi all,
> > > > > > > > >>>
> > > > > > > > >>> This follows up on the earlier lazy-consensus
> > > > > > > > >>> <
> > > > >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm86gs4mqt8hq1n26g0pp0fq5h5g0x2q9&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cdfc5e56a73b5436493ed08debfd8f869%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159135797644068%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8wtAN5hykPvP%2Fm6RnBzvSAoZ1HiUHrKHR624IIJWKFg%3D&reserved=0<https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> ><https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm86gs4mqt8hq1n26g0pp0fq5h5g0x2q9&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cdfc5e56a73b5436493ed08debfd8f869%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159135797667463%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ah4Qw8vYGk61Jzv0DqbpU2Q%2FjeSsxUiUu9m%2BgX4%2FU08%3D&reserved=0<https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>>
> > > > > > > and
> > > > > > > > >>> related discussions around deprecating AirflowException.
> > > > > > > > >>>
> > > > > > > > >>> While working on a cleanup PR for the Google Cloud Run
> > > operator
> > > > > in
> > > > > > > the
> > > > > > > > >>> Google provider (#67769), Amogh pointed out that
> replacing
> > > > > > > > AirflowException
> > > > > > > > >>> with built-in or custom exceptions is a breaking change.
> > > Users
> > > > > may
> > > > > > > > rely on
> > > > > > > > >>> AirflowException in callbacks (on_failure_callback,
> > > > > > > on_retry_callback)
> > > > > > > > or
> > > > > > > > >>> in custom wrappers that catch it explicitly.
> > > > > > > > >>>
> > > > > > > > >>> Rather than making these changes incrementally across
> > > > individual
> > > > > > PRs,
> > > > > > > > I'd
> > > > > > > > >>> like to discuss a coordinated migration strategy.
> > > > > > > > >>> Proposal
> > > > > > > > >>>
> > > > > > > > >>> *1. Emit a deprecation warning from
> > > > > **AirflowException.__init__**.*
> > > > > > > > >>>
> > > > > > > > >>> Although emitting warnings when exceptions are raised is
> > > > > uncommon,
> > > > > > it
> > > > > > > > is
> > > > > > > > >>> justified here because users may depend on
> AirflowException
> > > in
> > > > > > > > callbacks
> > > > > > > > >>> and exception handlers. This provides advance notice
> before
> > > > > > > > >>> package-specific migrations occur.
> > > > > > > > >>>
> > > > > > > > >>> *2. Migrate per package in the next breaking release of
> > that
> > > > > > > package.*
> > > > > > > > >>>
> > > > > > > > >>> Require a single migration PR per package (airflow-core
> and
> > > > each
> > > > > > > > provider),
> > > > > > > > >>> rather than spreading exception changes across multiple
> PRs
> > > and
> > > > > > > > releases.
> > > > > > > > >>> The migration would be included in that package's next
> > > breaking
> > > > > > > > release and
> > > > > > > > >>> documented accordingly.
> > > > > > > > >>> Questions
> > > > > > > > >>>
> > > > > > > > >>>    - Should we add a deprecation warning to
> > AirflowException?
> > > > > > > > >>>    - Is the per-package migration approach reasonable?
> > > > > > > > >>>    - Should we define a project-wide exception taxonomy,
> or
> > > > leave
> > > > > > > > exception
> > > > > > > > >>>    choices to individual provider teams?
> > > > > > > > >>>
> > > > > > > > >>> If there's agreement, I'll follow up with a
> lazy-consensus
> > > > > > proposal.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Shahar
> > > > > > > > >>>
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Amogh Desai <am...@apache.org>]

David - It is specificity. `AirflowException` is a catch-all exception and
whenever
it is raised, the caller has no information about what went wrong without
parsing
the message string. An exception such as `ConnectionTimeoutError` on the
other
hand tells you immediately what happened and lets you handle different
failure
modes differently in code.

You cannot write `except AirflowException` and meaningfully distinguish a
transient
network failure from a permanent config error without inspecting the message

On the contrary, builtins like ValueError, ConnectionError, TimeoutError
integrate
naturally with standard python tooling and are familiar to users who are
not deeply
involved Airflow users and to add, provider-specific exception types give
provider
maintainers a clean(er) surface to document resolution steps

The goal is not to do this overnight but to stop using it as a default when
something
more specific is appropriate and/or available.

Thanks & Regards,
Amogh Desai


On Mon, Jun 1, 2026 at 5:22 PM Blain David <da...@infrabel.be> wrote:

> Probably a stupid question from my side, but why do we want to migrate
> away from AirflowException?
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Monday, June 01, 2026 10:48
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [DISCUSS] Migrating from AirflowException to built-in /
> custom exceptions
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel,
> stuur deze e-mail als bijlage naar abuse@infrabel.be<mailto:
> abuse@infrabel.be>.
>
> - Where does the database live and who owns it? Is this a new repo, part of
> the docs, something in CLAUDE.md?
>
> Just generated docs in airflow.apache.org where the errors point at. No
> "database" per-se
>
> - What does the write path look like? Agents write during PR review, but
> what would be the trigger? any new exception type,
> any failing test, or something else?
>
> Just regular PR updates and AGENTS.md instructing to reconcile it whenever
> something about exceptions change.
>
> On Mon, Jun 1, 2026 at 7:56 AM Amogh Desai <am...@apache.org> wrote:
>
> > I like TP's dual-subclass approach. I see it as the most well-scoped and
> > the cleanest solution here and I think it
> > resolves most of the open questions. Raise a specific exception that
> > inherits from both AirflowException and the
> > target type, the existing catch based code will keep working, new code
> gets
> > specificity, no deprecation cycles or windows
> > to manage.
> >
> > On Jarek's suggestions database idea, I think that's worth a separate
> > thread. The exception hierarchy should not be designed
> > around a storage/retrieval system that does not exist yet. And the
> database
> > idea does not actually depend on a new exception
> > hierarchy either, it could work off existing error messages. Conflating
> the
> > two makes both harder to evaluate.
> >
> > Jarek, a couple of questions if you want to push the suggestions database
> > idea forward:
> >
> > - Where does the database live and who owns it? Is this a new repo, part
> of
> > the docs, something in CLAUDE.md?
> > - What does the write path look like? Agents write during PR review, but
> > what would be the trigger? any new exception type,
> > any failing test, or something else?
> >
> > Happy to ask further details and see it in a different proposal if we
> have
> > appetite at the moment.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Sun, May 31, 2026 at 10:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > Okay. That sounds great. It would be interesting to see what the
> system
> > >
> > > Well.. If others agree that is :). That is just one of the options :)
> > >
> > > On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <sm...@gmail.com>
> > > wrote:
> > >
> > > > Okay. That sounds great. It would be interesting to see what the
> system
> > > > looks like when it is live.
> > > >
> > > > Thanks,
> > > > Sameer Mesiah.
> > > >
> > > > On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > > Hi Sameer,
> > > > >
> > > > > Thanks for your thoughtful questions! Here are my thoughts on those
> > > > points:
> > > > >
> > > > > 1)  I see the value here mainly in terms of efficiency and
> > > > sustainability.
> > > > > By computing a solution once and storing it, we can save the time
> and
> > > > > tokens an LLM would otherwise use to solve the same problem over
> and
> > > > over.
> > > > > It's a nice way to be more environmentally friendly, too!
> > > > >
> > > > > 2)  Since many contributions are already agent-assisted, we can
> > address
> > > > > hallucinations by adding some guidance to AGENTS.md. Rather than
> > > relying
> > > > on
> > > > > periodic checks, we can ask agents to perform their own consistency
> > and
> > > > > hallucination checks during the PR process. This keeps a human
> > > naturally
> > > > in
> > > > > the loop during review, turning token usage into a great
> opportunity
> > > for
> > > > > automated quality control—which we ask our contributors' agents to
> > > > perform.
> > > > >
> > > > > This is actually a very interesting "twist" - because this also
> > nicely
> > > > > approaches the "assymetry" we have by having a lot more AI assisted
> > > > > contributions. We can ask - in our AGENTS.md those agents to do
> > **way**
> > > > mor
> > > > > things to make our projects better - all of them done on the local
> > > > machines
> > > > > of the contributors, with their tokens being spent.
> > > > >
> > > > > Best,
> > > > > Jarek Potiuk
> > > > >
> > > > >
> > > > > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <
> smesiah971@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jarek,
> > > > > >
> > > > > > I do have a few questions about the "suggestions database" point:
> > > > > >
> > > > > > 1) Previously, it was agreed that since LLMs work well with
> > > > unstructured
> > > > > > error messages, there was minimal benefit to an additional
> taxonomy
> > > > layer
> > > > > > on top of existing exceptions. I was just hoping you could
> > elaborate
> > > on
> > > > > the
> > > > > > marginal value of a structured data store over better exception
> > > > messages?
> > > > > >
> > > > > > 2) If we were to implement this "suggestions database", do you
> > > believe
> > > > > the
> > > > > > hallucination rate of the current agentic systems we have
> available
> > > is
> > > > > low
> > > > > > enough that it requires minimal manual intervention to the extent
> > > that
> > > > it
> > > > > > is worthwhile to introduce? If a maintainer/committer/contributor
> > has
> > > > to
> > > > > > skim through the database periodically to remove hallucinations,
> > > would
> > > > we
> > > > > > not be expending tokens in favour of negligible time savings?
> > > > > >
> > > > > > Granted, I could be overestimating the hallucination rate of
> these
> > > > > agentic
> > > > > > systems (I have never used them personally) so it might be the
> > ideal
> > > > > > solution. It's an interesting idea anyway so I would love to hear
> > > more
> > > > > > about it.
> > > > > >
> > > > > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > > > > >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Ff275xo3tjd4olsmrc8nggncs62fjnl0x&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf89431ab18864bb78cf808debfba8eae%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159005223102938%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8N88zAIQMIBsIQQYUei89ZlY30vpdbOylnl%2BUbZiFjQ%3D&reserved=0
> <https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x>
> > ).
> > > I
> > > > > see
> > > > > > > two main approaches:
> > > > > > >
> > > > > > > 1.  Generic Python exceptions: Simple and concise, but
> difficult
> > > for
> > > > > > > providing nuanced diagnostic or resolution guidance in logs.
> > > > > > > 2.  Custom exception hierarchy: Allows us to accumulate
> specific
> > > > > > > troubleshooting documentation for users. This also enables the
> > > > > > non-breaking
> > > > > > > compatibility TP mentioned.
> > > > > > >
> > > > > > > While I previously favored simplicity, I now lean toward a
> > > structured
> > > > > > > hierarchy. In an agentic context, we could automate the
> > maintenance
> > > > of
> > > > > a
> > > > > > > "suggestions database" by scanning past issues. This would
> > provide
> > > > > users
> > > > > > > (and agents) with clear, documented resolution steps,
> ultimately
> > > > saving
> > > > > > > time and tokens.
> > > > > > >
> > > > > > > Best,
> > > > > > > Jarek Potiuk
> > > > > > >
> > > > > > >
> > > > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > > > > > dev@airflow.apache.org> wrote:
> > > > > > >
> > > > > > > > I think we can make this non-breaking entirely? We can simply
> > > > raise a
> > > > > > > > subclass of both AirflowException and whatever we want to
> > migrate
> > > > to.
> > > > > > > >
> > > > > > > > If we want to, we can emit a deprecation warning if
> > > > AirflowException
> > > > > is
> > > > > > > > imported directly by the user, but I feel even that is
> > > particularly
> > > > > > > needed.
> > > > > > > >
> > > > > > > > Raising a custom exception that’s as specific as practical is
> > > > always
> > > > > > the
> > > > > > > > best practice. Using a custom exception that subclasses both
> > > > built-in
> > > > > > and
> > > > > > > > AirflowException (but not AirflowException itself) is a good
> > idea
> > > > on
> > > > > > its
> > > > > > > > own.
> > > > > > > >
> > > > > > > > TP
> > > > > > > >
> > > > > > > >
> > > > > > > > > On 31 May 2026, at 21:25, Jens Scheffler <
> > jscheffl@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I do not have a stron opinion on this, with the past
> decision
> > > we
> > > > > > mainly
> > > > > > > > wanted to ensure code is clean.
> > > > > > > > >
> > > > > > > > > For me also Option (3) would be viable: We leave existing
> > code
> > > as
> > > > > it
> > > > > > is
> > > > > > > > and just ensure that new code is clean. But I am also okay to
> > > join
> > > > > the
> > > > > > > > other majority opinion.
> > > > > > > > >
> > > > > > > > > Jens
> > > > > > > > >
> > > > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> Thanks for opening this thread. This was long overdue
> > > > considering
> > > > > > that
> > > > > > > > >> current migration efforts for AirflowException are more
> > > > scattered
> > > > > > > rather
> > > > > > > > >> than a consolidated pre-planned wave.
> > > > > > > > >>
> > > > > > > > >> In general, I agree with your overall approach. But there
> > are
> > > a
> > > > > few
> > > > > > > > things
> > > > > > > > >> I want to raise in response to your proposal:
> > > > > > > > >>
> > > > > > > > >> On 1)
> > > > > > > > >>
> > > > > > > > >> I think Amogh already expanded on thiss somewhat. But I
> > > believe
> > > > it
> > > > > > is
> > > > > > > > >> important to understand that AirflowException is not
> > > constrained
> > > > > to
> > > > > > > > >> the provider layer but often bubble up from Airflow
> core.For
> > > > > example
> > > > > > > > >> task_runner.py explicitly catches AirflowException as part
> > of
> > > > task
> > > > > > > > >> execution and retry handling, and there are also usages in
> > the
> > > > > hook
> > > > > > > and
> > > > > > > > >> connection layers.
> > > > > > > > >>
> > > > > > > > >> But depending on whether we scope this to core + providers
> > vs
> > > > > > > providers
> > > > > > > > >> only, the migration strategy looks quite different.  If
> the
> > > > > > intention
> > > > > > > is
> > > > > > > > >> provider-only migration, users will continue to encounter
> > > > > > > > AirflowException
> > > > > > > > >> from core Airflow components, which may make the overall
> > > > exception
> > > > > > > model
> > > > > > > > >> feel somewhat inconsistent. If the intention is
> project-wide
> > > > > > > > deprecation, I
> > > > > > > > >> think we need a broader plan than provider-level
> migrations
> > > > alone.
> > > > > > > > >>
> > > > > > > > >> Also, as Amogh already implied (he is free to correct me
> in
> > > > case I
> > > > > > > > >> misinterpreted his reservations), deprecation warnings may
> > not
> > > > be
> > > > > > > > >> worthwhile here. Unlike a deprecated API or method, this
> is
> > > not
> > > > > > > > something
> > > > > > > > >> users are actively choosing to call. In most cases,
> Airflow
> > > > itself
> > > > > > is
> > > > > > > > >> deciding which exception type to raise. I am leaning
> against
> > > > > > warnings
> > > > > > > > due
> > > > > > > > >> to the potential for log pollution and the relatively
> > limited
> > > > > value
> > > > > > > they
> > > > > > > > >> provide.
> > > > > > > > >>
> > > > > > > > >> On 2)
> > > > > > > > >>
> > > > > > > > >> I agree with your per-package migration proposal. But as
> > Amogh
> > > > > said,
> > > > > > > we
> > > > > > > > >> should ensure new exceptions inherit from AirflowException
> > to
> > > > > > prevent
> > > > > > > > >> breaking changes across the provider ecosystem. I must add
> > > that
> > > > > > there
> > > > > > > > will
> > > > > > > > >> inevitably be a period where users run multiple providers,
> > > some
> > > > > > > migrated
> > > > > > > > >> and some not. This may result in a mixture of exception
> > types
> > > > > > > appearing
> > > > > > > > in
> > > > > > > > >> logs and stack traces. While not ideal, this is probably
> > > > > unavoidable
> > > > > > > > given
> > > > > > > > >> independent provider versioning. As long as migrations are
> > > > clearly
> > > > > > > > >> documented, I do not see this as a major issue.
> > > > > > > > >>
> > > > > > > > >> My answers to your questions:
> > > > > > > > >>
> > > > > > > > >> *Should we add a deprecation warning to
> AirflowException?*
> > > > > > > > >>
> > > > > > > > >> I am +0. Not strongly opposed, but I am leaning no due to
> > the
> > > > risk
> > > > > > of
> > > > > > > > log
> > > > > > > > >> pollution and the fact that the warning would be emitted
> by
> > > > > Airflow
> > > > > > > > itself
> > > > > > > > >> rather than as a result of an explicit user action.
> > > > > > > > >>
> > > > > > > > >> *Is the per-package migration approach reasonable?*
> > > > > > > > >>
> > > > > > > > >> Yes. No blocking concerns from my side.
> > > > > > > > >>
> > > > > > > > >> *Should we define a project-wide exception taxonomy, or
> > leave
> > > > > > > exception
> > > > > > > > >> choices to individual provider teams?*
> > > > > > > > >>
> > > > > > > > >> I would leave this to individual provider teams. The
> > providers
> > > > > have
> > > > > > > very
> > > > > > > > >> different APIs, error models, and opinions around
> exception
> > > > > > handling.
> > > > > > > > >> Attempting to enforce a central taxonomy across all
> > providers
> > > > may
> > > > > be
> > > > > > > > >> impractical and may not provide much practical value over
> > what
> > > > we
> > > > > > > > currently
> > > > > > > > >> have in place.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Sameer Mesiah.
> > > > > > > > >>
> > > > > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <
> > > shahar@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi all,
> > > > > > > > >>>
> > > > > > > > >>> This follows up on the earlier lazy-consensus
> > > > > > > > >>> <
> > > > >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm86gs4mqt8hq1n26g0pp0fq5h5g0x2q9&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf89431ab18864bb78cf808debfba8eae%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159005223124368%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=LqVG4rOvDCEQKE%2BJ3TCzkzlum220KN%2F50jethHYsVGo%3D&reserved=0
> ><https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > > > > > > and
> > > > > > > > >>> related discussions around deprecating AirflowException.
> > > > > > > > >>>
> > > > > > > > >>> While working on a cleanup PR for the Google Cloud Run
> > > operator
> > > > > in
> > > > > > > the
> > > > > > > > >>> Google provider (#67769), Amogh pointed out that
> replacing
> > > > > > > > AirflowException
> > > > > > > > >>> with built-in or custom exceptions is a breaking change.
> > > Users
> > > > > may
> > > > > > > > rely on
> > > > > > > > >>> AirflowException in callbacks (on_failure_callback,
> > > > > > > on_retry_callback)
> > > > > > > > or
> > > > > > > > >>> in custom wrappers that catch it explicitly.
> > > > > > > > >>>
> > > > > > > > >>> Rather than making these changes incrementally across
> > > > individual
> > > > > > PRs,
> > > > > > > > I'd
> > > > > > > > >>> like to discuss a coordinated migration strategy.
> > > > > > > > >>> Proposal
> > > > > > > > >>>
> > > > > > > > >>> *1. Emit a deprecation warning from
> > > > > **AirflowException.__init__**.*
> > > > > > > > >>>
> > > > > > > > >>> Although emitting warnings when exceptions are raised is
> > > > > uncommon,
> > > > > > it
> > > > > > > > is
> > > > > > > > >>> justified here because users may depend on
> AirflowException
> > > in
> > > > > > > > callbacks
> > > > > > > > >>> and exception handlers. This provides advance notice
> before
> > > > > > > > >>> package-specific migrations occur.
> > > > > > > > >>>
> > > > > > > > >>> *2. Migrate per package in the next breaking release of
> > that
> > > > > > > package.*
> > > > > > > > >>>
> > > > > > > > >>> Require a single migration PR per package (airflow-core
> and
> > > > each
> > > > > > > > provider),
> > > > > > > > >>> rather than spreading exception changes across multiple
> PRs
> > > and
> > > > > > > > releases.
> > > > > > > > >>> The migration would be included in that package's next
> > > breaking
> > > > > > > > release and
> > > > > > > > >>> documented accordingly.
> > > > > > > > >>> Questions
> > > > > > > > >>>
> > > > > > > > >>>    - Should we add a deprecation warning to
> > AirflowException?
> > > > > > > > >>>    - Is the per-package migration approach reasonable?
> > > > > > > > >>>    - Should we define a project-wide exception taxonomy,
> or
> > > > leave
> > > > > > > > exception
> > > > > > > > >>>    choices to individual provider teams?
> > > > > > > > >>>
> > > > > > > > >>> If there's agreement, I'll follow up with a
> lazy-consensus
> > > > > > proposal.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Shahar
> > > > > > > > >>>
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Blain David <da...@infrabel.be>]

Probably a stupid question from my side, but why do we want to migrate away from AirflowException?

________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Monday, June 01, 2026 10:48
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.

- Where does the database live and who owns it? Is this a new repo, part of
the docs, something in CLAUDE.md?

Just generated docs in airflow.apache.org where the errors point at. No
"database" per-se

- What does the write path look like? Agents write during PR review, but
what would be the trigger? any new exception type,
any failing test, or something else?

Just regular PR updates and AGENTS.md instructing to reconcile it whenever
something about exceptions change.

On Mon, Jun 1, 2026 at 7:56 AM Amogh Desai <am...@apache.org> wrote:

> I like TP's dual-subclass approach. I see it as the most well-scoped and
> the cleanest solution here and I think it
> resolves most of the open questions. Raise a specific exception that
> inherits from both AirflowException and the
> target type, the existing catch based code will keep working, new code gets
> specificity, no deprecation cycles or windows
> to manage.
>
> On Jarek's suggestions database idea, I think that's worth a separate
> thread. The exception hierarchy should not be designed
> around a storage/retrieval system that does not exist yet. And the database
> idea does not actually depend on a new exception
> hierarchy either, it could work off existing error messages. Conflating the
> two makes both harder to evaluate.
>
> Jarek, a couple of questions if you want to push the suggestions database
> idea forward:
>
> - Where does the database live and who owns it? Is this a new repo, part of
> the docs, something in CLAUDE.md?
> - What does the write path look like? Agents write during PR review, but
> what would be the trigger? any new exception type,
> any failing test, or something else?
>
> Happy to ask further details and see it in a different proposal if we have
> appetite at the moment.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Sun, May 31, 2026 at 10:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > Okay. That sounds great. It would be interesting to see what the system
> >
> > Well.. If others agree that is :). That is just one of the options :)
> >
> > On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <sm...@gmail.com>
> > wrote:
> >
> > > Okay. That sounds great. It would be interesting to see what the system
> > > looks like when it is live.
> > >
> > > Thanks,
> > > Sameer Mesiah.
> > >
> > > On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hi Sameer,
> > > >
> > > > Thanks for your thoughtful questions! Here are my thoughts on those
> > > points:
> > > >
> > > > 1)  I see the value here mainly in terms of efficiency and
> > > sustainability.
> > > > By computing a solution once and storing it, we can save the time and
> > > > tokens an LLM would otherwise use to solve the same problem over and
> > > over.
> > > > It's a nice way to be more environmentally friendly, too!
> > > >
> > > > 2)  Since many contributions are already agent-assisted, we can
> address
> > > > hallucinations by adding some guidance to AGENTS.md. Rather than
> > relying
> > > on
> > > > periodic checks, we can ask agents to perform their own consistency
> and
> > > > hallucination checks during the PR process. This keeps a human
> > naturally
> > > in
> > > > the loop during review, turning token usage into a great opportunity
> > for
> > > > automated quality control—which we ask our contributors' agents to
> > > perform.
> > > >
> > > > This is actually a very interesting "twist" - because this also
> nicely
> > > > approaches the "assymetry" we have by having a lot more AI assisted
> > > > contributions. We can ask - in our AGENTS.md those agents to do
> **way**
> > > mor
> > > > things to make our projects better - all of them done on the local
> > > machines
> > > > of the contributors, with their tokens being spent.
> > > >
> > > > Best,
> > > > Jarek Potiuk
> > > >
> > > >
> > > > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <sm...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jarek,
> > > > >
> > > > > I do have a few questions about the "suggestions database" point:
> > > > >
> > > > > 1) Previously, it was agreed that since LLMs work well with
> > > unstructured
> > > > > error messages, there was minimal benefit to an additional taxonomy
> > > layer
> > > > > on top of existing exceptions. I was just hoping you could
> elaborate
> > on
> > > > the
> > > > > marginal value of a structured data store over better exception
> > > messages?
> > > > >
> > > > > 2) If we were to implement this "suggestions database", do you
> > believe
> > > > the
> > > > > hallucination rate of the current agentic systems we have available
> > is
> > > > low
> > > > > enough that it requires minimal manual intervention to the extent
> > that
> > > it
> > > > > is worthwhile to introduce? If a maintainer/committer/contributor
> has
> > > to
> > > > > skim through the database periodically to remove hallucinations,
> > would
> > > we
> > > > > not be expending tokens in favour of negligible time savings?
> > > > >
> > > > > Granted, I could be overestimating the hallucination rate of these
> > > > agentic
> > > > > systems (I have never used them personally) so it might be the
> ideal
> > > > > solution. It's an interesting idea anyway so I would love to hear
> > more
> > > > > about it.
> > > > >
> > > > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Ff275xo3tjd4olsmrc8nggncs62fjnl0x&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf89431ab18864bb78cf808debfba8eae%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159005223102938%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8N88zAIQMIBsIQQYUei89ZlY30vpdbOylnl%2BUbZiFjQ%3D&reserved=0<https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x>
> ).
> > I
> > > > see
> > > > > > two main approaches:
> > > > > >
> > > > > > 1.  Generic Python exceptions: Simple and concise, but difficult
> > for
> > > > > > providing nuanced diagnostic or resolution guidance in logs.
> > > > > > 2.  Custom exception hierarchy: Allows us to accumulate specific
> > > > > > troubleshooting documentation for users. This also enables the
> > > > > non-breaking
> > > > > > compatibility TP mentioned.
> > > > > >
> > > > > > While I previously favored simplicity, I now lean toward a
> > structured
> > > > > > hierarchy. In an agentic context, we could automate the
> maintenance
> > > of
> > > > a
> > > > > > "suggestions database" by scanning past issues. This would
> provide
> > > > users
> > > > > > (and agents) with clear, documented resolution steps, ultimately
> > > saving
> > > > > > time and tokens.
> > > > > >
> > > > > > Best,
> > > > > > Jarek Potiuk
> > > > > >
> > > > > >
> > > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > > > > dev@airflow.apache.org> wrote:
> > > > > >
> > > > > > > I think we can make this non-breaking entirely? We can simply
> > > raise a
> > > > > > > subclass of both AirflowException and whatever we want to
> migrate
> > > to.
> > > > > > >
> > > > > > > If we want to, we can emit a deprecation warning if
> > > AirflowException
> > > > is
> > > > > > > imported directly by the user, but I feel even that is
> > particularly
> > > > > > needed.
> > > > > > >
> > > > > > > Raising a custom exception that’s as specific as practical is
> > > always
> > > > > the
> > > > > > > best practice. Using a custom exception that subclasses both
> > > built-in
> > > > > and
> > > > > > > AirflowException (but not AirflowException itself) is a good
> idea
> > > on
> > > > > its
> > > > > > > own.
> > > > > > >
> > > > > > > TP
> > > > > > >
> > > > > > >
> > > > > > > > On 31 May 2026, at 21:25, Jens Scheffler <
> jscheffl@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I do not have a stron opinion on this, with the past decision
> > we
> > > > > mainly
> > > > > > > wanted to ensure code is clean.
> > > > > > > >
> > > > > > > > For me also Option (3) would be viable: We leave existing
> code
> > as
> > > > it
> > > > > is
> > > > > > > and just ensure that new code is clean. But I am also okay to
> > join
> > > > the
> > > > > > > other majority opinion.
> > > > > > > >
> > > > > > > > Jens
> > > > > > > >
> > > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> Thanks for opening this thread. This was long overdue
> > > considering
> > > > > that
> > > > > > > >> current migration efforts for AirflowException are more
> > > scattered
> > > > > > rather
> > > > > > > >> than a consolidated pre-planned wave.
> > > > > > > >>
> > > > > > > >> In general, I agree with your overall approach. But there
> are
> > a
> > > > few
> > > > > > > things
> > > > > > > >> I want to raise in response to your proposal:
> > > > > > > >>
> > > > > > > >> On 1)
> > > > > > > >>
> > > > > > > >> I think Amogh already expanded on thiss somewhat. But I
> > believe
> > > it
> > > > > is
> > > > > > > >> important to understand that AirflowException is not
> > constrained
> > > > to
> > > > > > > >> the provider layer but often bubble up from Airflow core.For
> > > > example
> > > > > > > >> task_runner.py explicitly catches AirflowException as part
> of
> > > task
> > > > > > > >> execution and retry handling, and there are also usages in
> the
> > > > hook
> > > > > > and
> > > > > > > >> connection layers.
> > > > > > > >>
> > > > > > > >> But depending on whether we scope this to core + providers
> vs
> > > > > > providers
> > > > > > > >> only, the migration strategy looks quite different.  If the
> > > > > intention
> > > > > > is
> > > > > > > >> provider-only migration, users will continue to encounter
> > > > > > > AirflowException
> > > > > > > >> from core Airflow components, which may make the overall
> > > exception
> > > > > > model
> > > > > > > >> feel somewhat inconsistent. If the intention is project-wide
> > > > > > > deprecation, I
> > > > > > > >> think we need a broader plan than provider-level migrations
> > > alone.
> > > > > > > >>
> > > > > > > >> Also, as Amogh already implied (he is free to correct me in
> > > case I
> > > > > > > >> misinterpreted his reservations), deprecation warnings may
> not
> > > be
> > > > > > > >> worthwhile here. Unlike a deprecated API or method, this is
> > not
> > > > > > > something
> > > > > > > >> users are actively choosing to call. In most cases, Airflow
> > > itself
> > > > > is
> > > > > > > >> deciding which exception type to raise. I am leaning against
> > > > > warnings
> > > > > > > due
> > > > > > > >> to the potential for log pollution and the relatively
> limited
> > > > value
> > > > > > they
> > > > > > > >> provide.
> > > > > > > >>
> > > > > > > >> On 2)
> > > > > > > >>
> > > > > > > >> I agree with your per-package migration proposal. But as
> Amogh
> > > > said,
> > > > > > we
> > > > > > > >> should ensure new exceptions inherit from AirflowException
> to
> > > > > prevent
> > > > > > > >> breaking changes across the provider ecosystem. I must add
> > that
> > > > > there
> > > > > > > will
> > > > > > > >> inevitably be a period where users run multiple providers,
> > some
> > > > > > migrated
> > > > > > > >> and some not. This may result in a mixture of exception
> types
> > > > > > appearing
> > > > > > > in
> > > > > > > >> logs and stack traces. While not ideal, this is probably
> > > > unavoidable
> > > > > > > given
> > > > > > > >> independent provider versioning. As long as migrations are
> > > clearly
> > > > > > > >> documented, I do not see this as a major issue.
> > > > > > > >>
> > > > > > > >> My answers to your questions:
> > > > > > > >>
> > > > > > > >> *Should we add a deprecation warning to  AirflowException?*
> > > > > > > >>
> > > > > > > >> I am +0. Not strongly opposed, but I am leaning no due to
> the
> > > risk
> > > > > of
> > > > > > > log
> > > > > > > >> pollution and the fact that the warning would be emitted by
> > > > Airflow
> > > > > > > itself
> > > > > > > >> rather than as a result of an explicit user action.
> > > > > > > >>
> > > > > > > >> *Is the per-package migration approach reasonable?*
> > > > > > > >>
> > > > > > > >> Yes. No blocking concerns from my side.
> > > > > > > >>
> > > > > > > >> *Should we define a project-wide exception taxonomy, or
> leave
> > > > > > exception
> > > > > > > >> choices to individual provider teams?*
> > > > > > > >>
> > > > > > > >> I would leave this to individual provider teams. The
> providers
> > > > have
> > > > > > very
> > > > > > > >> different APIs, error models, and opinions around exception
> > > > > handling.
> > > > > > > >> Attempting to enforce a central taxonomy across all
> providers
> > > may
> > > > be
> > > > > > > >> impractical and may not provide much practical value over
> what
> > > we
> > > > > > > currently
> > > > > > > >> have in place.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Sameer Mesiah.
> > > > > > > >>
> > > > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <
> > shahar@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi all,
> > > > > > > >>>
> > > > > > > >>> This follows up on the earlier lazy-consensus
> > > > > > > >>> <
> > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm86gs4mqt8hq1n26g0pp0fq5h5g0x2q9&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf89431ab18864bb78cf808debfba8eae%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159005223124368%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=LqVG4rOvDCEQKE%2BJ3TCzkzlum220KN%2F50jethHYsVGo%3D&reserved=0><https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > > > > > and
> > > > > > > >>> related discussions around deprecating AirflowException.
> > > > > > > >>>
> > > > > > > >>> While working on a cleanup PR for the Google Cloud Run
> > operator
> > > > in
> > > > > > the
> > > > > > > >>> Google provider (#67769), Amogh pointed out that replacing
> > > > > > > AirflowException
> > > > > > > >>> with built-in or custom exceptions is a breaking change.
> > Users
> > > > may
> > > > > > > rely on
> > > > > > > >>> AirflowException in callbacks (on_failure_callback,
> > > > > > on_retry_callback)
> > > > > > > or
> > > > > > > >>> in custom wrappers that catch it explicitly.
> > > > > > > >>>
> > > > > > > >>> Rather than making these changes incrementally across
> > > individual
> > > > > PRs,
> > > > > > > I'd
> > > > > > > >>> like to discuss a coordinated migration strategy.
> > > > > > > >>> Proposal
> > > > > > > >>>
> > > > > > > >>> *1. Emit a deprecation warning from
> > > > **AirflowException.__init__**.*
> > > > > > > >>>
> > > > > > > >>> Although emitting warnings when exceptions are raised is
> > > > uncommon,
> > > > > it
> > > > > > > is
> > > > > > > >>> justified here because users may depend on AirflowException
> > in
> > > > > > > callbacks
> > > > > > > >>> and exception handlers. This provides advance notice before
> > > > > > > >>> package-specific migrations occur.
> > > > > > > >>>
> > > > > > > >>> *2. Migrate per package in the next breaking release of
> that
> > > > > > package.*
> > > > > > > >>>
> > > > > > > >>> Require a single migration PR per package (airflow-core and
> > > each
> > > > > > > provider),
> > > > > > > >>> rather than spreading exception changes across multiple PRs
> > and
> > > > > > > releases.
> > > > > > > >>> The migration would be included in that package's next
> > breaking
> > > > > > > release and
> > > > > > > >>> documented accordingly.
> > > > > > > >>> Questions
> > > > > > > >>>
> > > > > > > >>>    - Should we add a deprecation warning to
> AirflowException?
> > > > > > > >>>    - Is the per-package migration approach reasonable?
> > > > > > > >>>    - Should we define a project-wide exception taxonomy, or
> > > leave
> > > > > > > exception
> > > > > > > >>>    choices to individual provider teams?
> > > > > > > >>>
> > > > > > > >>> If there's agreement, I'll follow up with a lazy-consensus
> > > > > proposal.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Shahar
> > > > > > > >>>
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Jarek Potiuk <ja...@potiuk.com>]

- Where does the database live and who owns it? Is this a new repo, part of
the docs, something in CLAUDE.md?

Just generated docs in airflow.apache.org where the errors point at. No
"database" per-se

- What does the write path look like? Agents write during PR review, but
what would be the trigger? any new exception type,
any failing test, or something else?

Just regular PR updates and AGENTS.md instructing to reconcile it whenever
something about exceptions change.

On Mon, Jun 1, 2026 at 7:56 AM Amogh Desai <am...@apache.org> wrote:

> I like TP's dual-subclass approach. I see it as the most well-scoped and
> the cleanest solution here and I think it
> resolves most of the open questions. Raise a specific exception that
> inherits from both AirflowException and the
> target type, the existing catch based code will keep working, new code gets
> specificity, no deprecation cycles or windows
> to manage.
>
> On Jarek's suggestions database idea, I think that's worth a separate
> thread. The exception hierarchy should not be designed
> around a storage/retrieval system that does not exist yet. And the database
> idea does not actually depend on a new exception
> hierarchy either, it could work off existing error messages. Conflating the
> two makes both harder to evaluate.
>
> Jarek, a couple of questions if you want to push the suggestions database
> idea forward:
>
> - Where does the database live and who owns it? Is this a new repo, part of
> the docs, something in CLAUDE.md?
> - What does the write path look like? Agents write during PR review, but
> what would be the trigger? any new exception type,
> any failing test, or something else?
>
> Happy to ask further details and see it in a different proposal if we have
> appetite at the moment.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Sun, May 31, 2026 at 10:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > Okay. That sounds great. It would be interesting to see what the system
> >
> > Well.. If others agree that is :). That is just one of the options :)
> >
> > On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <sm...@gmail.com>
> > wrote:
> >
> > > Okay. That sounds great. It would be interesting to see what the system
> > > looks like when it is live.
> > >
> > > Thanks,
> > > Sameer Mesiah.
> > >
> > > On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hi Sameer,
> > > >
> > > > Thanks for your thoughtful questions! Here are my thoughts on those
> > > points:
> > > >
> > > > 1)  I see the value here mainly in terms of efficiency and
> > > sustainability.
> > > > By computing a solution once and storing it, we can save the time and
> > > > tokens an LLM would otherwise use to solve the same problem over and
> > > over.
> > > > It's a nice way to be more environmentally friendly, too!
> > > >
> > > > 2)  Since many contributions are already agent-assisted, we can
> address
> > > > hallucinations by adding some guidance to AGENTS.md. Rather than
> > relying
> > > on
> > > > periodic checks, we can ask agents to perform their own consistency
> and
> > > > hallucination checks during the PR process. This keeps a human
> > naturally
> > > in
> > > > the loop during review, turning token usage into a great opportunity
> > for
> > > > automated quality control—which we ask our contributors' agents to
> > > perform.
> > > >
> > > > This is actually a very interesting "twist" - because this also
> nicely
> > > > approaches the "assymetry" we have by having a lot more AI assisted
> > > > contributions. We can ask - in our AGENTS.md those agents to do
> **way**
> > > mor
> > > > things to make our projects better - all of them done on the local
> > > machines
> > > > of the contributors, with their tokens being spent.
> > > >
> > > > Best,
> > > > Jarek Potiuk
> > > >
> > > >
> > > > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <sm...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jarek,
> > > > >
> > > > > I do have a few questions about the "suggestions database" point:
> > > > >
> > > > > 1) Previously, it was agreed that since LLMs work well with
> > > unstructured
> > > > > error messages, there was minimal benefit to an additional taxonomy
> > > layer
> > > > > on top of existing exceptions. I was just hoping you could
> elaborate
> > on
> > > > the
> > > > > marginal value of a structured data store over better exception
> > > messages?
> > > > >
> > > > > 2) If we were to implement this "suggestions database", do you
> > believe
> > > > the
> > > > > hallucination rate of the current agentic systems we have available
> > is
> > > > low
> > > > > enough that it requires minimal manual intervention to the extent
> > that
> > > it
> > > > > is worthwhile to introduce? If a maintainer/committer/contributor
> has
> > > to
> > > > > skim through the database periodically to remove hallucinations,
> > would
> > > we
> > > > > not be expending tokens in favour of negligible time savings?
> > > > >
> > > > > Granted, I could be overestimating the hallucination rate of these
> > > > agentic
> > > > > systems (I have never used them personally) so it might be the
> ideal
> > > > > solution. It's an interesting idea anyway so I would love to hear
> > more
> > > > > about it.
> > > > >
> > > > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > > > > https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x
> ).
> > I
> > > > see
> > > > > > two main approaches:
> > > > > >
> > > > > > 1.  Generic Python exceptions: Simple and concise, but difficult
> > for
> > > > > > providing nuanced diagnostic or resolution guidance in logs.
> > > > > > 2.  Custom exception hierarchy: Allows us to accumulate specific
> > > > > > troubleshooting documentation for users. This also enables the
> > > > > non-breaking
> > > > > > compatibility TP mentioned.
> > > > > >
> > > > > > While I previously favored simplicity, I now lean toward a
> > structured
> > > > > > hierarchy. In an agentic context, we could automate the
> maintenance
> > > of
> > > > a
> > > > > > "suggestions database" by scanning past issues. This would
> provide
> > > > users
> > > > > > (and agents) with clear, documented resolution steps, ultimately
> > > saving
> > > > > > time and tokens.
> > > > > >
> > > > > > Best,
> > > > > > Jarek Potiuk
> > > > > >
> > > > > >
> > > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > > > > dev@airflow.apache.org> wrote:
> > > > > >
> > > > > > > I think we can make this non-breaking entirely? We can simply
> > > raise a
> > > > > > > subclass of both AirflowException and whatever we want to
> migrate
> > > to.
> > > > > > >
> > > > > > > If we want to, we can emit a deprecation warning if
> > > AirflowException
> > > > is
> > > > > > > imported directly by the user, but I feel even that is
> > particularly
> > > > > > needed.
> > > > > > >
> > > > > > > Raising a custom exception that’s as specific as practical is
> > > always
> > > > > the
> > > > > > > best practice. Using a custom exception that subclasses both
> > > built-in
> > > > > and
> > > > > > > AirflowException (but not AirflowException itself) is a good
> idea
> > > on
> > > > > its
> > > > > > > own.
> > > > > > >
> > > > > > > TP
> > > > > > >
> > > > > > >
> > > > > > > > On 31 May 2026, at 21:25, Jens Scheffler <
> jscheffl@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I do not have a stron opinion on this, with the past decision
> > we
> > > > > mainly
> > > > > > > wanted to ensure code is clean.
> > > > > > > >
> > > > > > > > For me also Option (3) would be viable: We leave existing
> code
> > as
> > > > it
> > > > > is
> > > > > > > and just ensure that new code is clean. But I am also okay to
> > join
> > > > the
> > > > > > > other majority opinion.
> > > > > > > >
> > > > > > > > Jens
> > > > > > > >
> > > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> Thanks for opening this thread. This was long overdue
> > > considering
> > > > > that
> > > > > > > >> current migration efforts for AirflowException are more
> > > scattered
> > > > > > rather
> > > > > > > >> than a consolidated pre-planned wave.
> > > > > > > >>
> > > > > > > >> In general, I agree with your overall approach. But there
> are
> > a
> > > > few
> > > > > > > things
> > > > > > > >> I want to raise in response to your proposal:
> > > > > > > >>
> > > > > > > >> On 1)
> > > > > > > >>
> > > > > > > >> I think Amogh already expanded on thiss somewhat. But I
> > believe
> > > it
> > > > > is
> > > > > > > >> important to understand that AirflowException is not
> > constrained
> > > > to
> > > > > > > >> the provider layer but often bubble up from Airflow core.For
> > > > example
> > > > > > > >> task_runner.py explicitly catches AirflowException as part
> of
> > > task
> > > > > > > >> execution and retry handling, and there are also usages in
> the
> > > > hook
> > > > > > and
> > > > > > > >> connection layers.
> > > > > > > >>
> > > > > > > >> But depending on whether we scope this to core + providers
> vs
> > > > > > providers
> > > > > > > >> only, the migration strategy looks quite different.  If the
> > > > > intention
> > > > > > is
> > > > > > > >> provider-only migration, users will continue to encounter
> > > > > > > AirflowException
> > > > > > > >> from core Airflow components, which may make the overall
> > > exception
> > > > > > model
> > > > > > > >> feel somewhat inconsistent. If the intention is project-wide
> > > > > > > deprecation, I
> > > > > > > >> think we need a broader plan than provider-level migrations
> > > alone.
> > > > > > > >>
> > > > > > > >> Also, as Amogh already implied (he is free to correct me in
> > > case I
> > > > > > > >> misinterpreted his reservations), deprecation warnings may
> not
> > > be
> > > > > > > >> worthwhile here. Unlike a deprecated API or method, this is
> > not
> > > > > > > something
> > > > > > > >> users are actively choosing to call. In most cases, Airflow
> > > itself
> > > > > is
> > > > > > > >> deciding which exception type to raise. I am leaning against
> > > > > warnings
> > > > > > > due
> > > > > > > >> to the potential for log pollution and the relatively
> limited
> > > > value
> > > > > > they
> > > > > > > >> provide.
> > > > > > > >>
> > > > > > > >> On 2)
> > > > > > > >>
> > > > > > > >> I agree with your per-package migration proposal. But as
> Amogh
> > > > said,
> > > > > > we
> > > > > > > >> should ensure new exceptions inherit from AirflowException
> to
> > > > > prevent
> > > > > > > >> breaking changes across the provider ecosystem. I must add
> > that
> > > > > there
> > > > > > > will
> > > > > > > >> inevitably be a period where users run multiple providers,
> > some
> > > > > > migrated
> > > > > > > >> and some not. This may result in a mixture of exception
> types
> > > > > > appearing
> > > > > > > in
> > > > > > > >> logs and stack traces. While not ideal, this is probably
> > > > unavoidable
> > > > > > > given
> > > > > > > >> independent provider versioning. As long as migrations are
> > > clearly
> > > > > > > >> documented, I do not see this as a major issue.
> > > > > > > >>
> > > > > > > >> My answers to your questions:
> > > > > > > >>
> > > > > > > >> *Should we add a deprecation warning to  AirflowException?*
> > > > > > > >>
> > > > > > > >> I am +0. Not strongly opposed, but I am leaning no due to
> the
> > > risk
> > > > > of
> > > > > > > log
> > > > > > > >> pollution and the fact that the warning would be emitted by
> > > > Airflow
> > > > > > > itself
> > > > > > > >> rather than as a result of an explicit user action.
> > > > > > > >>
> > > > > > > >> *Is the per-package migration approach reasonable?*
> > > > > > > >>
> > > > > > > >> Yes. No blocking concerns from my side.
> > > > > > > >>
> > > > > > > >> *Should we define a project-wide exception taxonomy, or
> leave
> > > > > > exception
> > > > > > > >> choices to individual provider teams?*
> > > > > > > >>
> > > > > > > >> I would leave this to individual provider teams. The
> providers
> > > > have
> > > > > > very
> > > > > > > >> different APIs, error models, and opinions around exception
> > > > > handling.
> > > > > > > >> Attempting to enforce a central taxonomy across all
> providers
> > > may
> > > > be
> > > > > > > >> impractical and may not provide much practical value over
> what
> > > we
> > > > > > > currently
> > > > > > > >> have in place.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Sameer Mesiah.
> > > > > > > >>
> > > > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <
> > shahar@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi all,
> > > > > > > >>>
> > > > > > > >>> This follows up on the earlier lazy-consensus
> > > > > > > >>> <
> > > > https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > > > > > and
> > > > > > > >>> related discussions around deprecating AirflowException.
> > > > > > > >>>
> > > > > > > >>> While working on a cleanup PR for the Google Cloud Run
> > operator
> > > > in
> > > > > > the
> > > > > > > >>> Google provider (#67769), Amogh pointed out that replacing
> > > > > > > AirflowException
> > > > > > > >>> with built-in or custom exceptions is a breaking change.
> > Users
> > > > may
> > > > > > > rely on
> > > > > > > >>> AirflowException in callbacks (on_failure_callback,
> > > > > > on_retry_callback)
> > > > > > > or
> > > > > > > >>> in custom wrappers that catch it explicitly.
> > > > > > > >>>
> > > > > > > >>> Rather than making these changes incrementally across
> > > individual
> > > > > PRs,
> > > > > > > I'd
> > > > > > > >>> like to discuss a coordinated migration strategy.
> > > > > > > >>> Proposal
> > > > > > > >>>
> > > > > > > >>> *1. Emit a deprecation warning from
> > > > **AirflowException.__init__**.*
> > > > > > > >>>
> > > > > > > >>> Although emitting warnings when exceptions are raised is
> > > > uncommon,
> > > > > it
> > > > > > > is
> > > > > > > >>> justified here because users may depend on AirflowException
> > in
> > > > > > > callbacks
> > > > > > > >>> and exception handlers. This provides advance notice before
> > > > > > > >>> package-specific migrations occur.
> > > > > > > >>>
> > > > > > > >>> *2. Migrate per package in the next breaking release of
> that
> > > > > > package.*
> > > > > > > >>>
> > > > > > > >>> Require a single migration PR per package (airflow-core and
> > > each
> > > > > > > provider),
> > > > > > > >>> rather than spreading exception changes across multiple PRs
> > and
> > > > > > > releases.
> > > > > > > >>> The migration would be included in that package's next
> > breaking
> > > > > > > release and
> > > > > > > >>> documented accordingly.
> > > > > > > >>> Questions
> > > > > > > >>>
> > > > > > > >>>    - Should we add a deprecation warning to
> AirflowException?
> > > > > > > >>>    - Is the per-package migration approach reasonable?
> > > > > > > >>>    - Should we define a project-wide exception taxonomy, or
> > > leave
> > > > > > > exception
> > > > > > > >>>    choices to individual provider teams?
> > > > > > > >>>
> > > > > > > >>> If there's agreement, I'll follow up with a lazy-consensus
> > > > > proposal.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Shahar
> > > > > > > >>>
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Amogh Desai <am...@apache.org>]

I like TP's dual-subclass approach. I see it as the most well-scoped and
the cleanest solution here and I think it
resolves most of the open questions. Raise a specific exception that
inherits from both AirflowException and the
target type, the existing catch based code will keep working, new code gets
specificity, no deprecation cycles or windows
to manage.

On Jarek's suggestions database idea, I think that's worth a separate
thread. The exception hierarchy should not be designed
around a storage/retrieval system that does not exist yet. And the database
idea does not actually depend on a new exception
hierarchy either, it could work off existing error messages. Conflating the
two makes both harder to evaluate.

Jarek, a couple of questions if you want to push the suggestions database
idea forward:

- Where does the database live and who owns it? Is this a new repo, part of
the docs, something in CLAUDE.md?
- What does the write path look like? Agents write during PR review, but
what would be the trigger? any new exception type,
any failing test, or something else?

Happy to ask further details and see it in a different proposal if we have
appetite at the moment.

Thanks & Regards,
Amogh Desai


On Sun, May 31, 2026 at 10:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> > Okay. That sounds great. It would be interesting to see what the system
>
> Well.. If others agree that is :). That is just one of the options :)
>
> On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <sm...@gmail.com>
> wrote:
>
> > Okay. That sounds great. It would be interesting to see what the system
> > looks like when it is live.
> >
> > Thanks,
> > Sameer Mesiah.
> >
> > On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Hi Sameer,
> > >
> > > Thanks for your thoughtful questions! Here are my thoughts on those
> > points:
> > >
> > > 1)  I see the value here mainly in terms of efficiency and
> > sustainability.
> > > By computing a solution once and storing it, we can save the time and
> > > tokens an LLM would otherwise use to solve the same problem over and
> > over.
> > > It's a nice way to be more environmentally friendly, too!
> > >
> > > 2)  Since many contributions are already agent-assisted, we can address
> > > hallucinations by adding some guidance to AGENTS.md. Rather than
> relying
> > on
> > > periodic checks, we can ask agents to perform their own consistency and
> > > hallucination checks during the PR process. This keeps a human
> naturally
> > in
> > > the loop during review, turning token usage into a great opportunity
> for
> > > automated quality control—which we ask our contributors' agents to
> > perform.
> > >
> > > This is actually a very interesting "twist" - because this also nicely
> > > approaches the "assymetry" we have by having a lot more AI assisted
> > > contributions. We can ask - in our AGENTS.md those agents to do **way**
> > mor
> > > things to make our projects better - all of them done on the local
> > machines
> > > of the contributors, with their tokens being spent.
> > >
> > > Best,
> > > Jarek Potiuk
> > >
> > >
> > > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <sm...@gmail.com>
> > > wrote:
> > >
> > > > Hi Jarek,
> > > >
> > > > I do have a few questions about the "suggestions database" point:
> > > >
> > > > 1) Previously, it was agreed that since LLMs work well with
> > unstructured
> > > > error messages, there was minimal benefit to an additional taxonomy
> > layer
> > > > on top of existing exceptions. I was just hoping you could elaborate
> on
> > > the
> > > > marginal value of a structured data store over better exception
> > messages?
> > > >
> > > > 2) If we were to implement this "suggestions database", do you
> believe
> > > the
> > > > hallucination rate of the current agentic systems we have available
> is
> > > low
> > > > enough that it requires minimal manual intervention to the extent
> that
> > it
> > > > is worthwhile to introduce? If a maintainer/committer/contributor has
> > to
> > > > skim through the database periodically to remove hallucinations,
> would
> > we
> > > > not be expending tokens in favour of negligible time savings?
> > > >
> > > > Granted, I could be overestimating the hallucination rate of these
> > > agentic
> > > > systems (I have never used them personally) so it might be the ideal
> > > > solution. It's an interesting idea anyway so I would love to hear
> more
> > > > about it.
> > > >
> > > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > > > https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x).
> I
> > > see
> > > > > two main approaches:
> > > > >
> > > > > 1.  Generic Python exceptions: Simple and concise, but difficult
> for
> > > > > providing nuanced diagnostic or resolution guidance in logs.
> > > > > 2.  Custom exception hierarchy: Allows us to accumulate specific
> > > > > troubleshooting documentation for users. This also enables the
> > > > non-breaking
> > > > > compatibility TP mentioned.
> > > > >
> > > > > While I previously favored simplicity, I now lean toward a
> structured
> > > > > hierarchy. In an agentic context, we could automate the maintenance
> > of
> > > a
> > > > > "suggestions database" by scanning past issues. This would provide
> > > users
> > > > > (and agents) with clear, documented resolution steps, ultimately
> > saving
> > > > > time and tokens.
> > > > >
> > > > > Best,
> > > > > Jarek Potiuk
> > > > >
> > > > >
> > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > > > dev@airflow.apache.org> wrote:
> > > > >
> > > > > > I think we can make this non-breaking entirely? We can simply
> > raise a
> > > > > > subclass of both AirflowException and whatever we want to migrate
> > to.
> > > > > >
> > > > > > If we want to, we can emit a deprecation warning if
> > AirflowException
> > > is
> > > > > > imported directly by the user, but I feel even that is
> particularly
> > > > > needed.
> > > > > >
> > > > > > Raising a custom exception that’s as specific as practical is
> > always
> > > > the
> > > > > > best practice. Using a custom exception that subclasses both
> > built-in
> > > > and
> > > > > > AirflowException (but not AirflowException itself) is a good idea
> > on
> > > > its
> > > > > > own.
> > > > > >
> > > > > > TP
> > > > > >
> > > > > >
> > > > > > > On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I do not have a stron opinion on this, with the past decision
> we
> > > > mainly
> > > > > > wanted to ensure code is clean.
> > > > > > >
> > > > > > > For me also Option (3) would be viable: We leave existing code
> as
> > > it
> > > > is
> > > > > > and just ensure that new code is clean. But I am also okay to
> join
> > > the
> > > > > > other majority opinion.
> > > > > > >
> > > > > > > Jens
> > > > > > >
> > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> Thanks for opening this thread. This was long overdue
> > considering
> > > > that
> > > > > > >> current migration efforts for AirflowException are more
> > scattered
> > > > > rather
> > > > > > >> than a consolidated pre-planned wave.
> > > > > > >>
> > > > > > >> In general, I agree with your overall approach. But there are
> a
> > > few
> > > > > > things
> > > > > > >> I want to raise in response to your proposal:
> > > > > > >>
> > > > > > >> On 1)
> > > > > > >>
> > > > > > >> I think Amogh already expanded on thiss somewhat. But I
> believe
> > it
> > > > is
> > > > > > >> important to understand that AirflowException is not
> constrained
> > > to
> > > > > > >> the provider layer but often bubble up from Airflow core.For
> > > example
> > > > > > >> task_runner.py explicitly catches AirflowException as part of
> > task
> > > > > > >> execution and retry handling, and there are also usages in the
> > > hook
> > > > > and
> > > > > > >> connection layers.
> > > > > > >>
> > > > > > >> But depending on whether we scope this to core + providers vs
> > > > > providers
> > > > > > >> only, the migration strategy looks quite different.  If the
> > > > intention
> > > > > is
> > > > > > >> provider-only migration, users will continue to encounter
> > > > > > AirflowException
> > > > > > >> from core Airflow components, which may make the overall
> > exception
> > > > > model
> > > > > > >> feel somewhat inconsistent. If the intention is project-wide
> > > > > > deprecation, I
> > > > > > >> think we need a broader plan than provider-level migrations
> > alone.
> > > > > > >>
> > > > > > >> Also, as Amogh already implied (he is free to correct me in
> > case I
> > > > > > >> misinterpreted his reservations), deprecation warnings may not
> > be
> > > > > > >> worthwhile here. Unlike a deprecated API or method, this is
> not
> > > > > > something
> > > > > > >> users are actively choosing to call. In most cases, Airflow
> > itself
> > > > is
> > > > > > >> deciding which exception type to raise. I am leaning against
> > > > warnings
> > > > > > due
> > > > > > >> to the potential for log pollution and the relatively limited
> > > value
> > > > > they
> > > > > > >> provide.
> > > > > > >>
> > > > > > >> On 2)
> > > > > > >>
> > > > > > >> I agree with your per-package migration proposal. But as Amogh
> > > said,
> > > > > we
> > > > > > >> should ensure new exceptions inherit from AirflowException to
> > > > prevent
> > > > > > >> breaking changes across the provider ecosystem. I must add
> that
> > > > there
> > > > > > will
> > > > > > >> inevitably be a period where users run multiple providers,
> some
> > > > > migrated
> > > > > > >> and some not. This may result in a mixture of exception types
> > > > > appearing
> > > > > > in
> > > > > > >> logs and stack traces. While not ideal, this is probably
> > > unavoidable
> > > > > > given
> > > > > > >> independent provider versioning. As long as migrations are
> > clearly
> > > > > > >> documented, I do not see this as a major issue.
> > > > > > >>
> > > > > > >> My answers to your questions:
> > > > > > >>
> > > > > > >> *Should we add a deprecation warning to  AirflowException?*
> > > > > > >>
> > > > > > >> I am +0. Not strongly opposed, but I am leaning no due to the
> > risk
> > > > of
> > > > > > log
> > > > > > >> pollution and the fact that the warning would be emitted by
> > > Airflow
> > > > > > itself
> > > > > > >> rather than as a result of an explicit user action.
> > > > > > >>
> > > > > > >> *Is the per-package migration approach reasonable?*
> > > > > > >>
> > > > > > >> Yes. No blocking concerns from my side.
> > > > > > >>
> > > > > > >> *Should we define a project-wide exception taxonomy, or leave
> > > > > exception
> > > > > > >> choices to individual provider teams?*
> > > > > > >>
> > > > > > >> I would leave this to individual provider teams. The providers
> > > have
> > > > > very
> > > > > > >> different APIs, error models, and opinions around exception
> > > > handling.
> > > > > > >> Attempting to enforce a central taxonomy across all providers
> > may
> > > be
> > > > > > >> impractical and may not provide much practical value over what
> > we
> > > > > > currently
> > > > > > >> have in place.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Sameer Mesiah.
> > > > > > >>
> > > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <
> shahar@apache.org
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >>> Hi all,
> > > > > > >>>
> > > > > > >>> This follows up on the earlier lazy-consensus
> > > > > > >>> <
> > > https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > > > > and
> > > > > > >>> related discussions around deprecating AirflowException.
> > > > > > >>>
> > > > > > >>> While working on a cleanup PR for the Google Cloud Run
> operator
> > > in
> > > > > the
> > > > > > >>> Google provider (#67769), Amogh pointed out that replacing
> > > > > > AirflowException
> > > > > > >>> with built-in or custom exceptions is a breaking change.
> Users
> > > may
> > > > > > rely on
> > > > > > >>> AirflowException in callbacks (on_failure_callback,
> > > > > on_retry_callback)
> > > > > > or
> > > > > > >>> in custom wrappers that catch it explicitly.
> > > > > > >>>
> > > > > > >>> Rather than making these changes incrementally across
> > individual
> > > > PRs,
> > > > > > I'd
> > > > > > >>> like to discuss a coordinated migration strategy.
> > > > > > >>> Proposal
> > > > > > >>>
> > > > > > >>> *1. Emit a deprecation warning from
> > > **AirflowException.__init__**.*
> > > > > > >>>
> > > > > > >>> Although emitting warnings when exceptions are raised is
> > > uncommon,
> > > > it
> > > > > > is
> > > > > > >>> justified here because users may depend on AirflowException
> in
> > > > > > callbacks
> > > > > > >>> and exception handlers. This provides advance notice before
> > > > > > >>> package-specific migrations occur.
> > > > > > >>>
> > > > > > >>> *2. Migrate per package in the next breaking release of that
> > > > > package.*
> > > > > > >>>
> > > > > > >>> Require a single migration PR per package (airflow-core and
> > each
> > > > > > provider),
> > > > > > >>> rather than spreading exception changes across multiple PRs
> and
> > > > > > releases.
> > > > > > >>> The migration would be included in that package's next
> breaking
> > > > > > release and
> > > > > > >>> documented accordingly.
> > > > > > >>> Questions
> > > > > > >>>
> > > > > > >>>    - Should we add a deprecation warning to AirflowException?
> > > > > > >>>    - Is the per-package migration approach reasonable?
> > > > > > >>>    - Should we define a project-wide exception taxonomy, or
> > leave
> > > > > > exception
> > > > > > >>>    choices to individual provider teams?
> > > > > > >>>
> > > > > > >>> If there's agreement, I'll follow up with a lazy-consensus
> > > > proposal.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Shahar
> > > > > > >>>
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Jarek Potiuk <ja...@potiuk.com>]

> Okay. That sounds great. It would be interesting to see what the system

Well.. If others agree that is :). That is just one of the options :)

On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <sm...@gmail.com> wrote:

> Okay. That sounds great. It would be interesting to see what the system
> looks like when it is live.
>
> Thanks,
> Sameer Mesiah.
>
> On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hi Sameer,
> >
> > Thanks for your thoughtful questions! Here are my thoughts on those
> points:
> >
> > 1)  I see the value here mainly in terms of efficiency and
> sustainability.
> > By computing a solution once and storing it, we can save the time and
> > tokens an LLM would otherwise use to solve the same problem over and
> over.
> > It's a nice way to be more environmentally friendly, too!
> >
> > 2)  Since many contributions are already agent-assisted, we can address
> > hallucinations by adding some guidance to AGENTS.md. Rather than relying
> on
> > periodic checks, we can ask agents to perform their own consistency and
> > hallucination checks during the PR process. This keeps a human naturally
> in
> > the loop during review, turning token usage into a great opportunity for
> > automated quality control—which we ask our contributors' agents to
> perform.
> >
> > This is actually a very interesting "twist" - because this also nicely
> > approaches the "assymetry" we have by having a lot more AI assisted
> > contributions. We can ask - in our AGENTS.md those agents to do **way**
> mor
> > things to make our projects better - all of them done on the local
> machines
> > of the contributors, with their tokens being spent.
> >
> > Best,
> > Jarek Potiuk
> >
> >
> > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <sm...@gmail.com>
> > wrote:
> >
> > > Hi Jarek,
> > >
> > > I do have a few questions about the "suggestions database" point:
> > >
> > > 1) Previously, it was agreed that since LLMs work well with
> unstructured
> > > error messages, there was minimal benefit to an additional taxonomy
> layer
> > > on top of existing exceptions. I was just hoping you could elaborate on
> > the
> > > marginal value of a structured data store over better exception
> messages?
> > >
> > > 2) If we were to implement this "suggestions database", do you believe
> > the
> > > hallucination rate of the current agentic systems we have available is
> > low
> > > enough that it requires minimal manual intervention to the extent that
> it
> > > is worthwhile to introduce? If a maintainer/committer/contributor has
> to
> > > skim through the database periodically to remove hallucinations, would
> we
> > > not be expending tokens in favour of negligible time savings?
> > >
> > > Granted, I could be overestimating the hallucination rate of these
> > agentic
> > > systems (I have never used them personally) so it might be the ideal
> > > solution. It's an interesting idea anyway so I would love to hear more
> > > about it.
> > >
> > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > > https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x). I
> > see
> > > > two main approaches:
> > > >
> > > > 1.  Generic Python exceptions: Simple and concise, but difficult for
> > > > providing nuanced diagnostic or resolution guidance in logs.
> > > > 2.  Custom exception hierarchy: Allows us to accumulate specific
> > > > troubleshooting documentation for users. This also enables the
> > > non-breaking
> > > > compatibility TP mentioned.
> > > >
> > > > While I previously favored simplicity, I now lean toward a structured
> > > > hierarchy. In an agentic context, we could automate the maintenance
> of
> > a
> > > > "suggestions database" by scanning past issues. This would provide
> > users
> > > > (and agents) with clear, documented resolution steps, ultimately
> saving
> > > > time and tokens.
> > > >
> > > > Best,
> > > > Jarek Potiuk
> > > >
> > > >
> > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > > dev@airflow.apache.org> wrote:
> > > >
> > > > > I think we can make this non-breaking entirely? We can simply
> raise a
> > > > > subclass of both AirflowException and whatever we want to migrate
> to.
> > > > >
> > > > > If we want to, we can emit a deprecation warning if
> AirflowException
> > is
> > > > > imported directly by the user, but I feel even that is particularly
> > > > needed.
> > > > >
> > > > > Raising a custom exception that’s as specific as practical is
> always
> > > the
> > > > > best practice. Using a custom exception that subclasses both
> built-in
> > > and
> > > > > AirflowException (but not AirflowException itself) is a good idea
> on
> > > its
> > > > > own.
> > > > >
> > > > > TP
> > > > >
> > > > >
> > > > > > On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I do not have a stron opinion on this, with the past decision we
> > > mainly
> > > > > wanted to ensure code is clean.
> > > > > >
> > > > > > For me also Option (3) would be viable: We leave existing code as
> > it
> > > is
> > > > > and just ensure that new code is clean. But I am also okay to join
> > the
> > > > > other majority opinion.
> > > > > >
> > > > > > Jens
> > > > > >
> > > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> Thanks for opening this thread. This was long overdue
> considering
> > > that
> > > > > >> current migration efforts for AirflowException are more
> scattered
> > > > rather
> > > > > >> than a consolidated pre-planned wave.
> > > > > >>
> > > > > >> In general, I agree with your overall approach. But there are a
> > few
> > > > > things
> > > > > >> I want to raise in response to your proposal:
> > > > > >>
> > > > > >> On 1)
> > > > > >>
> > > > > >> I think Amogh already expanded on thiss somewhat. But I believe
> it
> > > is
> > > > > >> important to understand that AirflowException is not constrained
> > to
> > > > > >> the provider layer but often bubble up from Airflow core.For
> > example
> > > > > >> task_runner.py explicitly catches AirflowException as part of
> task
> > > > > >> execution and retry handling, and there are also usages in the
> > hook
> > > > and
> > > > > >> connection layers.
> > > > > >>
> > > > > >> But depending on whether we scope this to core + providers vs
> > > > providers
> > > > > >> only, the migration strategy looks quite different.  If the
> > > intention
> > > > is
> > > > > >> provider-only migration, users will continue to encounter
> > > > > AirflowException
> > > > > >> from core Airflow components, which may make the overall
> exception
> > > > model
> > > > > >> feel somewhat inconsistent. If the intention is project-wide
> > > > > deprecation, I
> > > > > >> think we need a broader plan than provider-level migrations
> alone.
> > > > > >>
> > > > > >> Also, as Amogh already implied (he is free to correct me in
> case I
> > > > > >> misinterpreted his reservations), deprecation warnings may not
> be
> > > > > >> worthwhile here. Unlike a deprecated API or method, this is not
> > > > > something
> > > > > >> users are actively choosing to call. In most cases, Airflow
> itself
> > > is
> > > > > >> deciding which exception type to raise. I am leaning against
> > > warnings
> > > > > due
> > > > > >> to the potential for log pollution and the relatively limited
> > value
> > > > they
> > > > > >> provide.
> > > > > >>
> > > > > >> On 2)
> > > > > >>
> > > > > >> I agree with your per-package migration proposal. But as Amogh
> > said,
> > > > we
> > > > > >> should ensure new exceptions inherit from AirflowException to
> > > prevent
> > > > > >> breaking changes across the provider ecosystem. I must add that
> > > there
> > > > > will
> > > > > >> inevitably be a period where users run multiple providers, some
> > > > migrated
> > > > > >> and some not. This may result in a mixture of exception types
> > > > appearing
> > > > > in
> > > > > >> logs and stack traces. While not ideal, this is probably
> > unavoidable
> > > > > given
> > > > > >> independent provider versioning. As long as migrations are
> clearly
> > > > > >> documented, I do not see this as a major issue.
> > > > > >>
> > > > > >> My answers to your questions:
> > > > > >>
> > > > > >> *Should we add a deprecation warning to  AirflowException?*
> > > > > >>
> > > > > >> I am +0. Not strongly opposed, but I am leaning no due to the
> risk
> > > of
> > > > > log
> > > > > >> pollution and the fact that the warning would be emitted by
> > Airflow
> > > > > itself
> > > > > >> rather than as a result of an explicit user action.
> > > > > >>
> > > > > >> *Is the per-package migration approach reasonable?*
> > > > > >>
> > > > > >> Yes. No blocking concerns from my side.
> > > > > >>
> > > > > >> *Should we define a project-wide exception taxonomy, or leave
> > > > exception
> > > > > >> choices to individual provider teams?*
> > > > > >>
> > > > > >> I would leave this to individual provider teams. The providers
> > have
> > > > very
> > > > > >> different APIs, error models, and opinions around exception
> > > handling.
> > > > > >> Attempting to enforce a central taxonomy across all providers
> may
> > be
> > > > > >> impractical and may not provide much practical value over what
> we
> > > > > currently
> > > > > >> have in place.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Sameer Mesiah.
> > > > > >>
> > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <shahar@apache.org
> >
> > > > wrote:
> > > > > >>
> > > > > >>> Hi all,
> > > > > >>>
> > > > > >>> This follows up on the earlier lazy-consensus
> > > > > >>> <
> > https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > > > and
> > > > > >>> related discussions around deprecating AirflowException.
> > > > > >>>
> > > > > >>> While working on a cleanup PR for the Google Cloud Run operator
> > in
> > > > the
> > > > > >>> Google provider (#67769), Amogh pointed out that replacing
> > > > > AirflowException
> > > > > >>> with built-in or custom exceptions is a breaking change. Users
> > may
> > > > > rely on
> > > > > >>> AirflowException in callbacks (on_failure_callback,
> > > > on_retry_callback)
> > > > > or
> > > > > >>> in custom wrappers that catch it explicitly.
> > > > > >>>
> > > > > >>> Rather than making these changes incrementally across
> individual
> > > PRs,
> > > > > I'd
> > > > > >>> like to discuss a coordinated migration strategy.
> > > > > >>> Proposal
> > > > > >>>
> > > > > >>> *1. Emit a deprecation warning from
> > **AirflowException.__init__**.*
> > > > > >>>
> > > > > >>> Although emitting warnings when exceptions are raised is
> > uncommon,
> > > it
> > > > > is
> > > > > >>> justified here because users may depend on AirflowException in
> > > > > callbacks
> > > > > >>> and exception handlers. This provides advance notice before
> > > > > >>> package-specific migrations occur.
> > > > > >>>
> > > > > >>> *2. Migrate per package in the next breaking release of that
> > > > package.*
> > > > > >>>
> > > > > >>> Require a single migration PR per package (airflow-core and
> each
> > > > > provider),
> > > > > >>> rather than spreading exception changes across multiple PRs and
> > > > > releases.
> > > > > >>> The migration would be included in that package's next breaking
> > > > > release and
> > > > > >>> documented accordingly.
> > > > > >>> Questions
> > > > > >>>
> > > > > >>>    - Should we add a deprecation warning to AirflowException?
> > > > > >>>    - Is the per-package migration approach reasonable?
> > > > > >>>    - Should we define a project-wide exception taxonomy, or
> leave
> > > > > exception
> > > > > >>>    choices to individual provider teams?
> > > > > >>>
> > > > > >>> If there's agreement, I'll follow up with a lazy-consensus
> > > proposal.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>>
> > > > > >>> Shahar
> > > > > >>>
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Sameer Mesiah <sm...@gmail.com>]

Okay. That sounds great. It would be interesting to see what the system
looks like when it is live.

Thanks,
Sameer Mesiah.

On Sun, 31 May 2026 at 17:15, Jarek Potiuk <ja...@potiuk.com> wrote:

> Hi Sameer,
>
> Thanks for your thoughtful questions! Here are my thoughts on those points:
>
> 1)  I see the value here mainly in terms of efficiency and sustainability.
> By computing a solution once and storing it, we can save the time and
> tokens an LLM would otherwise use to solve the same problem over and over.
> It's a nice way to be more environmentally friendly, too!
>
> 2)  Since many contributions are already agent-assisted, we can address
> hallucinations by adding some guidance to AGENTS.md. Rather than relying on
> periodic checks, we can ask agents to perform their own consistency and
> hallucination checks during the PR process. This keeps a human naturally in
> the loop during review, turning token usage into a great opportunity for
> automated quality control—which we ask our contributors' agents to perform.
>
> This is actually a very interesting "twist" - because this also nicely
> approaches the "assymetry" we have by having a lot more AI assisted
> contributions. We can ask - in our AGENTS.md those agents to do **way** mor
> things to make our projects better - all of them done on the local machines
> of the contributors, with their tokens being spent.
>
> Best,
> Jarek Potiuk
>
>
> On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <sm...@gmail.com>
> wrote:
>
> > Hi Jarek,
> >
> > I do have a few questions about the "suggestions database" point:
> >
> > 1) Previously, it was agreed that since LLMs work well with unstructured
> > error messages, there was minimal benefit to an additional taxonomy layer
> > on top of existing exceptions. I was just hoping you could elaborate on
> the
> > marginal value of a structured data store over better exception messages?
> >
> > 2) If we were to implement this "suggestions database", do you believe
> the
> > hallucination rate of the current agentic systems we have available is
> low
> > enough that it requires minimal manual intervention to the extent that it
> > is worthwhile to introduce? If a maintainer/committer/contributor has to
> > skim through the database periodically to remove hallucinations, would we
> > not be expending tokens in favour of negligible time savings?
> >
> > Granted, I could be overestimating the hallucination rate of these
> agentic
> > systems (I have never used them personally) so it might be the ideal
> > solution. It's an interesting idea anyway so I would love to hear more
> > about it.
> >
> > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Hi all,
> > >
> > > TP's suggestion aligns with Omkar’s earlier proposal (
> > > https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x). I
> see
> > > two main approaches:
> > >
> > > 1.  Generic Python exceptions: Simple and concise, but difficult for
> > > providing nuanced diagnostic or resolution guidance in logs.
> > > 2.  Custom exception hierarchy: Allows us to accumulate specific
> > > troubleshooting documentation for users. This also enables the
> > non-breaking
> > > compatibility TP mentioned.
> > >
> > > While I previously favored simplicity, I now lean toward a structured
> > > hierarchy. In an agentic context, we could automate the maintenance of
> a
> > > "suggestions database" by scanning past issues. This would provide
> users
> > > (and agents) with clear, documented resolution steps, ultimately saving
> > > time and tokens.
> > >
> > > Best,
> > > Jarek Potiuk
> > >
> > >
> > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > > dev@airflow.apache.org> wrote:
> > >
> > > > I think we can make this non-breaking entirely? We can simply raise a
> > > > subclass of both AirflowException and whatever we want to migrate to.
> > > >
> > > > If we want to, we can emit a deprecation warning if AirflowException
> is
> > > > imported directly by the user, but I feel even that is particularly
> > > needed.
> > > >
> > > > Raising a custom exception that’s as specific as practical is always
> > the
> > > > best practice. Using a custom exception that subclasses both built-in
> > and
> > > > AirflowException (but not AirflowException itself) is a good idea on
> > its
> > > > own.
> > > >
> > > > TP
> > > >
> > > >
> > > > > On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org>
> > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I do not have a stron opinion on this, with the past decision we
> > mainly
> > > > wanted to ensure code is clean.
> > > > >
> > > > > For me also Option (3) would be viable: We leave existing code as
> it
> > is
> > > > and just ensure that new code is clean. But I am also okay to join
> the
> > > > other majority opinion.
> > > > >
> > > > > Jens
> > > > >
> > > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > > >> Hi,
> > > > >>
> > > > >> Thanks for opening this thread. This was long overdue considering
> > that
> > > > >> current migration efforts for AirflowException are more scattered
> > > rather
> > > > >> than a consolidated pre-planned wave.
> > > > >>
> > > > >> In general, I agree with your overall approach. But there are a
> few
> > > > things
> > > > >> I want to raise in response to your proposal:
> > > > >>
> > > > >> On 1)
> > > > >>
> > > > >> I think Amogh already expanded on thiss somewhat. But I believe it
> > is
> > > > >> important to understand that AirflowException is not constrained
> to
> > > > >> the provider layer but often bubble up from Airflow core.For
> example
> > > > >> task_runner.py explicitly catches AirflowException as part of task
> > > > >> execution and retry handling, and there are also usages in the
> hook
> > > and
> > > > >> connection layers.
> > > > >>
> > > > >> But depending on whether we scope this to core + providers vs
> > > providers
> > > > >> only, the migration strategy looks quite different.  If the
> > intention
> > > is
> > > > >> provider-only migration, users will continue to encounter
> > > > AirflowException
> > > > >> from core Airflow components, which may make the overall exception
> > > model
> > > > >> feel somewhat inconsistent. If the intention is project-wide
> > > > deprecation, I
> > > > >> think we need a broader plan than provider-level migrations alone.
> > > > >>
> > > > >> Also, as Amogh already implied (he is free to correct me in case I
> > > > >> misinterpreted his reservations), deprecation warnings may not be
> > > > >> worthwhile here. Unlike a deprecated API or method, this is not
> > > > something
> > > > >> users are actively choosing to call. In most cases, Airflow itself
> > is
> > > > >> deciding which exception type to raise. I am leaning against
> > warnings
> > > > due
> > > > >> to the potential for log pollution and the relatively limited
> value
> > > they
> > > > >> provide.
> > > > >>
> > > > >> On 2)
> > > > >>
> > > > >> I agree with your per-package migration proposal. But as Amogh
> said,
> > > we
> > > > >> should ensure new exceptions inherit from AirflowException to
> > prevent
> > > > >> breaking changes across the provider ecosystem. I must add that
> > there
> > > > will
> > > > >> inevitably be a period where users run multiple providers, some
> > > migrated
> > > > >> and some not. This may result in a mixture of exception types
> > > appearing
> > > > in
> > > > >> logs and stack traces. While not ideal, this is probably
> unavoidable
> > > > given
> > > > >> independent provider versioning. As long as migrations are clearly
> > > > >> documented, I do not see this as a major issue.
> > > > >>
> > > > >> My answers to your questions:
> > > > >>
> > > > >> *Should we add a deprecation warning to  AirflowException?*
> > > > >>
> > > > >> I am +0. Not strongly opposed, but I am leaning no due to the risk
> > of
> > > > log
> > > > >> pollution and the fact that the warning would be emitted by
> Airflow
> > > > itself
> > > > >> rather than as a result of an explicit user action.
> > > > >>
> > > > >> *Is the per-package migration approach reasonable?*
> > > > >>
> > > > >> Yes. No blocking concerns from my side.
> > > > >>
> > > > >> *Should we define a project-wide exception taxonomy, or leave
> > > exception
> > > > >> choices to individual provider teams?*
> > > > >>
> > > > >> I would leave this to individual provider teams. The providers
> have
> > > very
> > > > >> different APIs, error models, and opinions around exception
> > handling.
> > > > >> Attempting to enforce a central taxonomy across all providers may
> be
> > > > >> impractical and may not provide much practical value over what we
> > > > currently
> > > > >> have in place.
> > > > >>
> > > > >> Thanks,
> > > > >> Sameer Mesiah.
> > > > >>
> > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org>
> > > wrote:
> > > > >>
> > > > >>> Hi all,
> > > > >>>
> > > > >>> This follows up on the earlier lazy-consensus
> > > > >>> <
> https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > > and
> > > > >>> related discussions around deprecating AirflowException.
> > > > >>>
> > > > >>> While working on a cleanup PR for the Google Cloud Run operator
> in
> > > the
> > > > >>> Google provider (#67769), Amogh pointed out that replacing
> > > > AirflowException
> > > > >>> with built-in or custom exceptions is a breaking change. Users
> may
> > > > rely on
> > > > >>> AirflowException in callbacks (on_failure_callback,
> > > on_retry_callback)
> > > > or
> > > > >>> in custom wrappers that catch it explicitly.
> > > > >>>
> > > > >>> Rather than making these changes incrementally across individual
> > PRs,
> > > > I'd
> > > > >>> like to discuss a coordinated migration strategy.
> > > > >>> Proposal
> > > > >>>
> > > > >>> *1. Emit a deprecation warning from
> **AirflowException.__init__**.*
> > > > >>>
> > > > >>> Although emitting warnings when exceptions are raised is
> uncommon,
> > it
> > > > is
> > > > >>> justified here because users may depend on AirflowException in
> > > > callbacks
> > > > >>> and exception handlers. This provides advance notice before
> > > > >>> package-specific migrations occur.
> > > > >>>
> > > > >>> *2. Migrate per package in the next breaking release of that
> > > package.*
> > > > >>>
> > > > >>> Require a single migration PR per package (airflow-core and each
> > > > provider),
> > > > >>> rather than spreading exception changes across multiple PRs and
> > > > releases.
> > > > >>> The migration would be included in that package's next breaking
> > > > release and
> > > > >>> documented accordingly.
> > > > >>> Questions
> > > > >>>
> > > > >>>    - Should we add a deprecation warning to AirflowException?
> > > > >>>    - Is the per-package migration approach reasonable?
> > > > >>>    - Should we define a project-wide exception taxonomy, or leave
> > > > exception
> > > > >>>    choices to individual provider teams?
> > > > >>>
> > > > >>> If there's agreement, I'll follow up with a lazy-consensus
> > proposal.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>>
> > > > >>> Shahar
> > > > >>>
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Jarek Potiuk <ja...@potiuk.com>]

Hi Sameer,

Thanks for your thoughtful questions! Here are my thoughts on those points:

1)  I see the value here mainly in terms of efficiency and sustainability.
By computing a solution once and storing it, we can save the time and
tokens an LLM would otherwise use to solve the same problem over and over.
It's a nice way to be more environmentally friendly, too!

2)  Since many contributions are already agent-assisted, we can address
hallucinations by adding some guidance to AGENTS.md. Rather than relying on
periodic checks, we can ask agents to perform their own consistency and
hallucination checks during the PR process. This keeps a human naturally in
the loop during review, turning token usage into a great opportunity for
automated quality control—which we ask our contributors' agents to perform.

This is actually a very interesting "twist" - because this also nicely
approaches the "assymetry" we have by having a lot more AI assisted
contributions. We can ask - in our AGENTS.md those agents to do **way** mor
things to make our projects better - all of them done on the local machines
of the contributors, with their tokens being spent.

Best,
Jarek Potiuk


On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <sm...@gmail.com> wrote:

> Hi Jarek,
>
> I do have a few questions about the "suggestions database" point:
>
> 1) Previously, it was agreed that since LLMs work well with unstructured
> error messages, there was minimal benefit to an additional taxonomy layer
> on top of existing exceptions. I was just hoping you could elaborate on the
> marginal value of a structured data store over better exception messages?
>
> 2) If we were to implement this "suggestions database", do you believe the
> hallucination rate of the current agentic systems we have available is low
> enough that it requires minimal manual intervention to the extent that it
> is worthwhile to introduce? If a maintainer/committer/contributor has to
> skim through the database periodically to remove hallucinations, would we
> not be expending tokens in favour of negligible time savings?
>
> Granted, I could be overestimating the hallucination rate of these agentic
> systems (I have never used them personally) so it might be the ideal
> solution. It's an interesting idea anyway so I would love to hear more
> about it.
>
> On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hi all,
> >
> > TP's suggestion aligns with Omkar’s earlier proposal (
> > https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x). I see
> > two main approaches:
> >
> > 1.  Generic Python exceptions: Simple and concise, but difficult for
> > providing nuanced diagnostic or resolution guidance in logs.
> > 2.  Custom exception hierarchy: Allows us to accumulate specific
> > troubleshooting documentation for users. This also enables the
> non-breaking
> > compatibility TP mentioned.
> >
> > While I previously favored simplicity, I now lean toward a structured
> > hierarchy. In an agentic context, we could automate the maintenance of a
> > "suggestions database" by scanning past issues. This would provide users
> > (and agents) with clear, documented resolution steps, ultimately saving
> > time and tokens.
> >
> > Best,
> > Jarek Potiuk
> >
> >
> > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> > dev@airflow.apache.org> wrote:
> >
> > > I think we can make this non-breaking entirely? We can simply raise a
> > > subclass of both AirflowException and whatever we want to migrate to.
> > >
> > > If we want to, we can emit a deprecation warning if AirflowException is
> > > imported directly by the user, but I feel even that is particularly
> > needed.
> > >
> > > Raising a custom exception that’s as specific as practical is always
> the
> > > best practice. Using a custom exception that subclasses both built-in
> and
> > > AirflowException (but not AirflowException itself) is a good idea on
> its
> > > own.
> > >
> > > TP
> > >
> > >
> > > > On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org>
> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I do not have a stron opinion on this, with the past decision we
> mainly
> > > wanted to ensure code is clean.
> > > >
> > > > For me also Option (3) would be viable: We leave existing code as it
> is
> > > and just ensure that new code is clean. But I am also okay to join the
> > > other majority opinion.
> > > >
> > > > Jens
> > > >
> > > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > > >> Hi,
> > > >>
> > > >> Thanks for opening this thread. This was long overdue considering
> that
> > > >> current migration efforts for AirflowException are more scattered
> > rather
> > > >> than a consolidated pre-planned wave.
> > > >>
> > > >> In general, I agree with your overall approach. But there are a few
> > > things
> > > >> I want to raise in response to your proposal:
> > > >>
> > > >> On 1)
> > > >>
> > > >> I think Amogh already expanded on thiss somewhat. But I believe it
> is
> > > >> important to understand that AirflowException is not constrained to
> > > >> the provider layer but often bubble up from Airflow core.For example
> > > >> task_runner.py explicitly catches AirflowException as part of task
> > > >> execution and retry handling, and there are also usages in the hook
> > and
> > > >> connection layers.
> > > >>
> > > >> But depending on whether we scope this to core + providers vs
> > providers
> > > >> only, the migration strategy looks quite different.  If the
> intention
> > is
> > > >> provider-only migration, users will continue to encounter
> > > AirflowException
> > > >> from core Airflow components, which may make the overall exception
> > model
> > > >> feel somewhat inconsistent. If the intention is project-wide
> > > deprecation, I
> > > >> think we need a broader plan than provider-level migrations alone.
> > > >>
> > > >> Also, as Amogh already implied (he is free to correct me in case I
> > > >> misinterpreted his reservations), deprecation warnings may not be
> > > >> worthwhile here. Unlike a deprecated API or method, this is not
> > > something
> > > >> users are actively choosing to call. In most cases, Airflow itself
> is
> > > >> deciding which exception type to raise. I am leaning against
> warnings
> > > due
> > > >> to the potential for log pollution and the relatively limited value
> > they
> > > >> provide.
> > > >>
> > > >> On 2)
> > > >>
> > > >> I agree with your per-package migration proposal. But as Amogh said,
> > we
> > > >> should ensure new exceptions inherit from AirflowException to
> prevent
> > > >> breaking changes across the provider ecosystem. I must add that
> there
> > > will
> > > >> inevitably be a period where users run multiple providers, some
> > migrated
> > > >> and some not. This may result in a mixture of exception types
> > appearing
> > > in
> > > >> logs and stack traces. While not ideal, this is probably unavoidable
> > > given
> > > >> independent provider versioning. As long as migrations are clearly
> > > >> documented, I do not see this as a major issue.
> > > >>
> > > >> My answers to your questions:
> > > >>
> > > >> *Should we add a deprecation warning to  AirflowException?*
> > > >>
> > > >> I am +0. Not strongly opposed, but I am leaning no due to the risk
> of
> > > log
> > > >> pollution and the fact that the warning would be emitted by Airflow
> > > itself
> > > >> rather than as a result of an explicit user action.
> > > >>
> > > >> *Is the per-package migration approach reasonable?*
> > > >>
> > > >> Yes. No blocking concerns from my side.
> > > >>
> > > >> *Should we define a project-wide exception taxonomy, or leave
> > exception
> > > >> choices to individual provider teams?*
> > > >>
> > > >> I would leave this to individual provider teams. The providers have
> > very
> > > >> different APIs, error models, and opinions around exception
> handling.
> > > >> Attempting to enforce a central taxonomy across all providers may be
> > > >> impractical and may not provide much practical value over what we
> > > currently
> > > >> have in place.
> > > >>
> > > >> Thanks,
> > > >> Sameer Mesiah.
> > > >>
> > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org>
> > wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> This follows up on the earlier lazy-consensus
> > > >>> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> > and
> > > >>> related discussions around deprecating AirflowException.
> > > >>>
> > > >>> While working on a cleanup PR for the Google Cloud Run operator in
> > the
> > > >>> Google provider (#67769), Amogh pointed out that replacing
> > > AirflowException
> > > >>> with built-in or custom exceptions is a breaking change. Users may
> > > rely on
> > > >>> AirflowException in callbacks (on_failure_callback,
> > on_retry_callback)
> > > or
> > > >>> in custom wrappers that catch it explicitly.
> > > >>>
> > > >>> Rather than making these changes incrementally across individual
> PRs,
> > > I'd
> > > >>> like to discuss a coordinated migration strategy.
> > > >>> Proposal
> > > >>>
> > > >>> *1. Emit a deprecation warning from **AirflowException.__init__**.*
> > > >>>
> > > >>> Although emitting warnings when exceptions are raised is uncommon,
> it
> > > is
> > > >>> justified here because users may depend on AirflowException in
> > > callbacks
> > > >>> and exception handlers. This provides advance notice before
> > > >>> package-specific migrations occur.
> > > >>>
> > > >>> *2. Migrate per package in the next breaking release of that
> > package.*
> > > >>>
> > > >>> Require a single migration PR per package (airflow-core and each
> > > provider),
> > > >>> rather than spreading exception changes across multiple PRs and
> > > releases.
> > > >>> The migration would be included in that package's next breaking
> > > release and
> > > >>> documented accordingly.
> > > >>> Questions
> > > >>>
> > > >>>    - Should we add a deprecation warning to AirflowException?
> > > >>>    - Is the per-package migration approach reasonable?
> > > >>>    - Should we define a project-wide exception taxonomy, or leave
> > > exception
> > > >>>    choices to individual provider teams?
> > > >>>
> > > >>> If there's agreement, I'll follow up with a lazy-consensus
> proposal.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>>
> > > >>> Shahar
> > > >>>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Sameer Mesiah <sm...@gmail.com>]

Hi Jarek,

I do have a few questions about the "suggestions database" point:

1) Previously, it was agreed that since LLMs work well with unstructured
error messages, there was minimal benefit to an additional taxonomy layer
on top of existing exceptions. I was just hoping you could elaborate on the
marginal value of a structured data store over better exception messages?

2) If we were to implement this "suggestions database", do you believe the
hallucination rate of the current agentic systems we have available is low
enough that it requires minimal manual intervention to the extent that it
is worthwhile to introduce? If a maintainer/committer/contributor has to
skim through the database periodically to remove hallucinations, would we
not be expending tokens in favour of negligible time savings?

Granted, I could be overestimating the hallucination rate of these agentic
systems (I have never used them personally) so it might be the ideal
solution. It's an interesting idea anyway so I would love to hear more
about it.

On Sun, 31 May 2026 at 15:37, Jarek Potiuk <ja...@potiuk.com> wrote:

> Hi all,
>
> TP's suggestion aligns with Omkar’s earlier proposal (
> https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x). I see
> two main approaches:
>
> 1.  Generic Python exceptions: Simple and concise, but difficult for
> providing nuanced diagnostic or resolution guidance in logs.
> 2.  Custom exception hierarchy: Allows us to accumulate specific
> troubleshooting documentation for users. This also enables the non-breaking
> compatibility TP mentioned.
>
> While I previously favored simplicity, I now lean toward a structured
> hierarchy. In an agentic context, we could automate the maintenance of a
> "suggestions database" by scanning past issues. This would provide users
> (and agents) with clear, documented resolution steps, ultimately saving
> time and tokens.
>
> Best,
> Jarek Potiuk
>
>
> On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
> dev@airflow.apache.org> wrote:
>
> > I think we can make this non-breaking entirely? We can simply raise a
> > subclass of both AirflowException and whatever we want to migrate to.
> >
> > If we want to, we can emit a deprecation warning if AirflowException is
> > imported directly by the user, but I feel even that is particularly
> needed.
> >
> > Raising a custom exception that’s as specific as practical is always the
> > best practice. Using a custom exception that subclasses both built-in and
> > AirflowException (but not AirflowException itself) is a good idea on its
> > own.
> >
> > TP
> >
> >
> > > On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org> wrote:
> > >
> > > Hi,
> > >
> > > I do not have a stron opinion on this, with the past decision we mainly
> > wanted to ensure code is clean.
> > >
> > > For me also Option (3) would be viable: We leave existing code as it is
> > and just ensure that new code is clean. But I am also okay to join the
> > other majority opinion.
> > >
> > > Jens
> > >
> > > On 31.05.26 14:08, Sameer Mesiah wrote:
> > >> Hi,
> > >>
> > >> Thanks for opening this thread. This was long overdue considering that
> > >> current migration efforts for AirflowException are more scattered
> rather
> > >> than a consolidated pre-planned wave.
> > >>
> > >> In general, I agree with your overall approach. But there are a few
> > things
> > >> I want to raise in response to your proposal:
> > >>
> > >> On 1)
> > >>
> > >> I think Amogh already expanded on thiss somewhat. But I believe it is
> > >> important to understand that AirflowException is not constrained to
> > >> the provider layer but often bubble up from Airflow core.For example
> > >> task_runner.py explicitly catches AirflowException as part of task
> > >> execution and retry handling, and there are also usages in the hook
> and
> > >> connection layers.
> > >>
> > >> But depending on whether we scope this to core + providers vs
> providers
> > >> only, the migration strategy looks quite different.  If the intention
> is
> > >> provider-only migration, users will continue to encounter
> > AirflowException
> > >> from core Airflow components, which may make the overall exception
> model
> > >> feel somewhat inconsistent. If the intention is project-wide
> > deprecation, I
> > >> think we need a broader plan than provider-level migrations alone.
> > >>
> > >> Also, as Amogh already implied (he is free to correct me in case I
> > >> misinterpreted his reservations), deprecation warnings may not be
> > >> worthwhile here. Unlike a deprecated API or method, this is not
> > something
> > >> users are actively choosing to call. In most cases, Airflow itself is
> > >> deciding which exception type to raise. I am leaning against warnings
> > due
> > >> to the potential for log pollution and the relatively limited value
> they
> > >> provide.
> > >>
> > >> On 2)
> > >>
> > >> I agree with your per-package migration proposal. But as Amogh said,
> we
> > >> should ensure new exceptions inherit from AirflowException to prevent
> > >> breaking changes across the provider ecosystem. I must add that there
> > will
> > >> inevitably be a period where users run multiple providers, some
> migrated
> > >> and some not. This may result in a mixture of exception types
> appearing
> > in
> > >> logs and stack traces. While not ideal, this is probably unavoidable
> > given
> > >> independent provider versioning. As long as migrations are clearly
> > >> documented, I do not see this as a major issue.
> > >>
> > >> My answers to your questions:
> > >>
> > >> *Should we add a deprecation warning to  AirflowException?*
> > >>
> > >> I am +0. Not strongly opposed, but I am leaning no due to the risk of
> > log
> > >> pollution and the fact that the warning would be emitted by Airflow
> > itself
> > >> rather than as a result of an explicit user action.
> > >>
> > >> *Is the per-package migration approach reasonable?*
> > >>
> > >> Yes. No blocking concerns from my side.
> > >>
> > >> *Should we define a project-wide exception taxonomy, or leave
> exception
> > >> choices to individual provider teams?*
> > >>
> > >> I would leave this to individual provider teams. The providers have
> very
> > >> different APIs, error models, and opinions around exception handling.
> > >> Attempting to enforce a central taxonomy across all providers may be
> > >> impractical and may not provide much practical value over what we
> > currently
> > >> have in place.
> > >>
> > >> Thanks,
> > >> Sameer Mesiah.
> > >>
> > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org>
> wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> This follows up on the earlier lazy-consensus
> > >>> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9>
> and
> > >>> related discussions around deprecating AirflowException.
> > >>>
> > >>> While working on a cleanup PR for the Google Cloud Run operator in
> the
> > >>> Google provider (#67769), Amogh pointed out that replacing
> > AirflowException
> > >>> with built-in or custom exceptions is a breaking change. Users may
> > rely on
> > >>> AirflowException in callbacks (on_failure_callback,
> on_retry_callback)
> > or
> > >>> in custom wrappers that catch it explicitly.
> > >>>
> > >>> Rather than making these changes incrementally across individual PRs,
> > I'd
> > >>> like to discuss a coordinated migration strategy.
> > >>> Proposal
> > >>>
> > >>> *1. Emit a deprecation warning from **AirflowException.__init__**.*
> > >>>
> > >>> Although emitting warnings when exceptions are raised is uncommon, it
> > is
> > >>> justified here because users may depend on AirflowException in
> > callbacks
> > >>> and exception handlers. This provides advance notice before
> > >>> package-specific migrations occur.
> > >>>
> > >>> *2. Migrate per package in the next breaking release of that
> package.*
> > >>>
> > >>> Require a single migration PR per package (airflow-core and each
> > provider),
> > >>> rather than spreading exception changes across multiple PRs and
> > releases.
> > >>> The migration would be included in that package's next breaking
> > release and
> > >>> documented accordingly.
> > >>> Questions
> > >>>
> > >>>    - Should we add a deprecation warning to AirflowException?
> > >>>    - Is the per-package migration approach reasonable?
> > >>>    - Should we define a project-wide exception taxonomy, or leave
> > exception
> > >>>    choices to individual provider teams?
> > >>>
> > >>> If there's agreement, I'll follow up with a lazy-consensus proposal.
> > >>>
> > >>> Thanks,
> > >>>
> > >>>
> > >>> Shahar
> > >>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Jarek Potiuk <ja...@potiuk.com>]

Hi all,

TP's suggestion aligns with Omkar’s earlier proposal (
https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x). I see
two main approaches:

1.  Generic Python exceptions: Simple and concise, but difficult for
providing nuanced diagnostic or resolution guidance in logs.
2.  Custom exception hierarchy: Allows us to accumulate specific
troubleshooting documentation for users. This also enables the non-breaking
compatibility TP mentioned.

While I previously favored simplicity, I now lean toward a structured
hierarchy. In an agentic context, we could automate the maintenance of a
"suggestions database" by scanning past issues. This would provide users
(and agents) with clear, documented resolution steps, ultimately saving
time and tokens.

Best,
Jarek Potiuk


On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev <
dev@airflow.apache.org> wrote:

> I think we can make this non-breaking entirely? We can simply raise a
> subclass of both AirflowException and whatever we want to migrate to.
>
> If we want to, we can emit a deprecation warning if AirflowException is
> imported directly by the user, but I feel even that is particularly needed.
>
> Raising a custom exception that’s as specific as practical is always the
> best practice. Using a custom exception that subclasses both built-in and
> AirflowException (but not AirflowException itself) is a good idea on its
> own.
>
> TP
>
>
> > On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org> wrote:
> >
> > Hi,
> >
> > I do not have a stron opinion on this, with the past decision we mainly
> wanted to ensure code is clean.
> >
> > For me also Option (3) would be viable: We leave existing code as it is
> and just ensure that new code is clean. But I am also okay to join the
> other majority opinion.
> >
> > Jens
> >
> > On 31.05.26 14:08, Sameer Mesiah wrote:
> >> Hi,
> >>
> >> Thanks for opening this thread. This was long overdue considering that
> >> current migration efforts for AirflowException are more scattered rather
> >> than a consolidated pre-planned wave.
> >>
> >> In general, I agree with your overall approach. But there are a few
> things
> >> I want to raise in response to your proposal:
> >>
> >> On 1)
> >>
> >> I think Amogh already expanded on thiss somewhat. But I believe it is
> >> important to understand that AirflowException is not constrained to
> >> the provider layer but often bubble up from Airflow core.For example
> >> task_runner.py explicitly catches AirflowException as part of task
> >> execution and retry handling, and there are also usages in the hook and
> >> connection layers.
> >>
> >> But depending on whether we scope this to core + providers vs providers
> >> only, the migration strategy looks quite different.  If the intention is
> >> provider-only migration, users will continue to encounter
> AirflowException
> >> from core Airflow components, which may make the overall exception model
> >> feel somewhat inconsistent. If the intention is project-wide
> deprecation, I
> >> think we need a broader plan than provider-level migrations alone.
> >>
> >> Also, as Amogh already implied (he is free to correct me in case I
> >> misinterpreted his reservations), deprecation warnings may not be
> >> worthwhile here. Unlike a deprecated API or method, this is not
> something
> >> users are actively choosing to call. In most cases, Airflow itself is
> >> deciding which exception type to raise. I am leaning against warnings
> due
> >> to the potential for log pollution and the relatively limited value they
> >> provide.
> >>
> >> On 2)
> >>
> >> I agree with your per-package migration proposal. But as Amogh said, we
> >> should ensure new exceptions inherit from AirflowException to prevent
> >> breaking changes across the provider ecosystem. I must add that there
> will
> >> inevitably be a period where users run multiple providers, some migrated
> >> and some not. This may result in a mixture of exception types appearing
> in
> >> logs and stack traces. While not ideal, this is probably unavoidable
> given
> >> independent provider versioning. As long as migrations are clearly
> >> documented, I do not see this as a major issue.
> >>
> >> My answers to your questions:
> >>
> >> *Should we add a deprecation warning to  AirflowException?*
> >>
> >> I am +0. Not strongly opposed, but I am leaning no due to the risk of
> log
> >> pollution and the fact that the warning would be emitted by Airflow
> itself
> >> rather than as a result of an explicit user action.
> >>
> >> *Is the per-package migration approach reasonable?*
> >>
> >> Yes. No blocking concerns from my side.
> >>
> >> *Should we define a project-wide exception taxonomy, or leave exception
> >> choices to individual provider teams?*
> >>
> >> I would leave this to individual provider teams. The providers have very
> >> different APIs, error models, and opinions around exception handling.
> >> Attempting to enforce a central taxonomy across all providers may be
> >> impractical and may not provide much practical value over what we
> currently
> >> have in place.
> >>
> >> Thanks,
> >> Sameer Mesiah.
> >>
> >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org> wrote:
> >>
> >>> Hi all,
> >>>
> >>> This follows up on the earlier lazy-consensus
> >>> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> and
> >>> related discussions around deprecating AirflowException.
> >>>
> >>> While working on a cleanup PR for the Google Cloud Run operator in the
> >>> Google provider (#67769), Amogh pointed out that replacing
> AirflowException
> >>> with built-in or custom exceptions is a breaking change. Users may
> rely on
> >>> AirflowException in callbacks (on_failure_callback, on_retry_callback)
> or
> >>> in custom wrappers that catch it explicitly.
> >>>
> >>> Rather than making these changes incrementally across individual PRs,
> I'd
> >>> like to discuss a coordinated migration strategy.
> >>> Proposal
> >>>
> >>> *1. Emit a deprecation warning from **AirflowException.__init__**.*
> >>>
> >>> Although emitting warnings when exceptions are raised is uncommon, it
> is
> >>> justified here because users may depend on AirflowException in
> callbacks
> >>> and exception handlers. This provides advance notice before
> >>> package-specific migrations occur.
> >>>
> >>> *2. Migrate per package in the next breaking release of that package.*
> >>>
> >>> Require a single migration PR per package (airflow-core and each
> provider),
> >>> rather than spreading exception changes across multiple PRs and
> releases.
> >>> The migration would be included in that package's next breaking
> release and
> >>> documented accordingly.
> >>> Questions
> >>>
> >>>    - Should we add a deprecation warning to AirflowException?
> >>>    - Is the per-package migration approach reasonable?
> >>>    - Should we define a project-wide exception taxonomy, or leave
> exception
> >>>    choices to individual provider teams?
> >>>
> >>> If there's agreement, I'll follow up with a lazy-consensus proposal.
> >>>
> >>> Thanks,
> >>>
> >>>
> >>> Shahar
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Tzu-ping Chung via dev <de...@airflow.apache.org>]

I think we can make this non-breaking entirely? We can simply raise a subclass of both AirflowException and whatever we want to migrate to.

If we want to, we can emit a deprecation warning if AirflowException is imported directly by the user, but I feel even that is particularly needed.

Raising a custom exception that’s as specific as practical is always the best practice. Using a custom exception that subclasses both built-in and AirflowException (but not AirflowException itself) is a good idea on its own.

TP


> On 31 May 2026, at 21:25, Jens Scheffler <js...@apache.org> wrote:
> 
> Hi,
> 
> I do not have a stron opinion on this, with the past decision we mainly wanted to ensure code is clean.
> 
> For me also Option (3) would be viable: We leave existing code as it is and just ensure that new code is clean. But I am also okay to join the other majority opinion.
> 
> Jens
> 
> On 31.05.26 14:08, Sameer Mesiah wrote:
>> Hi,
>> 
>> Thanks for opening this thread. This was long overdue considering that
>> current migration efforts for AirflowException are more scattered rather
>> than a consolidated pre-planned wave.
>> 
>> In general, I agree with your overall approach. But there are a few things
>> I want to raise in response to your proposal:
>> 
>> On 1)
>> 
>> I think Amogh already expanded on thiss somewhat. But I believe it is
>> important to understand that AirflowException is not constrained to
>> the provider layer but often bubble up from Airflow core.For example
>> task_runner.py explicitly catches AirflowException as part of task
>> execution and retry handling, and there are also usages in the hook and
>> connection layers.
>> 
>> But depending on whether we scope this to core + providers vs providers
>> only, the migration strategy looks quite different.  If the intention is
>> provider-only migration, users will continue to encounter  AirflowException
>> from core Airflow components, which may make the overall exception model
>> feel somewhat inconsistent. If the intention is project-wide deprecation, I
>> think we need a broader plan than provider-level migrations alone.
>> 
>> Also, as Amogh already implied (he is free to correct me in case I
>> misinterpreted his reservations), deprecation warnings may not be
>> worthwhile here. Unlike a deprecated API or method, this is not something
>> users are actively choosing to call. In most cases, Airflow itself is
>> deciding which exception type to raise. I am leaning against warnings due
>> to the potential for log pollution and the relatively limited value they
>> provide.
>> 
>> On 2)
>> 
>> I agree with your per-package migration proposal. But as Amogh said, we
>> should ensure new exceptions inherit from AirflowException to prevent
>> breaking changes across the provider ecosystem. I must add that there will
>> inevitably be a period where users run multiple providers, some migrated
>> and some not. This may result in a mixture of exception types appearing in
>> logs and stack traces. While not ideal, this is probably unavoidable given
>> independent provider versioning. As long as migrations are clearly
>> documented, I do not see this as a major issue.
>> 
>> My answers to your questions:
>> 
>> *Should we add a deprecation warning to  AirflowException?*
>> 
>> I am +0. Not strongly opposed, but I am leaning no due to the risk of log
>> pollution and the fact that the warning would be emitted by Airflow itself
>> rather than as a result of an explicit user action.
>> 
>> *Is the per-package migration approach reasonable?*
>> 
>> Yes. No blocking concerns from my side.
>> 
>> *Should we define a project-wide exception taxonomy, or leave exception
>> choices to individual provider teams?*
>> 
>> I would leave this to individual provider teams. The providers have very
>> different APIs, error models, and opinions around exception handling.
>> Attempting to enforce a central taxonomy across all providers may be
>> impractical and may not provide much practical value over what we currently
>> have in place.
>> 
>> Thanks,
>> Sameer Mesiah.
>> 
>> On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org> wrote:
>> 
>>> Hi all,
>>> 
>>> This follows up on the earlier lazy-consensus
>>> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> and
>>> related discussions around deprecating AirflowException.
>>> 
>>> While working on a cleanup PR for the Google Cloud Run operator in the
>>> Google provider (#67769), Amogh pointed out that replacing AirflowException
>>> with built-in or custom exceptions is a breaking change. Users may rely on
>>> AirflowException in callbacks (on_failure_callback, on_retry_callback) or
>>> in custom wrappers that catch it explicitly.
>>> 
>>> Rather than making these changes incrementally across individual PRs, I'd
>>> like to discuss a coordinated migration strategy.
>>> Proposal
>>> 
>>> *1. Emit a deprecation warning from **AirflowException.__init__**.*
>>> 
>>> Although emitting warnings when exceptions are raised is uncommon, it is
>>> justified here because users may depend on AirflowException in callbacks
>>> and exception handlers. This provides advance notice before
>>> package-specific migrations occur.
>>> 
>>> *2. Migrate per package in the next breaking release of that package.*
>>> 
>>> Require a single migration PR per package (airflow-core and each provider),
>>> rather than spreading exception changes across multiple PRs and releases.
>>> The migration would be included in that package's next breaking release and
>>> documented accordingly.
>>> Questions
>>> 
>>>    - Should we add a deprecation warning to AirflowException?
>>>    - Is the per-package migration approach reasonable?
>>>    - Should we define a project-wide exception taxonomy, or leave exception
>>>    choices to individual provider teams?
>>> 
>>> If there's agreement, I'll follow up with a lazy-consensus proposal.
>>> 
>>> Thanks,
>>> 
>>> 
>>> Shahar
>>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Jens Scheffler <js...@apache.org>]

Hi,

I do not have a stron opinion on this, with the past decision we mainly 
wanted to ensure code is clean.

For me also Option (3) would be viable: We leave existing code as it is 
and just ensure that new code is clean. But I am also okay to join the 
other majority opinion.

Jens

On 31.05.26 14:08, Sameer Mesiah wrote:
> Hi,
>
> Thanks for opening this thread. This was long overdue considering that
> current migration efforts for AirflowException are more scattered rather
> than a consolidated pre-planned wave.
>
> In general, I agree with your overall approach. But there are a few things
> I want to raise in response to your proposal:
>
> On 1)
>
> I think Amogh already expanded on thiss somewhat. But I believe it is
> important to understand that AirflowException is not constrained to
> the provider layer but often bubble up from Airflow core.For example
> task_runner.py explicitly catches AirflowException as part of task
> execution and retry handling, and there are also usages in the hook and
> connection layers.
>
> But depending on whether we scope this to core + providers vs providers
> only, the migration strategy looks quite different.  If the intention is
> provider-only migration, users will continue to encounter  AirflowException
> from core Airflow components, which may make the overall exception model
> feel somewhat inconsistent. If the intention is project-wide deprecation, I
> think we need a broader plan than provider-level migrations alone.
>
> Also, as Amogh already implied (he is free to correct me in case I
> misinterpreted his reservations), deprecation warnings may not be
> worthwhile here. Unlike a deprecated API or method, this is not something
> users are actively choosing to call. In most cases, Airflow itself is
> deciding which exception type to raise. I am leaning against warnings due
> to the potential for log pollution and the relatively limited value they
> provide.
>
> On 2)
>
> I agree with your per-package migration proposal. But as Amogh said, we
> should ensure new exceptions inherit from AirflowException to prevent
> breaking changes across the provider ecosystem. I must add that there will
> inevitably be a period where users run multiple providers, some migrated
> and some not. This may result in a mixture of exception types appearing in
> logs and stack traces. While not ideal, this is probably unavoidable given
> independent provider versioning. As long as migrations are clearly
> documented, I do not see this as a major issue.
>
> My answers to your questions:
>
> *Should we add a deprecation warning to  AirflowException?*
>
> I am +0. Not strongly opposed, but I am leaning no due to the risk of log
> pollution and the fact that the warning would be emitted by Airflow itself
> rather than as a result of an explicit user action.
>
> *Is the per-package migration approach reasonable?*
>
> Yes. No blocking concerns from my side.
>
> *Should we define a project-wide exception taxonomy, or leave exception
> choices to individual provider teams?*
>
> I would leave this to individual provider teams. The providers have very
> different APIs, error models, and opinions around exception handling.
> Attempting to enforce a central taxonomy across all providers may be
> impractical and may not provide much practical value over what we currently
> have in place.
>
> Thanks,
> Sameer Mesiah.
>
> On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org> wrote:
>
>> Hi all,
>>
>> This follows up on the earlier lazy-consensus
>> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> and
>> related discussions around deprecating AirflowException.
>>
>> While working on a cleanup PR for the Google Cloud Run operator in the
>> Google provider (#67769), Amogh pointed out that replacing AirflowException
>> with built-in or custom exceptions is a breaking change. Users may rely on
>> AirflowException in callbacks (on_failure_callback, on_retry_callback) or
>> in custom wrappers that catch it explicitly.
>>
>> Rather than making these changes incrementally across individual PRs, I'd
>> like to discuss a coordinated migration strategy.
>> Proposal
>>
>> *1. Emit a deprecation warning from **AirflowException.__init__**.*
>>
>> Although emitting warnings when exceptions are raised is uncommon, it is
>> justified here because users may depend on AirflowException in callbacks
>> and exception handlers. This provides advance notice before
>> package-specific migrations occur.
>>
>> *2. Migrate per package in the next breaking release of that package.*
>>
>> Require a single migration PR per package (airflow-core and each provider),
>> rather than spreading exception changes across multiple PRs and releases.
>> The migration would be included in that package's next breaking release and
>> documented accordingly.
>> Questions
>>
>>     - Should we add a deprecation warning to AirflowException?
>>     - Is the per-package migration approach reasonable?
>>     - Should we define a project-wide exception taxonomy, or leave exception
>>     choices to individual provider teams?
>>
>> If there's agreement, I'll follow up with a lazy-consensus proposal.
>>
>> Thanks,
>>
>>
>> Shahar
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Sameer Mesiah <sm...@gmail.com>]

Hi,

Thanks for opening this thread. This was long overdue considering that
current migration efforts for AirflowException are more scattered rather
than a consolidated pre-planned wave.

In general, I agree with your overall approach. But there are a few things
I want to raise in response to your proposal:

On 1)

I think Amogh already expanded on thiss somewhat. But I believe it is
important to understand that AirflowException is not constrained to
the provider layer but often bubble up from Airflow core.For example
task_runner.py explicitly catches AirflowException as part of task
execution and retry handling, and there are also usages in the hook and
connection layers.

But depending on whether we scope this to core + providers vs providers
only, the migration strategy looks quite different.  If the intention is
provider-only migration, users will continue to encounter  AirflowException
from core Airflow components, which may make the overall exception model
feel somewhat inconsistent. If the intention is project-wide deprecation, I
think we need a broader plan than provider-level migrations alone.

Also, as Amogh already implied (he is free to correct me in case I
misinterpreted his reservations), deprecation warnings may not be
worthwhile here. Unlike a deprecated API or method, this is not something
users are actively choosing to call. In most cases, Airflow itself is
deciding which exception type to raise. I am leaning against warnings due
to the potential for log pollution and the relatively limited value they
provide.

On 2)

I agree with your per-package migration proposal. But as Amogh said, we
should ensure new exceptions inherit from AirflowException to prevent
breaking changes across the provider ecosystem. I must add that there will
inevitably be a period where users run multiple providers, some migrated
and some not. This may result in a mixture of exception types appearing in
logs and stack traces. While not ideal, this is probably unavoidable given
independent provider versioning. As long as migrations are clearly
documented, I do not see this as a major issue.

My answers to your questions:

*Should we add a deprecation warning to  AirflowException?*

I am +0. Not strongly opposed, but I am leaning no due to the risk of log
pollution and the fact that the warning would be emitted by Airflow itself
rather than as a result of an explicit user action.

*Is the per-package migration approach reasonable?*

Yes. No blocking concerns from my side.

*Should we define a project-wide exception taxonomy, or leave exception
choices to individual provider teams?*

I would leave this to individual provider teams. The providers have very
different APIs, error models, and opinions around exception handling.
Attempting to enforce a central taxonomy across all providers may be
impractical and may not provide much practical value over what we currently
have in place.

Thanks,
Sameer Mesiah.

On Sat, 30 May 2026 at 10:59, Shahar Epstein <sh...@apache.org> wrote:

> Hi all,
>
> This follows up on the earlier lazy-consensus
> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> and
> related discussions around deprecating AirflowException.
>
> While working on a cleanup PR for the Google Cloud Run operator in the
> Google provider (#67769), Amogh pointed out that replacing AirflowException
> with built-in or custom exceptions is a breaking change. Users may rely on
> AirflowException in callbacks (on_failure_callback, on_retry_callback) or
> in custom wrappers that catch it explicitly.
>
> Rather than making these changes incrementally across individual PRs, I'd
> like to discuss a coordinated migration strategy.
> Proposal
>
> *1. Emit a deprecation warning from **AirflowException.__init__**.*
>
> Although emitting warnings when exceptions are raised is uncommon, it is
> justified here because users may depend on AirflowException in callbacks
> and exception handlers. This provides advance notice before
> package-specific migrations occur.
>
> *2. Migrate per package in the next breaking release of that package.*
>
> Require a single migration PR per package (airflow-core and each provider),
> rather than spreading exception changes across multiple PRs and releases.
> The migration would be included in that package's next breaking release and
> documented accordingly.
> Questions
>
>    - Should we add a deprecation warning to AirflowException?
>    - Is the per-package migration approach reasonable?
>    - Should we define a project-wide exception taxonomy, or leave exception
>    choices to individual provider teams?
>
> If there's agreement, I'll follow up with a lazy-consensus proposal.
>
> Thanks,
>
>
> Shahar
>

Re: [DISCUSS] Migrating from AirflowException to built-in / custom exceptions

[Amogh Desai <am...@apache.org>]

Hi Shahar,

Thanks for starting this discussion.

I think that the second proposal is the right call. Per-package migration
bundled into a breaking release is clean, the change is reviewable in one
PR, and providers being independently versioned means we can move at
different cadences without blocking core Airflow releases.

One thing to make note of here is that the new provider specific exceptions
should inherit from AirflowException during the migration window. Without
that,
packages that have already migrated silently break users on older catch
clauses.
With inheritance, the migration is incremental and safe across packages.

I do not think that the first proposal would hold up. The
DeprecationWarning fires
at the raise site (inside provider code), not at the catch site (ie: user
callbacks,
execute() methods). Users who `except AirflowException` never really see
it. It also
floods logs with warnings from every internal exception raise users do not
/ should not
care about.

Also worth noting that the breaking surface is wider than just callbacks.
Users catch
AirflowException inside execute() and @task functions too. And
AirflowSkipException, AirflowRescheduleException etc. are protocol
exceptions that the task runner catches
to drive state transitions. Those need their own explicit deprecation
cycle, separate from
the general provider exception migration.

Thanks & Regards,
Amogh Desai


On Sat, May 30, 2026 at 3:28 PM Shahar Epstein <sh...@apache.org> wrote:

> Hi all,
>
> This follows up on the earlier lazy-consensus
> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> and
> related discussions around deprecating AirflowException.
>
> While working on a cleanup PR for the Google Cloud Run operator in the
> Google provider (#67769), Amogh pointed out that replacing AirflowException
> with built-in or custom exceptions is a breaking change. Users may rely on
> AirflowException in callbacks (on_failure_callback, on_retry_callback) or
> in custom wrappers that catch it explicitly.
>
> Rather than making these changes incrementally across individual PRs, I'd
> like to discuss a coordinated migration strategy.
> Proposal
>
> *1. Emit a deprecation warning from **AirflowException.__init__**.*
>
> Although emitting warnings when exceptions are raised is uncommon, it is
> justified here because users may depend on AirflowException in callbacks
> and exception handlers. This provides advance notice before
> package-specific migrations occur.
>
> *2. Migrate per package in the next breaking release of that package.*
>
> Require a single migration PR per package (airflow-core and each provider),
> rather than spreading exception changes across multiple PRs and releases.
> The migration would be included in that package's next breaking release and
> documented accordingly.
> Questions
>
>    - Should we add a deprecation warning to AirflowException?
>    - Is the per-package migration approach reasonable?
>    - Should we define a project-wide exception taxonomy, or leave exception
>    choices to individual provider teams?
>
> If there's agreement, I'll follow up with a lazy-consensus proposal.
>
> Thanks,
>
>
> Shahar
>