[DISCUSS] Deadline Behavior Question

[Dennis Ferruzzi <fe...@apache.org>]

We've circled this question a few times and I am trying to nail down an answer that we can implement.

Given this situation: You create a Dag with a Deadline Alert for one hour from the Dag's queued_at time.  Ten minutes in, the Dag fails.  What do you expect to happen?

There are three clear options here:

1. The Deadline callback fires at ten minutes (on Dagrun failure).  If the Deadline Alert means "I expect this to succeed in under an hour," then after 10 minutes we know it can't succeed, so fire now.  This is what PR #68961 implements (behind an opt-in flag).

2. The callback never fires.  If the Deadline Alert means "I expect this to finish within an hour," then a failed run has finished, so the deadline is satisfied and the pending alert is pruned, the same way we already prune it on success.

3. The callback fires at the one-hour mark regardless.  The deadline timer runs to expiry independent of the run's state, so a failed run still triggers the alert 50 minutes after it died. This is the current behaviour on main.

What the docs say right now (emphasis mine, and keeping in mind we can change this based on the decision here):

`core-concepts/dags.rst` says: "This example will send an email notification if the Dag **hasn't finished** 30 minutes after it was queued."

`howto/deadline-alerts.rst` says the DAGRUN_LOGICAL_DATE reference "would trigger the alert if the Dag **hasn't completed** 15 minutes after it was scheduled to start," and describes AVERAGE_RUNTIME as alerting when a "Dag is **running longer** than expected."

So the docs currently frame it as (not finished | not completed | still running), they do not say `not succeeded`.  By that phrasing, a failed run HAS finished, which points at option 2, and the user can use the existing `on_failure_callback` if that was what they wanted .

But `howto/sla-to-deadlines.rst` describes the mechanism differently: "When a Dag run starts, calculate and store (DeadlineReference + interval).  The scheduler loop then checks periodically ... **if any of those times have passed** then execute callback," and stresses the callback "will execute 'immediately' ... and not wait until the Dag finishes first."  That description has no step that cancels a pending deadline when the run ends, which is the only place option 3 (today's behaviour) finds any support.

So the documentation contradicts itself: the conceptual pages point to option 1 or option 2, depending on how you read it, and the migration guide's mechanism describes option 3.  So I am asking: which of the three SHOULD be the behaviour, and should any of the others be an opt-in alternative?

My own opinion is that option 1 was the intent, and I'd like to see it implemented as a bug fix rather than an optional flag: a deadline means "did not succeed in time," a failed run cannot succeed, so fire.  If we keep a flag at all I'd make option 1 the default and allow an opt-out to prune on failure (option 2) for teams who already get failure signal from on_failure_callback and don't want a duplicate.  The counter-argument worth weighing, raised on #68961: an on-call who clears a failed run and retries would, under option 1, lose the "still hasn't recovered by the deadline" signal, which is an argument for option 2 or for keeping timing-based firing available.

Could you reply with the number you'd expect as a user, and whether you'd want the other behaviours available as an option?  If there's rough consensus I'll make the docs and #68961 match it.

Thanks,
   - ferruzzi

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Deadline Behavior Question

[Elad Kalif <el...@apache.org>]

I've seen different tools take different assumptions.
Before sharing my own opinion, can you clarify if it is feasible and
reasonable to avoid taking a decision of our own?
Can we delegate the decision to the dag author to decide per dag what
behavior they'd like?

On Wed, Oct 21, 2026 at 1:05 AM Dennis Ferruzzi <fe...@apache.org> wrote:

> We've circled this question a few times and I am trying to nail down an
> answer that we can implement.
>
> Given this situation: You create a Dag with a Deadline Alert for one hour
> from the Dag's queued_at time.  Ten minutes in, the Dag fails.  What do you
> expect to happen?
>
> There are three clear options here:
>
> 1. The Deadline callback fires at ten minutes (on Dagrun failure).  If the
> Deadline Alert means "I expect this to succeed in under an hour," then
> after 10 minutes we know it can't succeed, so fire now.  This is what PR
> #68961 implements (behind an opt-in flag).
>
> 2. The callback never fires.  If the Deadline Alert means "I expect this
> to finish within an hour," then a failed run has finished, so the deadline
> is satisfied and the pending alert is pruned, the same way we already prune
> it on success.
>
> 3. The callback fires at the one-hour mark regardless.  The deadline timer
> runs to expiry independent of the run's state, so a failed run still
> triggers the alert 50 minutes after it died. This is the current behaviour
> on main.
>
> What the docs say right now (emphasis mine, and keeping in mind we can
> change this based on the decision here):
>
> `core-concepts/dags.rst` says: "This example will send an email
> notification if the Dag **hasn't finished** 30 minutes after it was queued."
>
> `howto/deadline-alerts.rst` says the DAGRUN_LOGICAL_DATE reference "would
> trigger the alert if the Dag **hasn't completed** 15 minutes after it was
> scheduled to start," and describes AVERAGE_RUNTIME as alerting when a "Dag
> is **running longer** than expected."
>
> So the docs currently frame it as (not finished | not completed | still
> running), they do not say `not succeeded`.  By that phrasing, a failed run
> HAS finished, which points at option 2, and the user can use the existing
> `on_failure_callback` if that was what they wanted .
>
> But `howto/sla-to-deadlines.rst` describes the mechanism differently:
> "When a Dag run starts, calculate and store (DeadlineReference +
> interval).  The scheduler loop then checks periodically ... **if any of
> those times have passed** then execute callback," and stresses the callback
> "will execute 'immediately' ... and not wait until the Dag finishes
> first."  That description has no step that cancels a pending deadline when
> the run ends, which is the only place option 3 (today's behaviour) finds
> any support.
>
> So the documentation contradicts itself: the conceptual pages point to
> option 1 or option 2, depending on how you read it, and the migration
> guide's mechanism describes option 3.  So I am asking: which of the three
> SHOULD be the behaviour, and should any of the others be an opt-in
> alternative?
>
> My own opinion is that option 1 was the intent, and I'd like to see it
> implemented as a bug fix rather than an optional flag: a deadline means
> "did not succeed in time," a failed run cannot succeed, so fire.  If we
> keep a flag at all I'd make option 1 the default and allow an opt-out to
> prune on failure (option 2) for teams who already get failure signal from
> on_failure_callback and don't want a duplicate.  The counter-argument worth
> weighing, raised on #68961: an on-call who clears a failed run and retries
> would, under option 1, lose the "still hasn't recovered by the deadline"
> signal, which is an argument for option 2 or for keeping timing-based
> firing available.
>
> Could you reply with the number you'd expect as a user, and whether you'd
> want the other behaviours available as an option?  If there's rough
> consensus I'll make the docs and #68961 match it.
>
> Thanks,
>    - ferruzzi
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] Deadline Behavior Question

[Alex <al...@gmail.com>]

I would keep the semantics as simple and predictable as possible,
especially from the perspective of the person receiving the alert.

To me, there are three distinct operational questions:

A. Did the DAG run fail? → Self-mitigation or human intervention.

B. Have we reached HH:mm and the DAG has not succeeded? → Act on the missed
deadline.

C. Is the DAG run likely to miss its SLA? → Proactive intervention.

Each question represents a different signal and potentially a different
operational response.

If I configure a Deadline Alert to answer question B, I would find it
confusing if it fired earlier simply because the DAG failed. I already have
failure alerts for that.

A DAG might fail at 09:10, get retried or recovered, and still succeed
before the 10:00 deadline. I don't want the deadline alert to fire at
09:10, nor do I want the failure to cancel the deadline check.

So option 3 from Dennis's list is the closest to my expectation, with one
important distinction: at the deadline, I care whether the DAG has
succeeded. If it has recovered and succeeded before then, there should be
no alert.

Ultimately, simplicity is key for me. I want each alert to answer one clear
question and drive one clear operational decision. I think mixing failure
detection, deadline breaches, and predictions would make operations harder.

Thanks
Alex Guglielmone Nemi

On Wed, Oct 21, 2026, 04:54 Dev-iL <gi...@gmail.com> wrote:

> "Why not both?"
>
> We have a retry policy for exceptions, why not a deadline policy?
>
> To me, a deadline always meant an answer to the **business** concern of
> "data is n/a by time X". My equivalent for this in airflow2 is a
> dataset+timetable schedule, where a timed run represents the deadline check
> and always branches into a sensor that checks if the output is missing,
> then immediately exits with the outcome.
>
> Under this framing, time measured from the start of a run and whether a
> given dagrun stopped ahead of time are immaterial - it could've been rerun
> (same or different dagrun), an external recovery processes could've kicked
> in, etc. -- if the data is available by the end of this, I don't want to be
> notified, and vice versa.
>
> If, on the other hand, we treat deadlines as "nonfatal" versions of
> timeouts - then I think 2 - the callback should never fire and whatever
> side-effect the deadline callback has should be included in the timeout
> callback. I struggle to see the usefulness of 3 beyond what a failure
> callback already gives.
>
> Best,
> Dev-iL
>
>
> On Wed, 21 Oct 2026, 1:05 Dennis Ferruzzi, <fe...@apache.org> wrote:
>
> > We've circled this question a few times and I am trying to nail down an
> > answer that we can implement.
> >
> > Given this situation: You create a Dag with a Deadline Alert for one hour
> > from the Dag's queued_at time.  Ten minutes in, the Dag fails.  What do
> you
> > expect to happen?
> >
> > There are three clear options here:
> >
> > 1. The Deadline callback fires at ten minutes (on Dagrun failure).  If
> the
> > Deadline Alert means "I expect this to succeed in under an hour," then
> > after 10 minutes we know it can't succeed, so fire now.  This is what PR
> > #68961 implements (behind an opt-in flag).
> >
> > 2. The callback never fires.  If the Deadline Alert means "I expect this
> > to finish within an hour," then a failed run has finished, so the
> deadline
> > is satisfied and the pending alert is pruned, the same way we already
> prune
> > it on success.
> >
> > 3. The callback fires at the one-hour mark regardless.  The deadline
> timer
> > runs to expiry independent of the run's state, so a failed run still
> > triggers the alert 50 minutes after it died. This is the current
> behaviour
> > on main.
> >
> > What the docs say right now (emphasis mine, and keeping in mind we can
> > change this based on the decision here):
> >
> > `core-concepts/dags.rst` says: "This example will send an email
> > notification if the Dag **hasn't finished** 30 minutes after it was
> queued."
> >
> > `howto/deadline-alerts.rst` says the DAGRUN_LOGICAL_DATE reference "would
> > trigger the alert if the Dag **hasn't completed** 15 minutes after it was
> > scheduled to start," and describes AVERAGE_RUNTIME as alerting when a
> "Dag
> > is **running longer** than expected."
> >
> > So the docs currently frame it as (not finished | not completed | still
> > running), they do not say `not succeeded`.  By that phrasing, a failed
> run
> > HAS finished, which points at option 2, and the user can use the existing
> > `on_failure_callback` if that was what they wanted .
> >
> > But `howto/sla-to-deadlines.rst` describes the mechanism differently:
> > "When a Dag run starts, calculate and store (DeadlineReference +
> > interval).  The scheduler loop then checks periodically ... **if any of
> > those times have passed** then execute callback," and stresses the
> callback
> > "will execute 'immediately' ... and not wait until the Dag finishes
> > first."  That description has no step that cancels a pending deadline
> when
> > the run ends, which is the only place option 3 (today's behaviour) finds
> > any support.
> >
> > So the documentation contradicts itself: the conceptual pages point to
> > option 1 or option 2, depending on how you read it, and the migration
> > guide's mechanism describes option 3.  So I am asking: which of the three
> > SHOULD be the behaviour, and should any of the others be an opt-in
> > alternative?
> >
> > My own opinion is that option 1 was the intent, and I'd like to see it
> > implemented as a bug fix rather than an optional flag: a deadline means
> > "did not succeed in time," a failed run cannot succeed, so fire.  If we
> > keep a flag at all I'd make option 1 the default and allow an opt-out to
> > prune on failure (option 2) for teams who already get failure signal from
> > on_failure_callback and don't want a duplicate.  The counter-argument
> worth
> > weighing, raised on #68961: an on-call who clears a failed run and
> retries
> > would, under option 1, lose the "still hasn't recovered by the deadline"
> > signal, which is an argument for option 2 or for keeping timing-based
> > firing available.
> >
> > Could you reply with the number you'd expect as a user, and whether you'd
> > want the other behaviours available as an option?  If there's rough
> > consensus I'll make the docs and #68961 match it.
> >
> > Thanks,
> >    - ferruzzi
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [DISCUSS] Deadline Behavior Question

[Dev-iL <gi...@gmail.com>]

"Why not both?"

We have a retry policy for exceptions, why not a deadline policy?

To me, a deadline always meant an answer to the **business** concern of
"data is n/a by time X". My equivalent for this in airflow2 is a
dataset+timetable schedule, where a timed run represents the deadline check
and always branches into a sensor that checks if the output is missing,
then immediately exits with the outcome.

Under this framing, time measured from the start of a run and whether a
given dagrun stopped ahead of time are immaterial - it could've been rerun
(same or different dagrun), an external recovery processes could've kicked
in, etc. -- if the data is available by the end of this, I don't want to be
notified, and vice versa.

If, on the other hand, we treat deadlines as "nonfatal" versions of
timeouts - then I think 2 - the callback should never fire and whatever
side-effect the deadline callback has should be included in the timeout
callback. I struggle to see the usefulness of 3 beyond what a failure
callback already gives.

Best,
Dev-iL


On Wed, 21 Oct 2026, 1:05 Dennis Ferruzzi, <fe...@apache.org> wrote:

> We've circled this question a few times and I am trying to nail down an
> answer that we can implement.
>
> Given this situation: You create a Dag with a Deadline Alert for one hour
> from the Dag's queued_at time.  Ten minutes in, the Dag fails.  What do you
> expect to happen?
>
> There are three clear options here:
>
> 1. The Deadline callback fires at ten minutes (on Dagrun failure).  If the
> Deadline Alert means "I expect this to succeed in under an hour," then
> after 10 minutes we know it can't succeed, so fire now.  This is what PR
> #68961 implements (behind an opt-in flag).
>
> 2. The callback never fires.  If the Deadline Alert means "I expect this
> to finish within an hour," then a failed run has finished, so the deadline
> is satisfied and the pending alert is pruned, the same way we already prune
> it on success.
>
> 3. The callback fires at the one-hour mark regardless.  The deadline timer
> runs to expiry independent of the run's state, so a failed run still
> triggers the alert 50 minutes after it died. This is the current behaviour
> on main.
>
> What the docs say right now (emphasis mine, and keeping in mind we can
> change this based on the decision here):
>
> `core-concepts/dags.rst` says: "This example will send an email
> notification if the Dag **hasn't finished** 30 minutes after it was queued."
>
> `howto/deadline-alerts.rst` says the DAGRUN_LOGICAL_DATE reference "would
> trigger the alert if the Dag **hasn't completed** 15 minutes after it was
> scheduled to start," and describes AVERAGE_RUNTIME as alerting when a "Dag
> is **running longer** than expected."
>
> So the docs currently frame it as (not finished | not completed | still
> running), they do not say `not succeeded`.  By that phrasing, a failed run
> HAS finished, which points at option 2, and the user can use the existing
> `on_failure_callback` if that was what they wanted .
>
> But `howto/sla-to-deadlines.rst` describes the mechanism differently:
> "When a Dag run starts, calculate and store (DeadlineReference +
> interval).  The scheduler loop then checks periodically ... **if any of
> those times have passed** then execute callback," and stresses the callback
> "will execute 'immediately' ... and not wait until the Dag finishes
> first."  That description has no step that cancels a pending deadline when
> the run ends, which is the only place option 3 (today's behaviour) finds
> any support.
>
> So the documentation contradicts itself: the conceptual pages point to
> option 1 or option 2, depending on how you read it, and the migration
> guide's mechanism describes option 3.  So I am asking: which of the three
> SHOULD be the behaviour, and should any of the others be an opt-in
> alternative?
>
> My own opinion is that option 1 was the intent, and I'd like to see it
> implemented as a bug fix rather than an optional flag: a deadline means
> "did not succeed in time," a failed run cannot succeed, so fire.  If we
> keep a flag at all I'd make option 1 the default and allow an opt-out to
> prune on failure (option 2) for teams who already get failure signal from
> on_failure_callback and don't want a duplicate.  The counter-argument worth
> weighing, raised on #68961: an on-call who clears a failed run and retries
> would, under option 1, lose the "still hasn't recovered by the deadline"
> signal, which is an argument for option 2 or for keeping timing-based
> firing available.
>
> Could you reply with the number you'd expect as a user, and whether you'd
> want the other behaviours available as an option?  If there's rough
> consensus I'll make the docs and #68961 match it.
>
> Thanks,
>    - ferruzzi
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>