You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/09/29 09:50:49 UTC
[PR] Warn that a durable agent retry replays toward the same usage limit [airflow]
kaxil opened a new pull request, #73893:
URL: https://github.com/apache/airflow/pull/73893
Stacked on #73873, which this branch includes; only the last commit is new here.
With `durable=True`, a retry replays every cached model response and tool call, and replayed steps count toward `usage_limits` like live ones. A run that reached its limit therefore usually stops at the same step on every retry. Until now the task log only showed the same `UsageLimitExceeded` on each try, with nothing saying why the retries were not helping.
`AgentOperator` now logs a warning when a durable run reaches a usage limit. It includes the error, says a retry will likely stop at the same step if that is the task's own limit, and says that raising `usage_limits` is what lets the run go further. The durable execution guide says the same, including that an unset `usage_limits` still caps a run at pydantic-ai's default of 50 requests. Retry behaviour does not change.
**Why not stop retrying instead.** Failing the task once a retry reaches the limit purely from the cache looks like the obvious fix, but that only holds when everything in the run replays exactly. Several things run live on every attempt and can change the outcome: output functions and validators, tools asking for a retry, tool results the cache could not store, tools on routes the cache does not wrap, and agents run inside a tool, which can add their own usage or hit their own limits. Each of these can make an attempt look fully replayed and still leave a later attempt able to finish, so an operator-side fail-fast would sometimes drop a retry that would have succeeded. A redundant retry that only replays costs no model tokens, so the warning is the trade worth making.
Run end to end on a local Airflow 3.4.0 with a durable agent, `usage_limits={"request_limit": 1}` and `retries=1`. On try 1 the run makes its model call and tool call live, reaches the limit, and logs the warning:

On try 2 both steps replay from the cache (`replayed 2 cached steps ... cached 0 new steps`) and the run stops at the same limit, as the warning says:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Warn that a durable agent retry replays toward the same usage limit [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #73893:
URL: https://github.com/apache/airflow/pull/73893#issuecomment-5896123469
Test failing needs to be looked at here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Warn that a durable agent retry replays toward the same usage limit [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #73893:
URL: https://github.com/apache/airflow/pull/73893#issuecomment-5905292064
Closing: #73706 made the premise of this PR obsolete. Its `ReplayUsageLedger` stops replayed steps from counting toward `usage_limits`, so a durable retry no longer replays toward the limit it hit, and the cross-attempt budget covers what this warning was trying to explain. Rebasing would ship a warning that is no longer true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Warn that a durable agent retry replays toward the same usage limit [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #73893:
URL: https://github.com/apache/airflow/pull/73893#issuecomment-5896154534
3.0.6 is failing on the `cap_structlog` assertions — that needs `airflow._shared`, which isn't there before 3.1. Only the log check breaks, the counting's fine. Maybe pull the log assertion into a separate `AIRFLOW_V_3_1_PLUS` test and keep the count check running on all versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Warn that a durable agent retry replays toward the same usage limit [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed pull request #73893: Warn that a durable agent retry replays toward the same usage limit
URL: https://github.com/apache/airflow/pull/73893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org