You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/08/12 23:41:48 UTC
[PR] Operator extra links per attempt (draft C): use the task state store [airflow]
1fanwang opened a new pull request, #71522:
URL: https://github.com/apache/airflow/pull/71522
**TL;DR — operator extra links don't work per attempt** (#71471). After a retry the button
under attempt 1 opens attempt 2's logs. It hits every bundled provider linking to per-attempt
logs: EMR, Glue, Databricks, Dataproc, Livy.
Draft **C of three** for the same bug — pick a shape and I'll finish it and close the others.
| | Approach | Trade |
|---|---|---|
| A | #71518 — a row per attempt in XCom | smallest; core owns a key convention for every link |
| B | #71519 — the link opts in | no extra rows when unneeded; changes the serialized DAG format |
| **C** | **this PR — the task state store** | **nothing about the clear moves; 3.3+ only** |
## How
The worker writes each attempt's rendered link to the task state store as well as to XCom. The
store is not cleared on retry — that is what it exists for, and what the durable-execution work
in #69914 and #71211 relies on — so no exemption from the XCom clear is needed. XCom keeps
holding the latest attempt for anything already reading it; the reader prefers the store and
falls back to XCom.
`TaskScope` is keyed on `(dag_id, run_id, task_id, map_index)` with no attempt, so the try is
still encoded in the key. The store buys immunity from the clear, not a per-attempt dimension.
## Testing
Red to green on a real 3.3.0 install, store populated per attempt, XCom holding the latest.
<details><summary>Raw output</summary>
Before, stock 3.3.0 — attempt 1 resolves to attempt 2's URL:
```
$ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=sqlite:///af.db python c33.py
try 1 -> https://logs/attempt-2
try 2 -> https://logs/attempt-2
```
After:
```
$ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=sqlite:///af.db python c33.py
try 1 -> https://logs/attempt-1
try 2 -> https://logs/attempt-2
```
</details>
## Two costs
It puts UI-render data in a store built for job identity, and the reader queries
`task_state_store` directly rather than through the accessor workers use, so a deployment
pointing `[state_store] backend` elsewhere would need the api-server to reach it.
It also cannot ship on the 3.2 line: the store landed after 3.2.2, where
`airflow.models.task_state_store` and `airflow.sdk.state` are both absent. A and B work there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return each attempt's own URL from operator extra links [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang closed pull request #71522: Return each attempt's own URL from operator extra links
URL: https://github.com/apache/airflow/pull/71522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return each attempt's own URL from operator extra links [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #71522:
URL: https://github.com/apache/airflow/pull/71522#issuecomment-5829455921
@amoghrajesh Can you take a look at this, it is like trying to fix the same issue as #73315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return each attempt's own URL from operator extra links [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on PR #71522:
URL: https://github.com/apache/airflow/pull/71522#issuecomment-5839738273
Closing in favor of #73315 — @amoghrajesh's fix solves the same problem (operator extra links surviving across retries) with a simpler, self-contained approach: per-try-suffixed XCom keys plus skipping those keys during XCom clearing. Mine additionally depended on the task-state-store abstraction from #71307/#68936, which is still pending consensus on the mailing list, so 73315 is the better fix to land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org