Re: [I] TriggerDagRunOperator can fail with DagRunAlreadyExists after ambiguous execution API trigger retry [airflow]

["kalluripradeep (via GitHub)" <gi...@apache.org>]

kalluripradeep commented on issue #66905:
URL: https://github.com/apache/airflow/issues/66905#issuecomment-4766236919

   I've traced the full call stack for this issue and opened a fix PR: #68828
   
   **Root cause confirmed:**
   
   The tenacity `@retry` in `Client.request()` retries `httpx.RequestError` (transport errors). When `POST dag-runs/{dag_id}/{run_id}` creates the run server-side but the client sees a transport error, the retry gets `409 Conflict`. `DagRunOperations.trigger()` converts that to `ErrorResponse(DAGRUN_ALREADY_EXISTS)`. In `_handle_trigger_dag_run`, the `else` branch (when `skip_when_already_exists=False`, the default) marks the task **FAILED** without pushing `trigger_run_id` XCom.
   
   **Fix:**
   
   When `DAGRUN_ALREADY_EXISTS` arrives and `skip_when_already_exists=False`, the requested `run_id` already exists — the desired end state. The fix falls through to the normal success path (push XCom, honour `wait_for_completion`) instead of failing.
   
   The `skip_when_already_exists=True` branch is unchanged.
   
   Call stack for reference:
   1. `TriggerDagRunOperator._trigger_dag_af_3()` → raises `DagRunTriggerException`
   2. `task_runner.run()` → `_handle_trigger_dag_run(drte, context, ti, log)`  
   3. `SUPERVISOR_COMMS.send(TriggerDagRun(...))` → supervisor calls `client.dag_runs.trigger(...)`
   4. `DagRunOperations.trigger()`: POST → transport error → retry → `409 Conflict` → `ErrorResponse(DAGRUN_ALREADY_EXISTS)`
   5. Old: FAIL. New: fall through to XCom push + success.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org