You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "cedric-fauth (via GitHub)" <gi...@apache.org> on 2026/03/09 13:06:19 UTC
[I] Task completes successfully but gets killed and marked as failed [airflow]
cedric-fauth opened a new issue, #63183:
URL: https://github.com/apache/airflow/issues/63183
### Apache Airflow version
Other Airflow 3 version (please specify below)
### If "Other Airflow 3 version" selected, which one?
3.1.6
### What happened?
Many of my airflow DAGs are failing because some tasks are killed and marked as failed even if they completed successfully. This seems to happen randomly but often multiple DAGs fail at a similar time.
### Behavior:
Just after a task logs their last log message and returns there is an internal error with the airflow sdk involved:
```
{"logger": "airflow.task.operators.dags.custom_operators.DjangoOperator", "filename": "python.py", "lineno": 216, "event": "Done. Returned value was: None", "level": "info"}
{"logger": "task", "filename": "task_runner.py", "lineno": 1562, "error_detail": [{"exc_type": "AirflowRuntimeError", "exc_value": "API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error', 'detail': {'detail': {'reason': 'invalid_state', 'message': 'TI was not in the running state so it cannot be updated', 'previous_state': 'success'}}}", "exc_notes": [], "syntax_error": null, "is_cause": false, "frames": [{"filename": "/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", "lineno": 1555, "name": "main"}, {"filename": "/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", "lineno": 1083, "name": "run"}, {"filename": "/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", "lineno": 206, "name": "send"}, {"filename": "/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", "lineno": 270, "name": "_get_response"}, {"filename": "/bin/app/lib/python3.12/site-packages/airfl
ow/sdk/execution_time/comms.py", "lineno": 257, "name": "_from_frame"}], "is_group": false, "exceptions": []}], "event": "Top level error", "level": "error"}
{"exit_code": 1, "event": "Process exited abnormally", "level": "warning", "logger": "task"}
```
[{\"exc_notes\":[],\"exc_type\":\"AirflowRuntimeError\",\"exc_value\":\"API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error', 'detail': {'detail': {'reason': 'invalid_state', 'message': 'TI was not in the running state so it cannot be updated', 'previous_state': 'success'}}}\",\"exceptions\":[],\"frames\":[{\"filename\":\"/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py\",\"lineno\":1555,\"name\":\"main\"},{\"filename\":\"/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py\",\"lineno\":1083,\"name\":\"run\"},{\"filename\":\"/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py\",\"lineno\":206,\"name\":\"send\"},{\"filename\":\"/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py\",\"lineno\":270,\"name\":\"_get_response\"},{\"filename\":\"/bin/app/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py\",\"lineno\":257,\"name\":\"_from_frame\"}],\"is_c
ause\":false,\"is_group\":false,\"syntax_error\":\"\\u003cnil\\u003e\"}]
### What you think should happen instead?
After the task ends it should update its state to successful and updating the state should not result in an error. It looks a bit like airflow already updated the state and tries to update it again even if the task was already completed.
### How to reproduce
I don't know how this can be reproduced but there are similar issues to this one which handle different cases of `TI was not in the running state so it cannot be updated`.
### Operating System
WSL
### Versions of Apache Airflow Providers
apache-airflow[celery, redis, amazon, postgres, docker, fab]==3.1.6
apache-airflow-providers-fab==3.1.2
### Deployment
Other
### Deployment details
Deployed on ECS using a Fargate Cluster. One task per worker, scheduler, etc. Only one container running per task with additional logging sidecar containers.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task completes successfully but gets killed and marked as failed [airflow]
Posted by "cedric-fauth (via GitHub)" <gi...@apache.org>.
cedric-fauth commented on issue #63183:
URL: https://github.com/apache/airflow/issues/63183#issuecomment-4044447935
Thank you very much for finding a fix for this! Hope this will be merged 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task completes successfully but gets killed and marked as failed [airflow]
Posted by "sidshas03 (via GitHub)" <gi...@apache.org>.
sidshas03 commented on issue #63183:
URL: https://github.com/apache/airflow/issues/63183#issuecomment-4039681361
Opened PR #63354 with a fix for duplicate final-state update conflicts in task runner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task completes successfully but gets killed and marked as failed [airflow]
Posted by "sidshas03 (via GitHub)" <gi...@apache.org>.
sidshas03 commented on issue #63183:
URL: https://github.com/apache/airflow/issues/63183#issuecomment-4034814724
I can help debug this from my side. I’m on Airflow 3.1.6 and seeing the same `409 invalid_state` with `previous_state=success`.
I’ll run the same workload in two modes:
- current HA setup (multiple schedulers / API servers)
- single scheduler + single API server
For each failure I’ll share full logs (worker + scheduler + API server) for the same `dag_id/run_id/task_id` window. I’ll also check whether `task_instance_history` is missing try 1 rows when try 2 exists.
Likely related to #57618 / #60330, I can open a PR with a reproducible test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task completes successfully but gets killed and marked as failed [airflow]
Posted by "girishtulabandu (via GitHub)" <gi...@apache.org>.
girishtulabandu commented on issue #63183:
URL: https://github.com/apache/airflow/issues/63183#issuecomment-4152790326
This PR will fix the issue in Airflow 3.0.6 as well? as we are facing the same issue in 3.0.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task completes successfully but gets killed and marked as failed [airflow]
Posted by "sidshas03 (via GitHub)" <gi...@apache.org>.
sidshas03 commented on issue #63183:
URL: https://github.com/apache/airflow/issues/63183#issuecomment-4040533743
Main bug: #63183
Current fix PR: #63355 (this supersedes my earlier #63354, which I closed)
Related issues: #57618 and #60330
What #63355 fixes:
When task runner sends final SUCCESS, if API replies with `409 invalid_state` and `previous_state=success`, we now treat it as duplicate/idempotent state update and do not fail the task process.
What #63355 does NOT fix:
It does not solve the broader HA scheduler race conditions tracked in #57618 / #60330 (those are still open).
I understand #57618 and #60330 are broader HA scheduler race-condition problems and are still open. I can work on that next scope as well; I am already looking into it and will push follow-up changes in a separate update/PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task completes successfully but gets killed and marked as failed [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #63183: Task completes successfully but gets killed and marked as failed
URL: https://github.com/apache/airflow/issues/63183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org