You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "GustavoBuosi (via GitHub)" <gi...@apache.org> on 2026/06/05 09:29:30 UTC
[D] Possible Race Condition on Task Scheduling in Kubernetes Executor [airflow]
GitHub user GustavoBuosi created a discussion: Possible Race Condition on Task Scheduling in Kubernetes Executor
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Possible Race Condition caused by duplicated Task Attempts in Kubernetes Executor [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Possible Race Condition caused by duplicated Task Attempts in Kubernetes Executor
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
EDIT 1: we've come across #60330 which would at least prevent a new task attempt being generated in Airflow 3.1.8+, however we are still seeing open PRs with issues that may cause task execution inconsistencies like #63355.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
### Edit 1
We've come across #60330 which would at least prevent a new task attempt id being generated in Airflow 3.1.8+, however we are still seeing open PRs with issues that may cause task execution inconsistencies like #63355.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
### Edit 1
We've come across #60330 which would at least prevent a new task attempt id being generated in Airflow 3.1.8+ if the task is marked as UP_FOR_RESCHEDULE, however, the timeline of events here seem to be:
Scheduler 2 creates a new ti attempt -> Attempt 2 Updates/deletes task attempt id 1 from metastore, making attempt 1 disappear -> attempt 2 gets killed -> While attempt 1 was running and tried to mark itself as "UP_FOR_RESCHEDULE", it realized this state had been deleted or modified and returned a 409
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is still occurring [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is still occurring
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
### Edit 1
We've come across #60330 which would at least prevent a new task attempt id being generated in Airflow 3.1.8+ if the task is marked as UP_FOR_RESCHEDULE, however, the timeline of events here seem to be:
Scheduler 2 creates a new ti attempt -> Attempt 2 Updates/deletes task attempt id 1 from metastore, making attempt 1 disappear -> attempt 2 gets killed -> While attempt 1 was running and tried to mark itself as "UP_FOR_RESCHEDULE", it realized this state had been deleted or modified and returned a 409
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a comment on the discussion: Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved
Hello @UnknownHawkins , thank you for the reply. I have confirmed these are the versions we indeed have in our cluster.
While I do understand significant changes were made and should have been addressed in Airflow 2.7.3, from what I've seen they still were occurring in Airflow 3 (as the PR you have shared mentions 3.1.6).
There is a recently merged PR that handles other race conditions observed by users #60330, and it looks like it could tackle the error we have reported - however, its milestone is for Airflow 3.3.0 (a currently unreleased version today), and I am not sure it would prevent the bug reported.
If I my understanding is correct, #60330 **may** have fixed the reported error (`test_schedule_tis_empty_operator_is_noop_if_ti_already_running` seems to partially simulate it), but the fix would be available on 3.3.0. With that, I have two questions:
- Could you confirm if what I've said makes sense?
- If it does, do you think it's worth it to at least write a unit test that tries to replicate my case?
I would be happy to contribute to the project opening a PR with my colleagues.
GitHub link: https://github.com/apache/airflow/discussions/68071#discussioncomment-17506918
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Possible Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Possible Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
### Edit 1
We've come across #60330 which would at least prevent a new task attempt being generated in Airflow 3.1.8+, however we are still seeing open PRs with issues that may cause task execution inconsistencies like #63355.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved [airflow]
Posted by "UnknownHawkins (via GitHub)" <gi...@apache.org>.
GitHub user UnknownHawkins added a comment to the discussion: Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved
We'll produce answer: mention that duplicate task attempts race condition was addressed in Airflow 2.7.3 (PR #63183) and further improvements in 2.8.0; with Airflow 3.1.7 and provider 10.12.3 you should be safe, but ensure you have
GitHub link: https://github.com/apache/airflow/discussions/68071#discussioncomment-17480452
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Possible Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Possible Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
### Edit 1
We've come across #60330 which would at least prevent a new task attempt id being generated in Airflow 3.1.8+, however we are still seeing open PRs with issues that may cause task execution inconsistencies like #63355.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is still occurring [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a comment on the discussion: Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is still occurring
Hello @UnknownHawkins , thank you for the reply. I have confirmed these are the versions we indeed have in our cluster.
While I do understand significant changes were made and should have been addressed in Airflow 2.7.3, from what I've seen they still were occurring in Airflow 3 (as the PR you have shared mentions 3.1.6).
There is a recently merged PR that handles other race conditions observed by users #63183, and it looks like it could tackle the error we have reported - however, its milestone is for Airflow 3.3.0 (a currently unreleased version today), and I am not sure it would prevent the bug reported.
If I my understanding is correct, #60330 **may** have fixed the reported error (`test_schedule_tis_empty_operator_is_noop_if_ti_already_running` seems to partially simulate it), but the fix would be available on 3.3.0. With that, I have two questions:
- Could you confirm if what I've said makes sense?
- If it does, do you think it's worth it to at least write a unit test that tries to replicate my case?
I would be happy to contribute to the project opening a PR with my colleagues.
GitHub link: https://github.com/apache/airflow/discussions/68071#discussioncomment-17506918
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Possible Race Condition on Task Scheduling in Kubernetes Executor [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Possible Race Condition on Task Scheduling in Kubernetes Executor
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
EDIT 1: we've come across #60330 which would at least prevent a new task attempt being generated in Airflow 3.1.8+, however we are still seeing open PRs with issues that may cause task execution inconsistencies like #63355.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Possible Race Condition caused by duplicated Task Attempts in Kubernetes Executor [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a discussion: Possible Race Condition caused by duplicated Task Attempts in Kubernetes Executor
Hello! In my company, me, @NiltonDuarte and @leandroszikora are deploying Airflow 3.1.7 for many teams and providing support for their infrastructure. In the case we are reporting here, our current setup has:
* Airflow 3.1.7
* Python 3.13.11
* Postgres Metastore
* `apache-airflow-providers-cncf-kubernetes==10.12.3`
* Linux Images
* 3 API Server Pods
* 3 Scheduler Pods
* 3 DAG Processor Pods
Currently, we have already observed some issues that may affect task runtimes, and most of them were indirectly related to infra resources that interfered with the Scheduler and API Server behavior.
However, in an environment that only runs on OnDemand instances in Azure Kubernetes Services we realized that some tasks would still die abruptly, not related to OoM issues. This was observed during one of the attempts of an ExternalTaskSensor in Reschedule mode, that was running for approximately 3 hours and had a timeout of 8 hours.
Two workers were created for the same sensor in a ~1 second time window.
### Attempt 1 (started at 2026-06-04T00:38:48):
```json
{"timestamp":"2026-06-04T00:38:48.358475Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:55.696277Z","level":"info","event":"Rescheduling task, marking task as UP_FOR_RESCHEDULE","logger":"task","filename":"supervisor.py","lineno":1806}
{"timestamp":"2026-06-04T00:38:55.712667Z","level":"error","event":"API server error","status_code":409,"detail":{"detail":{"reason":"invalid_state","message":"TI was not in the running state so it cannot be updated","previous_state":"failed"}}, ...}
```
### Attempt 2 (started at 2026-06-04T00:38:49, one second later):
```json
{"timestamp":"2026-06-04T00:38:49.686111Z","level":"info","event":"Executing workload","workload": ...}
{"timestamp":"2026-06-04T00:38:50.500902Z","level":"info","event":"Process exited","pid":17,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":710}
```
We suspect two of out schedulers created a race condition and in the end we get a similar behavior as seen in #63183.
We have looked at 2 of our schedulers logs (the third one did not contain relevant traces) and interestingly enough found out that:
* Scheduler 1 started attempt 1 of the task. After it got killed, it constantly **kept emitting the event that attempt 1 failed continuously** - most likely it keeps fetching incorrect state:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.1, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
* Scheduler 2 started attempt 2 of the task. However, it emitted the event about this attempt once and moved on:
```json
2026-06-04T02:34:12.394506Z [warning ] Event: dag-... Failed, task: dag.task.2, annotations: <omitted> [airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher] loc=kubernetes_executor_utils.py:309
```
We expected that Scheduler 2 would be able to check that there is attempt 1 that is in UP_FOR_RESCHEDULE and would not generate a second run.
After searching in the repo, the only discussion we've found similar to this is #57041 - however, it has not been replied. As its author states, we also have checked that we have use_row_level_locking=True.
There does seem to have a PR that would fix the issue for cases when both task runs are on success #63355, but it seems our scenario described here is different.
We can provide additional logs and info if desired.
### Edit 1
We've come across #60330 which would at least prevent a new task attempt being generated in Airflow 3.1.8+, however we are still seeing open PRs with issues that may cause task execution inconsistencies like #63355.
GitHub link: https://github.com/apache/airflow/discussions/68071
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi edited a comment on the discussion: Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved
Hello @UnknownHawkins , thank you for the reply. I have confirmed these are the versions we indeed have in our cluster.
While I do understand significant changes were made and should have been addressed in Airflow 2.7.3, from what I've seen they still were occurring in Airflow 3 (as the PR you have shared mentions 3.1.6).
There is a recently merged PR that handles other race conditions observed by users #60330, and it looks like it could tackle the error we have reported - however, its milestone is for Airflow 3.3.0 (a currently unreleased version today), and I am not sure it would prevent the bug reported.
If I am understanding correctly #60330, **may** have fixed the reported error, but the fix would be available on 3.3.0. Do you think it makes sense to at least a unit test that tries to replicate my case?
I would be happy to contribute to the project opening a PR with my colleagues.
GitHub link: https://github.com/apache/airflow/discussions/68071#discussioncomment-17506918
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved [airflow]
Posted by "GustavoBuosi (via GitHub)" <gi...@apache.org>.
GitHub user GustavoBuosi added a comment to the discussion: Trying to understand if Race Condition caused by creation of duplicated Task Attempts in Kubernetes Executor is already solved
Hello @UnknownHawkins , thank you for the reply. I have confirmed these are the version we indeed have in our cluster.
While I do understand significant changes were made and should have been addressed in Airflow 2.7.3, from what I've seen they still were occurring in Airflow 3 (as the PR you have shared mentions 3.1.6).
There is a recently merged PR handles other race conditions observed by users #60330, and it looks like it could tackle the error we have reported - however, its milestone is for Airflow 3.3.0 (a currently unreleased version today).
If I am understanding correctly #60330, **may** have fixed the reported error, but the fix would be available on 3.3.0. Do you think it makes sense to at least a unit test that tries to replicate my case?
I would be happy to contribute to the project opening a PR with my colleagues.
GitHub link: https://github.com/apache/airflow/discussions/68071#discussioncomment-17506918
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org