You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/16 14:23:22 UTC
[GitHub] [airflow] Chris7 opened a new issue #21616: Deferred tasks being ran on every triggerer instance
Chris7 opened a new issue #21616:
URL: https://github.com/apache/airflow/issues/21616
### Apache Airflow version
2.2.3 (latest released)
### What happened
I'm currently using airflow triggers, but I've noticed that deferred tasks are being ran on every triggerer instance right away:
```
host 1 -> 2/10/2022 1:06:08 PM Job a702656f-01ce-4e7a-893a-5b42cdaa38e2 progressed from Unknown to RUNNABLE.
host 2 -> 2/10/2022 1:06:06 PM Job a702656f-01ce-4e7a-893a-5b42cdaa38e2 progressed from Unknown to RUNNABLE.
```
within 2 seconds, the job was issued on both triggerer instances.
### What you expected to happen
The deferred task is only scheduled on 1 triggerer instance.
### How to reproduce
Create many jobs that have to be deferred, and start multiple triggerers.
### Operating System
Ubuntu 20.04.3 LTS
### Versions of Apache Airflow Providers
2.2.3
### Deployment
Other Docker-based deployment
### Deployment details
A docker setup w/ multiple triggerer instances running.
### Anything else
I believe this issue is due to a race condition here: https://github.com/apache/airflow/blob/84a59a6d74510aff4059a3ca2da793d996e86fa1/airflow/models/trigger.py#L175. If multiple instances start at the same time, each instance will get the same tasks in their alive_job_ids.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #21616: Deferred tasks being ran on every triggerer instance
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #21616:
URL: https://github.com/apache/airflow/issues/21616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org