[jira] [Closed] (AIRFLOW-462) Concurrent Scheduler Jobs pushing the same task to queue

["Daniel Imberman (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Imberman closed AIRFLOW-462.
-----------------------------------
    Resolution: Auto Closed

> Concurrent Scheduler Jobs pushing the same task to queue
> --------------------------------------------------------
>
>                 Key: AIRFLOW-462
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-462
>             Project: Apache Airflow
>          Issue Type: Wish
>          Components: scheduler
>    Affects Versions: 1.7.0
>            Reporter: Yogesh
>            Priority: Major
>
> Hi,
> We are using airflow version 1.7.0 and we tried to implement high availability for airflow daemons in our production environment.
> Detailed high availability approach:
> -	Airflow running on two different machines with all the daemons(webserver, scheduler, execueor)
> -	Single mysql db repository pointed by two schedulers
> -	Replicated dag files in both the machines
> -       Running Single Rabbitmq Instance as message broker
> While doing so we came across below problem:
> -	A particular task was sent to executor twice (two entries in message queue) by two different schedulers. But, we see only single entry for the task instance in database which is correct.
> We just checked out the code and found below fact:
> -	before sending the task to executor it checks for task state in database and if its not already QUEUED it pushes that task to queue
> issue:
> As there is no locking implemented on the task instance in the database and both the Scheduler jobs are running so close that the second one might check for the status in the db before the first one updates that to QUEUED.
> We are not sure if in recent release this issue have been taken care of.
> Would you please help with some appropriate approach so that the high availability can be achieved.
> Thanks
> Yogesh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)