[jira] [Commented] (AIRFLOW-1491) Celery executor restarts result in duplicate tasks

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739839#comment-16739839 ] 

ASF GitHub Bot commented on AIRFLOW-1491:
-----------------------------------------

stale[bot] commented on pull request #2538: [AIRFLOW-1491] Recover celery queue on restart
URL: https://github.com/apache/airflow/pull/2538
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Celery executor restarts result in duplicate tasks
> --------------------------------------------------
>
>                 Key: AIRFLOW-1491
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1491
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: George Leslie-Waksman
>            Assignee: George Leslie-Waksman
>            Priority: Major
>
> Since the Celery executor uses internal state to determine what it has queued already, anything that wipes the executor state (such as a scheduler restart) will result in rescheduling of tasks.
> This bug exacerbates a number of task scheduling race conditions and can result in duplicate task execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)