[jira] [Commented] (AIRFLOW-224) Collect orphaned tasks in case of unclean shutdown

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333346#comment-15333346 ] 

ASF subversion and git services commented on AIRFLOW-224:
---------------------------------------------------------

Commit fb89276750d23fc19a2d57036bc59d3aef846a26 in incubator-airflow's branch refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fb89276 ]

[AIRFLOW-224] Collect orphaned tasks and reschedule them

Tasks can get orphaned if the scheduler is killed in the middle
of processing the tasks or if the MQ queue is cleared without
a worker having picked these up. Now tasks do not get set
to a scheduled state anymore if they have not been sent to the
executor yet. Next to that a garbage collector scans the executor
for tasks not being present and reschedules those if needed.


> Collect orphaned tasks in case of unclean shutdown
> --------------------------------------------------
>
>                 Key: AIRFLOW-224
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-224
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Bolke de Bruin
>             Fix For: Airflow 1.8
>
>
> Tasks in a "scheduled" state can get orphaned if either the executor or the scheduler is shutdown uncleanly (kill -9). 
> This should be addressed by making sure the scheduler confirms sending tasks to the executor and checking weather "scheduled" tasks are still in the queue of the executor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)