You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2017/12/08 09:22:01 UTC
[jira] [Commented] (AIRFLOW-1059) Reset_state_for_orphaned_task
should operate in batch for the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283249#comment-16283249 ]
ASF subversion and git services commented on AIRFLOW-1059:
----------------------------------------------------------
Commit 8626186ca8c244386a8a97fcaf6d4221270863da in incubator-airflow's branch refs/heads/master from GRANT NICHOLAS
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8626186 ]
[AIRFLOW-1884][AIRFLOW-1059] Reset orphaned task state for external dagruns
On scheduler startup, orphaned task instances have
their state cleared and are rescheduled to avoid
having tasks that are stuck in a QUEUED state
forever. Previously, this check ignored backfilled
and externally triggered dagruns, meaning that
backfilled and externally triggered dagruns could
have orphaned tasks that are stuck forever. This
changeset removes the special case logic for
externally triggered dagruns, ensuring that
externally triggered dagruns are crash safe. This
same fix cannot be applied to backfilled dagruns,
so for now backfilled dagruns are not crash safe.
Closes #2843 from grantnicholas/AIRFLOW-1884
> Reset_state_for_orphaned_task should operate in batch for the scheduler
> -----------------------------------------------------------------------
>
> Key: AIRFLOW-1059
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1059
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Alex Guziel
> Assignee: Alex Guziel
> Fix For: 1.9.0
>
>
> Scheduler startup is very slow due to resetting state making a query for each dag run. We should be able to do this in a constant number of queries which will increase scheduler startup time significantly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)