You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/24 02:39:26 UTC

[GitHub] [airflow] KevinYang21 opened a new issue #8989: Enable ignore_first_depends_on_past for scheduled run and applies on new tasks

KevinYang21 opened a new issue #8989:
URL: https://github.com/apache/airflow/issues/8989

**Description**
In `airflow backfill`, `-I` is available so that the task instance of the first DAG run will ignore depends_on_past setting. It is a bit confusing from its name that it is a concept only for backfill jobs. For scheduled jobs, this behavior is on by default. However, one case we've been seeing a lot is that when new tasks with `depends_on_past=True` was added, it will not be scheduled until its previous TI is marked as success. This is because when we are checking the taks deps, we consider a task instance the first task instance only when it's from the first DAG run of the DAG(roughly), [code](https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/prev_dagrun_dep.py#L46-L65).

Proposed new behavior:

**Use case / motivation**
So that after adding new task, people don't need to remember manually marking the previous TI as success before getting the new task running.

<!-- What do you want to happen?
Add a new property `ignore_first_depends_on_past` in BaseOperator and control its value through a config entry. If it is true, then first task instance of a task will not respect `depends_on_past` setting. A task instance will be considered as the first task instance of a task when there is not task instance in the DB with an execution_date earlier than it.

Or, making it the default behavior.

-->

**Related Issues**
No

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8989: Enable ignore_first_depends_on_past for scheduled run and applies on new tasks

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #8989:
URL: https://github.com/apache/airflow/issues/8989#issuecomment-633168481


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ganose commented on issue #8989: Enable ignore_first_depends_on_past for scheduled run and applies on new tasks

Posted by GitBox <gi...@apache.org>.

ganose commented on issue #8989:
URL: https://github.com/apache/airflow/issues/8989#issuecomment-1012177267


   I know this is quite old, but has this been implemented? Would be really useful


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] KevinYang21 commented on issue #8989: Enable ignore_first_depends_on_past for scheduled run and applies on new tasks

Posted by GitBox <gi...@apache.org>.

KevinYang21 commented on issue #8989:
URL: https://github.com/apache/airflow/issues/8989#issuecomment-633169381


   Plan to work on this myself, maybe later next week


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pingzh commented on issue #8989: Enable ignore_first_depends_on_past for scheduled run and applies on new tasks

Posted by GitBox <gi...@apache.org>.

pingzh commented on issue #8989:
URL: https://github.com/apache/airflow/issues/8989#issuecomment-1076615463


   @liamganose @kurtqq here is the PR https://github.com/apache/airflow/pull/22491


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kurtqq commented on issue #8989: Enable ignore_first_depends_on_past for scheduled run and applies on new tasks

Posted by GitBox <gi...@apache.org>.

kurtqq commented on issue #8989:
URL: https://github.com/apache/airflow/issues/8989#issuecomment-735269850


   @KevinYang21 this would be extremely helpful. did u implemented it?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org