You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Tobias Feldhaus <To...@localsearch.ch> on 2017/07/05 12:28:34 UTC
airflow backfill seems to ignore -I
Hi,
When running airflow (1.8.1) backfill with –I and –t like:
airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e 2017-05-31 google_pipelines
I would expect it to rerun that specific task and ignoring the dependencies. Instead I see this:
[2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO - Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 3 non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L, 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L}, upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
Am I doing it wrong?
Best,
Tobi
Re: airflow backfill seems to ignore -I
Posted by Dan Davydov <da...@airbnb.com.INVALID>.
Airflow dependencies were simplified a bit, -i no longer ignores failed
state tasks, check out the -A flag which ignores pretty much all
dependencies (including the failed state tasks), though depending on the
version you are using there is a bug that is being fixed here:
https://github.com/apache/incubator-airflow/pull/2327
On Wed, Jul 5, 2017 at 8:45 AM, Weiwei Zhang <vv...@gmail.com> wrote:
> I am using airflow 1.8.1 as well. It is able to pick up the rest of the
> tasks when using backfill with the only exception which is when there is a
> task failed and I had to clear the status to allow the backfill to work.
> Any ideas why it is behaving like this? The previous version 1.6.2 didn't
> require clearing the failed task before doing backfill.
>
> Thx a lot,
> Viv
>
> > On Jul 5, 2017, at 7:38 AM, Tobias Feldhaus <
> Tobias.Feldhaus@localsearch.ch> wrote:
> >
> > I’ve just pulled the newest master and build it; the behaviour is the
> same. How can it be that “–i” is not honoured and dependencies are checked?
> >
> >
> > On 05.07.2017, 15:49, "Tobias Feldhaus" <To...@localsearch.ch>
> wrote:
> >
> > But nonetheless, is it not possible to backfill and ignore the
> upstream dependencies with “-i” ?
> >
> > On 05.07.2017, 14:34, "Tobias Feldhaus" <Tobias.Feldhaus@localsearch.
> ch> wrote:
> >
> > I meant –i , but I just needed to manually set the upstream
> things to success and it worked. Nevermind.
> >
> > Best,
> > Tobi
> >
> > On 05.07.2017, 14:28, "Tobias Feldhaus" <
> Tobias.Feldhaus@localsearch.ch<ma...@localsearch.ch>>
> wrote:
> >
> > Hi,
> >
> > When running airflow (1.8.1) backfill with –I and –t like:
> >
> > airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e
> 2017-05-31 google_pipelines
> >
> > I would expect it to rerun that specific task and ignoring the
> dependencies. Instead I see this:
> >
> > [2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO -
> Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not
> met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30
> 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule
> 'all_success' requires all upstream tasks to have succeeded, but found 3
> non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L,
> 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L},
> upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
> >
> > Am I doing it wrong?
> >
> >
> >
> > Best,
> > Tobi
> >
> >
> >
> >
>
Re: airflow backfill seems to ignore -I
Posted by Weiwei Zhang <vv...@gmail.com>.
I am using airflow 1.8.1 as well. It is able to pick up the rest of the tasks when using backfill with the only exception which is when there is a task failed and I had to clear the status to allow the backfill to work. Any ideas why it is behaving like this? The previous version 1.6.2 didn't require clearing the failed task before doing backfill.
Thx a lot,
Viv
> On Jul 5, 2017, at 7:38 AM, Tobias Feldhaus <To...@localsearch.ch> wrote:
>
> I’ve just pulled the newest master and build it; the behaviour is the same. How can it be that “–i” is not honoured and dependencies are checked?
>
>
> On 05.07.2017, 15:49, "Tobias Feldhaus" <To...@localsearch.ch> wrote:
>
> But nonetheless, is it not possible to backfill and ignore the upstream dependencies with “-i” ?
>
> On 05.07.2017, 14:34, "Tobias Feldhaus" <To...@localsearch.ch> wrote:
>
> I meant –i , but I just needed to manually set the upstream things to success and it worked. Nevermind.
>
> Best,
> Tobi
>
> On 05.07.2017, 14:28, "Tobias Feldhaus" <To...@localsearch.ch>> wrote:
>
> Hi,
>
> When running airflow (1.8.1) backfill with –I and –t like:
>
> airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e 2017-05-31 google_pipelines
>
> I would expect it to rerun that specific task and ignoring the dependencies. Instead I see this:
>
> [2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO - Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 3 non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L, 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L}, upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
>
> Am I doing it wrong?
>
>
>
> Best,
> Tobi
>
>
>
>
Re: airflow backfill seems to ignore -I
Posted by Tobias Feldhaus <To...@localsearch.ch>.
I’ve just pulled the newest master and build it; the behaviour is the same. How can it be that “–i” is not honoured and dependencies are checked?
On 05.07.2017, 15:49, "Tobias Feldhaus" <To...@localsearch.ch> wrote:
But nonetheless, is it not possible to backfill and ignore the upstream dependencies with “-i” ?
On 05.07.2017, 14:34, "Tobias Feldhaus" <To...@localsearch.ch> wrote:
I meant –i , but I just needed to manually set the upstream things to success and it worked. Nevermind.
Best,
Tobi
On 05.07.2017, 14:28, "Tobias Feldhaus" <To...@localsearch.ch>> wrote:
Hi,
When running airflow (1.8.1) backfill with –I and –t like:
airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e 2017-05-31 google_pipelines
I would expect it to rerun that specific task and ignoring the dependencies. Instead I see this:
[2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO - Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 3 non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L, 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L}, upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
Am I doing it wrong?
Best,
Tobi
Re: airflow backfill seems to ignore -I
Posted by Tobias Feldhaus <To...@localsearch.ch>.
But nonetheless, is it not possible to backfill and ignore the upstream dependencies with “-i” ?
On 05.07.2017, 14:34, "Tobias Feldhaus" <To...@localsearch.ch> wrote:
I meant –i , but I just needed to manually set the upstream things to success and it worked. Nevermind.
Best,
Tobi
On 05.07.2017, 14:28, "Tobias Feldhaus" <To...@localsearch.ch>> wrote:
Hi,
When running airflow (1.8.1) backfill with –I and –t like:
airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e 2017-05-31 google_pipelines
I would expect it to rerun that specific task and ignoring the dependencies. Instead I see this:
[2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO - Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 3 non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L, 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L}, upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
Am I doing it wrong?
Best,
Tobi
Re: airflow backfill seems to ignore -I
Posted by Tobias Feldhaus <To...@localsearch.ch>.
I meant –i , but I just needed to manually set the upstream things to success and it worked. Nevermind.
Best,
Tobi
On 05.07.2017, 14:28, "Tobias Feldhaus" <To...@localsearch.ch>> wrote:
Hi,
When running airflow (1.8.1) backfill with –I and –t like:
airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e 2017-05-31 google_pipelines
I would expect it to rerun that specific task and ignoring the dependencies. Instead I see this:
[2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO - Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 3 non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L, 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L}, upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
Am I doing it wrong?
Best,
Tobi