Subdag tasks unable to start

[Victor Noagbodji <vn...@amplify-nation.com>]

Guys,

So we have a dag containing a subdag that triggers itself using TriggerDagRunOperator. Seems, after some research, that it's one way to simulate loops.

The triggered subdag gets added. But none of the tasks ever start. I researched the issue and no solution I found so far worked.

We are using UTC everywhere (db, worker, scheduler...). Start date is in the past (1 day ago). Schedule interval is "@monthly". Schedule interval for the subdag is "* * * * *".

Any idea what I am doing wrong?
Thanks for helping

Re: Subdag tasks unable to start

[Arthur Wiedmer <ar...@gmail.com>]

Glad it worked out!

Best,
Arthur

On Wed, May 9, 2018 at 3:26 AM Victor Noagbodji <
vnoagbodji@amplify-nation.com> wrote:

> Thanks Arthur.
>
> I went with having two DAGs as you suggested. For anyone reading, that's
> what worked for me.
>
> > On May 8, 2018, at 4:41 PM, Arthur Wiedmer <ar...@gmail.com>
> wrote:
> >
> > Hi Victor,
> >
> > Usually, we expect the Schedules to match with Subdags, I am not sure
> > Airflow supports heterogeneous schedules like this.
> >
> > What you could have is one Triggering DAG, and a second DAG that it
> > triggers at the right interval.
> >
> > Would you be able to explain further what you mean by simulate loops?
> Since
> > Airflow tries to schedule Directed *Acyclic* Graphs, we don't really
> > support a concept of loops.
> >
> > Best,
> > Arthur
>
>

Re: Subdag tasks unable to start

[Victor Noagbodji <vn...@amplify-nation.com>]

Thanks Arthur. 

I went with having two DAGs as you suggested. For anyone reading, that's what worked for me.

> On May 8, 2018, at 4:41 PM, Arthur Wiedmer <ar...@gmail.com> wrote:
> 
> Hi Victor,
> 
> Usually, we expect the Schedules to match with Subdags, I am not sure
> Airflow supports heterogeneous schedules like this.
> 
> What you could have is one Triggering DAG, and a second DAG that it
> triggers at the right interval.
> 
> Would you be able to explain further what you mean by simulate loops? Since
> Airflow tries to schedule Directed *Acyclic* Graphs, we don't really
> support a concept of loops.
> 
> Best,
> Arthur

Re: Subdag tasks unable to start

[Victor Noagbodji <vn...@amplify-nation.com>]

Thanks for replying : ) Thought I sent my messages to the wrong email or group.

> On May 8, 2018, at 4:41 PM, Arthur Wiedmer <ar...@gmail.com> wrote:
> 
> Hi Victor,
> 
> Usually, we expect the Schedules to match with Subdags, I am not sure
> Airflow supports heterogeneous schedules like this.

Good point. I don't understand the "* * * * *" schedule interval on the subdag either to be honest. We have used in other parts like that and I copied. I will match the parent DAG's schedule and see what it does.

> What you could have is one Triggering DAG, and a second DAG that it
> triggers at the right interval.

Thanks will also try this.

> Would you be able to explain further what you mean by simulate loops? Since
> Airflow tries to schedule Directed *Acyclic* Graphs, we don't really
> support a concept of loops

I know : ) So well, here's what I have (apologies for the bad diagram):

dag
-> start-dag
-> subdag
-> -> subdag-start
-> -> short-circuit-subdag (check if process should continue)
-> -> ... more tasks
-> -> triggerdaqgrun (with subdag id).
-> sqlsensor (check if process is still going)

In theory, it should work. In practice, it did not. The new subdag is added, but it never start. I have to start every task in it.

> Best,
> Arthur

Re: Subdag tasks unable to start

[Arthur Wiedmer <ar...@gmail.com>]

Hi Victor,

Usually, we expect the Schedules to match with Subdags, I am not sure
Airflow supports heterogeneous schedules like this.

What you could have is one Triggering DAG, and a second DAG that it
triggers at the right interval.

Would you be able to explain further what you mean by simulate loops? Since
Airflow tries to schedule Directed *Acyclic* Graphs, we don't really
support a concept of loops.

Best,
Arthur

Re: Subdag tasks unable to start

[Victor Noagbodji <vn...@amplify-nation.com>]

anyone?

> On May 8, 2018, at 9:01 AM, Victor Noagbodji <vn...@amplify-nation.com> wrote:
> 
> Guys,
> 
> So we have a dag containing a subdag that triggers itself using TriggerDagRunOperator. Seems, after some research, that it's one way to simulate loops.
> 
> The triggered subdag gets added. But none of the tasks ever start. I researched the issue and no solution I found so far worked.
> 
> We are using UTC everywhere (db, worker, scheduler...). Start date is in the past (1 day ago). Schedule interval is "@monthly". Schedule interval for the subdag is "* * * * *".
> 
> Any idea what I am doing wrong?
> Thanks for helping