You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Goutam Kumar Sahoo <go...@infosys.com> on 2018/09/07 11:21:17 UTC
TriggerDagRunOperator sub tasks are scheduled to run after few hours
HI Experts
In our project, we are trying to replicate the existing job scheduling implemented in Microsoft Orchestrater to new scheduler called "Apache Airflow" .
During this replication process we have an requirement to create Master DAG which in turn will call Other child DAGs based on different condition. We referred to the existing example (example_trigger_controller_dag, example_trigger_target_dag) available in Airflow and tried running them manually.
When we triggered the controller DAG manually , it is got triggered immediately with execution time created in local timezone (PDT) whereas the child DAG is scheduled to run few hours later. Even though child DAGRUN is showing as running, it is doing nothing and waiting for the time to satisfy. Our requirement is to trigger the Target/Child DAG run as soon it is triggered by controller DAG
[cid:image001.png@01D446CA.E00EEE40]
[cid:image002.png@01D446CA.E00EEE40]
If you see here , the trigger time and execution time is having a 7 hours GAP.
We have tried to solve this many different ways but couldn't succeed. We have asked for help in many forums but didn't get any satisfactory answer form any of them
Please help us ASAP as this issue is one of the show stopper for us .
Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours
Posted by Bolke de Bruin <bd...@gmail.com>.
Can you share the definition? And timezone settings?
Op vr 7 sep. 2018 20:53 schreef Maxime Beauchemin <
maximebeauchemin@gmail.com>:
> Is the issue timezone related? Personally I've only used Airflow in
> UTC-aligned environments so I can't help much on this topic. Bolke as
> contributed timezone awareness to the codebase in the past, I'm not sure
> what the common caveats may be.
>
> Max
>
> On Fri, Sep 7, 2018 at 4:29 AM Goutam Kumar Sahoo <
> goutamkumar.sahoo@infosys.com> wrote:
>
> > HI Experts
> >
> >
> >
> > In our project, we are trying to replicate the existing job scheduling
> > implemented in Microsoft Orchestrater to new scheduler called “Apache
> > Airflow” .
> >
> >
> >
> > During this replication process we have an requirement to create Master
> > DAG which in turn will call Other child DAGs based on different
> condition.
> > We referred to the existing example (example_trigger_controller_dag,
> > example_trigger_target_dag) available in Airflow and tried running them
> > manually.
> >
> >
> >
> > When we triggered the controller DAG manually , it is got triggered
> > immediately with execution time created in local timezone (PDT) whereas
> the
> > child DAG is scheduled to run few hours later. Even though child DAGRUN
> is
> > showing as running, it is doing nothing and waiting for the time to
> > satisfy. Our requirement is to trigger the Target/Child DAG run as soon
> it
> > is triggered by controller DAG
> >
> >
> >
> > If you see here , the trigger time and execution time is having a 7 hours
> > GAP.
> >
> >
> >
> > We have tried to solve this many different ways but couldn’t succeed. We
> > have asked for help in many forums but didn’t get any satisfactory answer
> > form any of them
> >
> >
> >
> >
> >
> > Please help us ASAP as this issue is one of the show stopper for us .
> >
> >
> >
> >
> >
> >
> >
>
Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours
Posted by Maxime Beauchemin <ma...@gmail.com>.
Is the issue timezone related? Personally I've only used Airflow in
UTC-aligned environments so I can't help much on this topic. Bolke as
contributed timezone awareness to the codebase in the past, I'm not sure
what the common caveats may be.
Max
On Fri, Sep 7, 2018 at 4:29 AM Goutam Kumar Sahoo <
goutamkumar.sahoo@infosys.com> wrote:
> HI Experts
>
>
>
> In our project, we are trying to replicate the existing job scheduling
> implemented in Microsoft Orchestrater to new scheduler called “Apache
> Airflow” .
>
>
>
> During this replication process we have an requirement to create Master
> DAG which in turn will call Other child DAGs based on different condition.
> We referred to the existing example (example_trigger_controller_dag,
> example_trigger_target_dag) available in Airflow and tried running them
> manually.
>
>
>
> When we triggered the controller DAG manually , it is got triggered
> immediately with execution time created in local timezone (PDT) whereas the
> child DAG is scheduled to run few hours later. Even though child DAGRUN is
> showing as running, it is doing nothing and waiting for the time to
> satisfy. Our requirement is to trigger the Target/Child DAG run as soon it
> is triggered by controller DAG
>
>
>
> If you see here , the trigger time and execution time is having a 7 hours
> GAP.
>
>
>
> We have tried to solve this many different ways but couldn’t succeed. We
> have asked for help in many forums but didn’t get any satisfactory answer
> form any of them
>
>
>
>
>
> Please help us ASAP as this issue is one of the show stopper for us .
>
>
>
>
>
>
>