You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Rafael Barbosa <rr...@gmail.com> on 2016/06/13 15:50:55 UTC
DAG dependencies on different frequencies
Hi everyone,
I tried looking around, but I cannot find an archive of this mailing list,
so I apologize if this question has been asked before.
I am looking for a why to set the dependency between two DAGs: a "Producer"
runs every day, appending data in a DB and a "Consumer" that runs weekly,
generating a summary report of the new data.
I would like to run the "Consumer" every Monday (for instance), but only
after the Monday instance of the daily "Producer" is complete. What would
be the best way to accomplish this?
Thanks,
Rafael
Re: DAG dependencies on different frequencies
Posted by Maxime Beauchemin <ma...@gmail.com>.
Another option that is somewhat of a hack is to use templating to run a
dummy workload.
So say for for a bash_operator you could have something like:
bash_command = """
{% if dayofweek(execution_date) == 'sunday' %}
./dostuff
{% else %}
echo "Skipping.;."
{% endif %}
"""
Max
On Tue, Jun 14, 2016 at 12:47 AM, Rafael Barbosa <rr...@gmail.com>
wrote:
> Hi Chris,
>
> It's not clear to me how to use the ExternalTaskSensor. In the case I
> described, as far as I understand, it can be used for the "consumer" to
> wait for every instance of the "producer" to finish. However, I would like
> the the "consumer" to wait 7 executions of the "producer".
>
> Am I missing something?
>
> Best,
> Rafael
>
> Rafael Barbosa
>
> On Tue, Jun 14, 2016 at 5:52 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > An alternative approach would be to use the ExternalTaskSensor to sense
> > when the task(s)/data that you depend upon has finished/arrived.
> >
> >
> >
> https://pythonhosted.org/airflow/code.html#airflow.operators.ExternalTaskSensor
> >
> > On Mon, Jun 13, 2016 at 9:08 AM, Laura Lorenz <ll...@industrydive.com>
> > wrote:
> >
> > > We do something like that here here in this way, and I'd love feedback
> > from
> > > anyone else if they do it a different way. Basically we use
> > > TriggerDagRunOperators and a task in the "Producer" that cares what day
> > of
> > > the week it is being run, and a "Consumer" DAG with no schedule
> interval
> > > that depends on being triggers when the right conditions of the
> > "Producer"
> > > DAG are met.
> > >
> > > In more detail, we have a daily DAG like your "Producer" that ingests
> > data
> > > every day. We want to run a series of tasks including generating a
> report
> > > in a "Consumer" DAG after after Monday's data has been processed; which
> > > would be on a Tuesday, after a run of "Producer" off the execution date
> > of
> > > Monday.
> > >
> > > The last task in "Producer" is a TriggerDagRunOperator that only
> triggers
> > > IF it notices that now() is Tuesday. This triggers an instance at the
> > right
> > > time of a child DAG of the "Consumer" type.
> > >
> > >
> > > On Mon, Jun 13, 2016 at 11:50 AM, Rafael Barbosa <rr...@gmail.com>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I tried looking around, but I cannot find an archive of this mailing
> > > list,
> > > > so I apologize if this question has been asked before.
> > > >
> > > > I am looking for a why to set the dependency between two DAGs: a
> > > "Producer"
> > > > runs every day, appending data in a DB and a "Consumer" that runs
> > weekly,
> > > > generating a summary report of the new data.
> > > >
> > > > I would like to run the "Consumer" every Monday (for instance), but
> > only
> > > > after the Monday instance of the daily "Producer" is complete. What
> > would
> > > > be the best way to accomplish this?
> > > >
> > > > Thanks,
> > > > Rafael
> > > >
> > >
> >
>
Re: DAG dependencies on different frequencies
Posted by Rafael Barbosa <rr...@gmail.com>.
Hi Chris,
It's not clear to me how to use the ExternalTaskSensor. In the case I
described, as far as I understand, it can be used for the "consumer" to
wait for every instance of the "producer" to finish. However, I would like
the the "consumer" to wait 7 executions of the "producer".
Am I missing something?
Best,
Rafael
Rafael Barbosa
On Tue, Jun 14, 2016 at 5:52 AM, Chris Riccomini <cr...@apache.org>
wrote:
> An alternative approach would be to use the ExternalTaskSensor to sense
> when the task(s)/data that you depend upon has finished/arrived.
>
>
> https://pythonhosted.org/airflow/code.html#airflow.operators.ExternalTaskSensor
>
> On Mon, Jun 13, 2016 at 9:08 AM, Laura Lorenz <ll...@industrydive.com>
> wrote:
>
> > We do something like that here here in this way, and I'd love feedback
> from
> > anyone else if they do it a different way. Basically we use
> > TriggerDagRunOperators and a task in the "Producer" that cares what day
> of
> > the week it is being run, and a "Consumer" DAG with no schedule interval
> > that depends on being triggers when the right conditions of the
> "Producer"
> > DAG are met.
> >
> > In more detail, we have a daily DAG like your "Producer" that ingests
> data
> > every day. We want to run a series of tasks including generating a report
> > in a "Consumer" DAG after after Monday's data has been processed; which
> > would be on a Tuesday, after a run of "Producer" off the execution date
> of
> > Monday.
> >
> > The last task in "Producer" is a TriggerDagRunOperator that only triggers
> > IF it notices that now() is Tuesday. This triggers an instance at the
> right
> > time of a child DAG of the "Consumer" type.
> >
> >
> > On Mon, Jun 13, 2016 at 11:50 AM, Rafael Barbosa <rr...@gmail.com>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I tried looking around, but I cannot find an archive of this mailing
> > list,
> > > so I apologize if this question has been asked before.
> > >
> > > I am looking for a why to set the dependency between two DAGs: a
> > "Producer"
> > > runs every day, appending data in a DB and a "Consumer" that runs
> weekly,
> > > generating a summary report of the new data.
> > >
> > > I would like to run the "Consumer" every Monday (for instance), but
> only
> > > after the Monday instance of the daily "Producer" is complete. What
> would
> > > be the best way to accomplish this?
> > >
> > > Thanks,
> > > Rafael
> > >
> >
>
Re: DAG dependencies on different frequencies
Posted by Chris Riccomini <cr...@apache.org>.
An alternative approach would be to use the ExternalTaskSensor to sense
when the task(s)/data that you depend upon has finished/arrived.
https://pythonhosted.org/airflow/code.html#airflow.operators.ExternalTaskSensor
On Mon, Jun 13, 2016 at 9:08 AM, Laura Lorenz <ll...@industrydive.com>
wrote:
> We do something like that here here in this way, and I'd love feedback from
> anyone else if they do it a different way. Basically we use
> TriggerDagRunOperators and a task in the "Producer" that cares what day of
> the week it is being run, and a "Consumer" DAG with no schedule interval
> that depends on being triggers when the right conditions of the "Producer"
> DAG are met.
>
> In more detail, we have a daily DAG like your "Producer" that ingests data
> every day. We want to run a series of tasks including generating a report
> in a "Consumer" DAG after after Monday's data has been processed; which
> would be on a Tuesday, after a run of "Producer" off the execution date of
> Monday.
>
> The last task in "Producer" is a TriggerDagRunOperator that only triggers
> IF it notices that now() is Tuesday. This triggers an instance at the right
> time of a child DAG of the "Consumer" type.
>
>
> On Mon, Jun 13, 2016 at 11:50 AM, Rafael Barbosa <rr...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > I tried looking around, but I cannot find an archive of this mailing
> list,
> > so I apologize if this question has been asked before.
> >
> > I am looking for a why to set the dependency between two DAGs: a
> "Producer"
> > runs every day, appending data in a DB and a "Consumer" that runs weekly,
> > generating a summary report of the new data.
> >
> > I would like to run the "Consumer" every Monday (for instance), but only
> > after the Monday instance of the daily "Producer" is complete. What would
> > be the best way to accomplish this?
> >
> > Thanks,
> > Rafael
> >
>
Re: DAG dependencies on different frequencies
Posted by Laura Lorenz <ll...@industrydive.com>.
We do something like that here here in this way, and I'd love feedback from
anyone else if they do it a different way. Basically we use
TriggerDagRunOperators and a task in the "Producer" that cares what day of
the week it is being run, and a "Consumer" DAG with no schedule interval
that depends on being triggers when the right conditions of the "Producer"
DAG are met.
In more detail, we have a daily DAG like your "Producer" that ingests data
every day. We want to run a series of tasks including generating a report
in a "Consumer" DAG after after Monday's data has been processed; which
would be on a Tuesday, after a run of "Producer" off the execution date of
Monday.
The last task in "Producer" is a TriggerDagRunOperator that only triggers
IF it notices that now() is Tuesday. This triggers an instance at the right
time of a child DAG of the "Consumer" type.
On Mon, Jun 13, 2016 at 11:50 AM, Rafael Barbosa <rr...@gmail.com>
wrote:
> Hi everyone,
>
> I tried looking around, but I cannot find an archive of this mailing list,
> so I apologize if this question has been asked before.
>
> I am looking for a why to set the dependency between two DAGs: a "Producer"
> runs every day, appending data in a DB and a "Consumer" that runs weekly,
> generating a summary report of the new data.
>
> I would like to run the "Consumer" every Monday (for instance), but only
> after the Monday instance of the daily "Producer" is complete. What would
> be the best way to accomplish this?
>
> Thanks,
> Rafael
>