New to Airflow

[jeff <je...@gmail.com>]

I am sure I am missing some basic concept but I am looking for how to do a
single operation that can fan out into many.  Say like I grab an rss feed
and there are 10 news articles.  I want to apply some kind of DAG on each
of those.  Is there a way to specify that or should the DAG that is pulling
the articles down simply trigger another DAG for each article.

Thank you in advance.

Re: New to Airflow

[Reed Villanueva <rv...@ucera.org>]

Newish to airflow as well, but here are some ideas...

DAGs definition files (the python files you put in the AIRFLOW_HOME/dags/
dir) are evaluated at the start of the job (see
https://stackoverflow.com/q/58567386/8236733), so if your rss-getting logic
happens mid-dag, the dag can't update itself dynamically to branch for each
article (so far as I know). You may have to have another dag that gets
triggered by the completion of this dag (see
https://www.astronomer.io/guides/trigger-dag-operator/) that reads to
article info from, say, a log file or json that gets written by the first
dag.

If getting the rss feed articles in the 1st thing your dag wants to do (or
if it does not depend on other preceding tasks), you could just have the
rss-getting logic in the dag definition file itself, put the article names
in a string list or dict (if want to use more info), and then use a loop to
create dag tasks for each article.

Also, just because you mention the idea of subdags, you can read more about
them in this article here (https://www.astronomer.io/guides/subdags/), but
I don't think this is going to be what you want.

On Fri, Nov 29, 2019 at 8:12 AM jeff <je...@gmail.com> wrote:

> I am sure I am missing some basic concept but I am looking for how to do a
> single operation that can fan out into many.  Say like I grab an rss feed
> and there are 10 news articles.  I want to apply some kind of DAG on each
> of those.  Is there a way to specify that or should the DAG that is pulling
> the articles down simply trigger another DAG for each article.
>
> Thank you in advance.
>

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.