Regarding airflow scheduler starting PID periodically.

[nitish sharma <ni...@gmail.com>]

Hi Team,

I have a situation where i want to create dag with unique dag id and run
that dag once as soon as it is created.

For eg :  1.py , 2.py are two different dags. 1.py (has 3 tasks) and
2.py(has 2 tasks).

Once these are run i would have next set of dags with me for eg 3.py, 4.py,
5.py and so on.

I see a challenger here as scheduler keeps generating process ID to
generate tasks

Re: Regarding airflow scheduler starting PID periodically.

[Taylor Edmiston <te...@gmail.com>]

If it is different than the TriggerDagRunOperator idea that Guillermo
mentioned, could you elaborate on your specific use case?

On Thu, May 10, 2018 at 2:40 AM Guillermo Rodríguez Cano <ws...@gmail.com>
wrote:

> Hello,
>
> I am not sure if I understood... you want to trigger DAGs based on the PIDs
> of the tasks?
>
> As you say (I am not an expert on OSes here) you don't control this.
>
> I think that the TriggerDagRunOperator (https://airflow.apache.org/co
> de.html#airflow.operators.TriggerDagRunOperator
> <https://airflow.apache.org/code.html#airflow.operators.TriggerDagRunOperator>)
> would work for you. Right?
>
> /Guillermo
>
>
> On Tue, May 8, 2018 at 5:06 PM nitish sharma <ni...@gmail.com>
> wrote:
>
> > Hi Team,
> >
> > I have a situation where i want to create dag with unique dag id and run
> > that dag once as soon as it is created.
> >
> > For eg :  1.py , 2.py are two different dags. 1.py (has 3 tasks) and
> > 2.py(has 2 tasks).
> >
> > Once these are run i would have next set of dags with me for eg 3.py,
> 4.py,
> > 5.py and so on.
> >
> > I see a challenger here as scheduler keeps generating process ID to
> > generate tasks
> >
> >
>
-- 
*Taylor Edmiston*
Blog <http://blog.tedmiston.com> | Stack Overflow CV
<https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>

Re: Regarding airflow scheduler starting PID periodically.

[Guillermo Rodríguez Cano <ws...@gmail.com>]

Hello,

I am not sure if I understood... you want to trigger DAGs based on the PIDs
of the tasks?

As you say (I am not an expert on OSes here) you don't control this.

I think that the TriggerDagRunOperator (https://airflow.apache.org/co
de.html#airflow.operators.TriggerDagRunOperator) would work for you. Right?

/Guillermo


On Tue, May 8, 2018 at 5:06 PM nitish sharma <ni...@gmail.com> wrote:

> Hi Team,
>
> I have a situation where i want to create dag with unique dag id and run
> that dag once as soon as it is created.
>
> For eg :  1.py , 2.py are two different dags. 1.py (has 3 tasks) and
> 2.py(has 2 tasks).
>
> Once these are run i would have next set of dags with me for eg 3.py, 4.py,
> 5.py and so on.
>
> I see a challenger here as scheduler keeps generating process ID to
> generate tasks
>
>