Re: Handling running tasks

[siddharth anand <sa...@apache.org>]

+1 on the uniqueness of the solution. Wondering how it worked!
-s


On Sun, Jul 10, 2016 at 9:02 AM, Cyril Scetbon <cy...@free.fr>
wrote:

> Interesting. Thanks for this solution Lance, gonna try it
> > On Jul 6, 2016, at 19:11, Lance Norskog <la...@gmail.com> wrote:
> >
> > You could use the XCOM feature to post a semaphore at the start of the
> task
> > and then remove it at the end.
> > Another task would see the semaphore and immediately quit.
> > If you get a race condition, the third 15-minute task will take care of
> the
> > problem, 15 minutes late.
> >
> > Lance
> >
> > On Wed, Jul 6, 2016 at 2:50 PM, Cyril Scetbon <cy...@free.fr>
> wrote:
> >
> >> Hi,
> >>
> >> I have dags with tasks that use same configuration and same schedule
> time
> >> frequencies.
> >>
> >> When I have x tasks in a dag they run in //. My dag is scheduled every
> 15
> >> minutes, but sometimes I have at least a task that runs for more than 15
> >> minutes and 2 identical tasks should not run at the same time (in my
> case)
> >> but it's what happens.
> >>
> >> So when the dag is kicked off (every 15 min) I need Airflow to run tasks
> >> only for those that are not running, which means :
> >>
> >> - if at time t1 tasks tk1 is running and tk2, ..., tkn are not running,
> I
> >> need Airflow to run only tk2,.., tkn but not tk1 cause it's already
> running
> >> - if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to
> run
> >> tk1,.., tkn
> >>
> >> I already tried using depends_on_past=True, however I need failed tasks
> to
> >> be kicked off, cause I can get some temporary issues.
> >>
> >> I also use an upstream task connected to all tasks in my dag to be able
> to
> >> run manually all tasks if I need to. (I don't use a frequency of 15
> minutes
> >> in all dags)
> >>
> >> Any idea ?
> >
> >
> >
> >
> > --
> > Lance Norskog
> > lance.norskog@gmail.com
> > Redwood City, CA
>
>