You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Chandu Kavar <cc...@gmail.com> on 2018/09/11 07:11:23 UTC

Delay between tasks executions issue

Hi All,

We are seeing there is a delay between triggering the next task. Please
find the attached screenshot.

First and the last task are very small. So, you can see the tiny object.

[image: image.png]
Do you suggest any airflow config can resolve this problem?

Thanks,
Chandu

Re: Delay between tasks executions issue

Posted by Chandu Kavar <cc...@gmail.com>.

Thanks a lot.

On Tue, Sep 11, 2018, 6:25 PM ramandumcs@gmail.com <ra...@gmail.com>
wrote:

> We use max_threads = number of scheduler cores.
>
> On 2018/09/11 09:49:53, Chandu Kavar <cc...@gmail.com> wrote:
> > Thanks Raman,
> >
> > Understood.
> >
> > We have around 500 DAGs. What value do you suggest for max_threads?
> >
> > On Tue, Sep 11, 2018, 5:44 PM ramandumcs@gmail.com <ramandumcs@gmail.com
> >
> > wrote:
> >
> > > Hi Chandu,
> > > How many dag files are there on the scheduler. As per my understanding
> > > scheduler processes each file to trigger any dag/task run. It spawns
> number
> > > of processes equivalent to "max_threads" count to parallelize file
> > > processing. So you can try by increasing airflow config's max_threads
> count.
> > >
> > > Thanks,
> > > Raman Gupta
> > >
> > > On 2018/09/11 07:11:23, Chandu Kavar <cc...@gmail.com> wrote:
> > > > Hi All,
> > > >
> > > > We are seeing there is a delay between triggering the next task.
> Please
> > > > find the attached screenshot.
> > > >
> > > > First and the last task are very small. So, you can see the tiny
> object.
> > > >
> > > > [image: image.png]
> > > > Do you suggest any airflow config can resolve this problem?
> > > >
> > > > Thanks,
> > > > Chandu
> > > >
> > >
> >
>

Re: Delay between tasks executions issue

Posted by ra...@gmail.com, ra...@gmail.com.

We use max_threads = number of scheduler cores.

On 2018/09/11 09:49:53, Chandu Kavar <cc...@gmail.com> wrote: 
> Thanks Raman,
> 
> Understood.
> 
> We have around 500 DAGs. What value do you suggest for max_threads?
> 
> On Tue, Sep 11, 2018, 5:44 PM ramandumcs@gmail.com <ra...@gmail.com>
> wrote:
> 
> > Hi Chandu,
> > How many dag files are there on the scheduler. As per my understanding
> > scheduler processes each file to trigger any dag/task run. It spawns number
> > of processes equivalent to "max_threads" count to parallelize file
> > processing. So you can try by increasing airflow config's max_threads count.
> >
> > Thanks,
> > Raman Gupta
> >
> > On 2018/09/11 07:11:23, Chandu Kavar <cc...@gmail.com> wrote:
> > > Hi All,
> > >
> > > We are seeing there is a delay between triggering the next task. Please
> > > find the attached screenshot.
> > >
> > > First and the last task are very small. So, you can see the tiny object.
> > >
> > > [image: image.png]
> > > Do you suggest any airflow config can resolve this problem?
> > >
> > > Thanks,
> > > Chandu
> > >
> >
>

Re: Delay between tasks executions issue

Posted by Chandu Kavar <cc...@gmail.com>.

Thanks Raman,

Understood.

We have around 500 DAGs. What value do you suggest for max_threads?

On Tue, Sep 11, 2018, 5:44 PM ramandumcs@gmail.com <ra...@gmail.com>
wrote:

> Hi Chandu,
> How many dag files are there on the scheduler. As per my understanding
> scheduler processes each file to trigger any dag/task run. It spawns number
> of processes equivalent to "max_threads" count to parallelize file
> processing. So you can try by increasing airflow config's max_threads count.
>
> Thanks,
> Raman Gupta
>
> On 2018/09/11 07:11:23, Chandu Kavar <cc...@gmail.com> wrote:
> > Hi All,
> >
> > We are seeing there is a delay between triggering the next task. Please
> > find the attached screenshot.
> >
> > First and the last task are very small. So, you can see the tiny object.
> >
> > [image: image.png]
> > Do you suggest any airflow config can resolve this problem?
> >
> > Thanks,
> > Chandu
> >
>

Re: Delay between tasks executions issue

Posted by ra...@gmail.com, ra...@gmail.com.

Hi Chandu,
How many dag files are there on the scheduler. As per my understanding scheduler processes each file to trigger any dag/task run. It spawns number of processes equivalent to "max_threads" count to parallelize file processing. So you can try by increasing airflow config's max_threads count.

Thanks,
Raman Gupta 

On 2018/09/11 07:11:23, Chandu Kavar <cc...@gmail.com> wrote: 
> Hi All,
> 
> We are seeing there is a delay between triggering the next task. Please
> find the attached screenshot.
> 
> First and the last task are very small. So, you can see the tiny object.
> 
> [image: image.png]
> Do you suggest any airflow config can resolve this problem?
> 
> Thanks,
> Chandu
>