You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by lunbingli <lu...@163.com> on 2020/11/24 03:57:08 UTC
Why is Airflow 1.10.12 much slower than 1.10.10?
Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my dags are scheduled much slower than before. One of my dag is as follows（please check attachment） and it has 400 Tasks of DummyOperator. It took only 4+mins with 1.10.10 while it took more than 16mins with 1.10.12.
Can someone tell me what the problem may be？I tried to compare the code diff between these two versions with https://github.com/apache/airflow/compare/1.10.10...1.10.12 and failed to draw a conclusion yet.
Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by Ash Berlin-Taylor <as...@apache.org>.
I can't say to what the difference might be between those versions, but
the great news is that such a dag on 2.0.0 (currently in beta3) is
/much much/ faster, for two reasons.
1. DummyOperators aren't executed anymore if they don't have any
callbacks -- the scheduler just marks them directly as success. ;)
2. My performance improvement work has been merged, and you should see
a significant performance improvement - I wouldn't be surprised to find
that dag run in seconds on 2.0.0beta3.
Are you able to try out this beta?
Thanks,
Ash
On Tue, 24 Nov, 2020 at 11:57, lunbingli <lu...@163.com> wrote:
> Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find
> my dags are scheduled much slower than before. One of my dag is as
> follows（please check attachment） and it has 400 Tasks of
> DummyOperator.*It took only 4+mins with 1.10.10 while it took more
> than 16mins with 1.10.12.*
>
> *Can someone tell me what the problem may be？*I tried to compare
> the code diff between these two versions with
> https://github.com/apache/airflow/compare/1.10.10...1.10.12
> <https://github.com/apache/airflow/discussions/url> and failed to
> draw a conclusion yet.
>
>
>
>
Re:Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by lunbingli <lu...@163.com>.
1.10.13 works fine and performs better than 1.10.10. Thanks for your help! @Kaxil Naik @Ash
在 2020-11-24 21:08:26，"lunbingli" <lu...@163.com> 写道：
Thanks。I will try 1.10.13
On 11/24/2020 20:22, Kaxil Naik wrote:
Can you test your DAG with 1.10.13rc1, it contains some performance improvements that should take care of it.
You can install it as follows:
pip install apache-airflow==1.10.13rc1
Regards,
Kaxil
On Tue, Nov 24, 2020 at 3:57 AM lunbingli <lu...@163.com> wrote:
Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my dags are scheduled much slower than before. One of my dag is as follows（please check attachment） and it has 400 Tasks of DummyOperator. It took only 4+mins with 1.10.10 while it took more than 16mins with 1.10.12.
Can someone tell me what the problem may be？I tried to compare the code diff between these two versions with https://github.com/apache/airflow/compare/1.10.10...1.10.12 and failed to draw a conclusion yet.
Re:Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by lunbingli <lu...@163.com>.
1.10.13 works fine and performs better than 1.10.10. Thanks for your help! @Kaxil Naik @Ash
在 2020-11-24 21:08:26，"lunbingli" <lu...@163.com> 写道：
Thanks。I will try 1.10.13
On 11/24/2020 20:22, Kaxil Naik wrote:
Can you test your DAG with 1.10.13rc1, it contains some performance improvements that should take care of it.
You can install it as follows:
pip install apache-airflow==1.10.13rc1
Regards,
Kaxil
On Tue, Nov 24, 2020 at 3:57 AM lunbingli <lu...@163.com> wrote:
Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my dags are scheduled much slower than before. One of my dag is as follows（please check attachment） and it has 400 Tasks of DummyOperator. It took only 4+mins with 1.10.10 while it took more than 16mins with 1.10.12.
Can someone tell me what the problem may be？I tried to compare the code diff between these two versions with https://github.com/apache/airflow/compare/1.10.10...1.10.12 and failed to draw a conclusion yet.
Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by lunbingli <lu...@163.com>.
Thanks。I will try 1.10.13
On 11/24/2020 20:22, Kaxil Naik wrote:
Can you test your DAG with 1.10.13rc1, it contains some performance improvements that should take care of it.
You can install it as follows:
pip install apache-airflow==1.10.13rc1
Regards,
Kaxil
On Tue, Nov 24, 2020 at 3:57 AM lunbingli <lu...@163.com> wrote:
Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my dags are scheduled much slower than before. One of my dag is as follows（please check attachment） and it has 400 Tasks of DummyOperator. It took only 4+mins with 1.10.10 while it took more than 16mins with 1.10.12.
Can someone tell me what the problem may be？I tried to compare the code diff between these two versions with https://github.com/apache/airflow/compare/1.10.10...1.10.12 and failed to draw a conclusion yet.
Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by lunbingli <lu...@163.com>.
Thanks。I will try 1.10.13
On 11/24/2020 20:22, Kaxil Naik wrote:
Can you test your DAG with 1.10.13rc1, it contains some performance improvements that should take care of it.
You can install it as follows:
pip install apache-airflow==1.10.13rc1
Regards,
Kaxil
On Tue, Nov 24, 2020 at 3:57 AM lunbingli <lu...@163.com> wrote:
Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my dags are scheduled much slower than before. One of my dag is as follows（please check attachment） and it has 400 Tasks of DummyOperator. It took only 4+mins with 1.10.10 while it took more than 16mins with 1.10.12.
Can someone tell me what the problem may be？I tried to compare the code diff between these two versions with https://github.com/apache/airflow/compare/1.10.10...1.10.12 and failed to draw a conclusion yet.
Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by Kaxil Naik <ka...@gmail.com>.
Can you test your DAG with 1.10.13rc1, it contains some performance
improvements that should take care of it.
You can install it as follows:
*pip install apache-airflow==1.10.13rc1*
Regards,
Kaxil
On Tue, Nov 24, 2020 at 3:57 AM lunbingli <lu...@163.com> wrote:
> Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my
> dags are scheduled much slower than before. One of my dag is as
> follows（please check attachment） and it has 400 Tasks of DummyOperator.*
> It took only 4+mins with 1.10.10 while it took more than 16mins with
> 1.10.12.*
> *Can someone tell me what the problem may be？*I tried to compare the code
> diff between these two versions with
> https://github.com/apache/airflow/compare/1.10.10...1.10.12
> <https://github.com/apache/airflow/discussions/url> and failed to draw a
> conclusion yet.
>
>
>
>
Re: Why is Airflow 1.10.12 much slower than 1.10.10?
Posted by Kaxil Naik <ka...@gmail.com>.
Can you test your DAG with 1.10.13rc1, it contains some performance
improvements that should take care of it.
You can install it as follows:
*pip install apache-airflow==1.10.13rc1*
Regards,
Kaxil
On Tue, Nov 24, 2020 at 3:57 AM lunbingli <lu...@163.com> wrote:
> Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my
> dags are scheduled much slower than before. One of my dag is as
> follows（please check attachment） and it has 400 Tasks of DummyOperator.*
> It took only 4+mins with 1.10.10 while it took more than 16mins with
> 1.10.12.*
> *Can someone tell me what the problem may be？*I tried to compare the code
> diff between these two versions with
> https://github.com/apache/airflow/compare/1.10.10...1.10.12
> <https://github.com/apache/airflow/discussions/url> and failed to draw a
> conclusion yet.
>
>
>
>