You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Marcin Szymański <ms...@gmail.com> on 2020/11/25 00:22:06 UTC
Dag dependencies
Hi all
Just had a chat with Ry and I've seen popular demand in the past, so I
guess it might be a good time and idea to contribute my DAG dependencies
plugin (https://github.com/ms32035/airflow-dag-dependencies) to core
Airflow. For that I'd need some community guidance on the following points:
1) I guess the feature is not large enough to require AIP and simple PR
would be enough
2) Right now the plugin refreshes the dependencies using last view time +
timeout. Any suggestions about better solutions?
3) Are there any upcoming scheduler changes that might impact the
functionality, like partitioning etc?
4) Any other caveats I might be missing?
Thanks
Marcin
Re: Dag dependencies
Posted by Marcin Szymański <ms...@gmail.com>.
Thanks. I'll check the impact of DAG serialization.
The scalability issues I was referring to would be more likely during UI
rendering, but I never tested that at scale.
On Wed, 25 Nov 2020, 01:20 Kamil Breguła, <ka...@polidea.com> wrote:
> Hello,
>
> I have a feeling this plugin doesn't work when DAG serialization is
> enabled.
>
> https://github.com/ms32035/airflow-dag-dependencies/blob/30108e750e2008bc5f043210f8c7fa728fe5a630/dag-dependencies-plugin/dag_dependencies_plugin.py#L65
> In Airflow 2.0, serialization is required and cannot be disabled.
>
> > Be careful if you have 1000s of DAGs, might not scale.
>
> This is also problematic. It seems to me that we can come up with a
> solution that will scale, e.g. when all information will be written to the
> database by the scheduler and the webserver will read this information.
>
> Best. regards,
> Kamil Breguła
>
> On Wed, Nov 25, 2020 at 1:22 AM Marcin Szymański <ms...@gmail.com>
> wrote:
>
>> Hi all
>>
>> Just had a chat with Ry and I've seen popular demand in the past, so I
>> guess it might be a good time and idea to contribute my DAG dependencies
>> plugin (https://github.com/ms32035/airflow-dag-dependencies) to core
>> Airflow. For that I'd need some community guidance on the following points:
>> 1) I guess the feature is not large enough to require AIP and simple PR
>> would be enough
>> 2) Right now the plugin refreshes the dependencies using last view time +
>> timeout. Any suggestions about better solutions?
>> 3) Are there any upcoming scheduler changes that might impact the
>> functionality, like partitioning etc?
>> 4) Any other caveats I might be missing?
>>
>> Thanks
>> Marcin
>>
>>
>>
Re: Dag dependencies
Posted by Kamil Breguła <ka...@polidea.com>.
Hello,
I have a feeling this plugin doesn't work when DAG serialization is enabled.
https://github.com/ms32035/airflow-dag-dependencies/blob/30108e750e2008bc5f043210f8c7fa728fe5a630/dag-dependencies-plugin/dag_dependencies_plugin.py#L65
In Airflow 2.0, serialization is required and cannot be disabled.
> Be careful if you have 1000s of DAGs, might not scale.
This is also problematic. It seems to me that we can come up with a
solution that will scale, e.g. when all information will be written to the
database by the scheduler and the webserver will read this information.
Best. regards,
Kamil Breguła
On Wed, Nov 25, 2020 at 1:22 AM Marcin Szymański <ms...@gmail.com> wrote:
> Hi all
>
> Just had a chat with Ry and I've seen popular demand in the past, so I
> guess it might be a good time and idea to contribute my DAG dependencies
> plugin (https://github.com/ms32035/airflow-dag-dependencies) to core
> Airflow. For that I'd need some community guidance on the following points:
> 1) I guess the feature is not large enough to require AIP and simple PR
> would be enough
> 2) Right now the plugin refreshes the dependencies using last view time +
> timeout. Any suggestions about better solutions?
> 3) Are there any upcoming scheduler changes that might impact the
> functionality, like partitioning etc?
> 4) Any other caveats I might be missing?
>
> Thanks
> Marcin
>
>
>