Atomic update of in-progress dags

[Mandeep Sandhu <ma...@gmail.com>]

Hi All,

Relative newbie here.

Is it possible (or is there a known pattern) for allowing atomic
updates of DAGs?

By atomic I mean, if a DAG is in progress running some version of the
code, then it should continue to use it even if the DAG code is
changed. The change should be used up only for new DAG runs started
after the update.

Eg I have a dag setup as:
A ---> B ---> C
Each step produces an intermediate build artifact which is used as
input by the next stage. If the DAG step is currently executing "B"
and I update the DAG code, I dont want the changes to be picked up
until "C" completes.

Does airflow controller provide any such feature or is this something
that each developer handles on their own?

Thanks for your time.

-mandeep

Re: Atomic update of in-progress dags

[Mandeep Sandhu <ma...@gmail.com>]

> No. There is no such feature yet. Assumption is that DAGs change "slowly" and without heavy structural changes that would do any harm in those situations.
>
> If you need "heavy" change to a DAG, you need to create a new DAG with new dag_id - this is current recommendation when it comes to "dag versioning"

Thanks for the suggestion. I'll look into deploying DAG with a new
version instead of updating the existing one.

>
> There is an Airflow Improvement Proposal https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning about DAG versioning where such atomic changes will likely become a reality - but it has not been worked on for quite some time (but this is is likely to change in the coming months and work on DAG Versioning will resume - or so I hear).

I'll keep an eye on its progress. Thanks for the link!

-mandeep


>
> J.
>
>
> On Thu, May 12, 2022 at 11:57 PM Mandeep Sandhu <ma...@gmail.com> wrote:
>>
>> Hi All,
>>
>> Relative newbie here.
>>
>> Is it possible (or is there a known pattern) for allowing atomic
>> updates of DAGs?
>>
>> By atomic I mean, if a DAG is in progress running some version of the
>> code, then it should continue to use it even if the DAG code is
>> changed. The change should be used up only for new DAG runs started
>> after the update.
>>
>> Eg I have a dag setup as:
>> A ---> B ---> C
>> Each step produces an intermediate build artifact which is used as
>> input by the next stage. If the DAG step is currently executing "B"
>> and I update the DAG code, I dont want the changes to be picked up
>> until "C" completes.
>>
>> Does airflow controller provide any such feature or is this something
>> that each developer handles on their own?
>>
>> Thanks for your time.
>>
>> -mandeep

Re: Atomic update of in-progress dags

[Jarek Potiuk <ja...@potiuk.com>]

No. There is no such feature yet. Assumption is that DAGs change "slowly"
and without heavy structural changes that would do any harm in those
situations.

If you need "heavy" change to a DAG, you need to create a new DAG with new
dag_id - this is current recommendation when it comes to "dag versioning"

There is an Airflow Improvement Proposal
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning
about DAG versioning where such atomic changes will likely become a reality
- but it has not been worked on for quite some time (but this is is likely
to change in the coming months and work on DAG Versioning will resume - or
so I hear).

J.


On Thu, May 12, 2022 at 11:57 PM Mandeep Sandhu <ma...@gmail.com>
wrote:

> Hi All,
>
> Relative newbie here.
>
> Is it possible (or is there a known pattern) for allowing atomic
> updates of DAGs?
>
> By atomic I mean, if a DAG is in progress running some version of the
> code, then it should continue to use it even if the DAG code is
> changed. The change should be used up only for new DAG runs started
> after the update.
>
> Eg I have a dag setup as:
> A ---> B ---> C
> Each step produces an intermediate build artifact which is used as
> input by the next stage. If the DAG step is currently executing "B"
> and I update the DAG code, I dont want the changes to be picked up
> until "C" completes.
>
> Does airflow controller provide any such feature or is this something
> that each developer handles on their own?
>
> Thanks for your time.
>
> -mandeep
>