You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "gbonazzoli (via GitHub)" <gi...@apache.org> on 2025/08/11 09:36:35 UTC
[D] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
GitHub user gbonazzoli created a discussion: Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS
### Apache Airflow version
3.0.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have a Python Source that generate `PythonOperator` Tasks on a DAG.
It is the load of the Staging Area in a Datawarehouse, so during this phase there are hundreds of tables that we need to load from the transactional database to the analytical one, Airflow limits the load through `max_active_tasks=32` DAG parameter.
The code is:
```python
with TaskGroup(group_id="MOVEX100") as DLTHUB_MOVEX100:
for source_table_list in [SOURCE_MOVEX100_TABLES]:
for source_table_obj in source_table_list:
table_name = source_table_obj["table_name"]
dlt_query = source_table_obj["dlt_query"]
dlt_load_movex100 = PythonOperator(
task_id="dlt_movex100_" + table_name,
python_callable=execute_dlt_movex100,
retries=3, # Retry up to 3 times (original + 3 retries)
retry_delay=timedelta(minutes=randrange(5) + 1), # Wait from 1 to 5 minutes between retries
templates_dict={
"table_name": f"{table_name}",
"dlt_query": f"{dlt_query}",
},
)
Beg_Load_Staging >> dlt_load_movex100 >> End_Load_Staging
```
The problem with this code is that every days Airflow 3.x registers hundreds of different versions for the DAG.
The reality is that is dynamically generated and so it is obvious that changes dynamically.
Is there anyway to limit the problem ?
### What you think should happen instead?
_No response_
### How to reproduce
I think that is a general problem afflicting Airflow 3.x with DAG that are dinamically generated through code.
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.6.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.4
apache-airflow-providers-fab==2.3.1
apache-airflow-providers-git==0.0.5
apache-airflow-providers-postgres==6.2.2
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-salesforce==5.11.2
apache-airflow-providers-smtp==2.1.2
apache-airflow-providers-ssh==4.1.2
apache-airflow-providers-standard==1.5.0
```
### Deployment
Virtualenv installation
### Deployment details
An ubuntu vanilla installation in python 3.12 virtual env.
### Anything else?
no
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/54348
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
> The reality is that is dynamically generated and so it is obvious that changes dynamically.
That's not true in general.
If your intention to generate a different dag every time it is parsed, then yes it will be dynamically changing and. as far as I know - you can disable versioning https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#disable-bundle-versioning for that bundle if your bundle contains DAG that change frequently and it is expected (but this is generally a bad practice)
Dynamically generated dags are NOT supposed to dynamically change every time they are parsed. They MAY change slower or faster - depending on how often the "input" to those dynamic dags change. For example if you have list of countries, and you generate dynamic dags based on that - your parsed dag should change only when you add a new country - every time (providing that your code to generate dynamic dag is well written) , parsing the same Dag **should** produce the same - dynamically generated - parsed representation.
So in your case you likely:
a) your intention is to change your dag every time it is parssed (which is quite strange, but then you should disable versioning for that bundle)
b) there is a problem with dynamic dag generation code - it **should** produce the same parsed representation, every time Dag is parsed.
There might be two resons for b)
1) your dynamic dag generation has problems and does not produce stable dag object structure or stable parameters. This might change for example when you do not use `fixed` start date - but use `days_ago` or similar - which is a bad practice, because days_ago() will change every time your dag is parsed. Or you can pass time() or other dynamically generated (and ever changing) content as part of your dag definition. Or you create objects in a random sequence every time you build your dynamic dag. This might happen for example when you run "SELECT * FROM" and do not ORDER the select - because the databases return data in such select in random order in general, and the order might change any time.
2) there is a bug in our Dag serialization code
Most likely it's 1) and you have to fix your dynamic dag generation to be "stable" if you want to version dag. Which means that serialized version of your Dag should be THE SAME if Dag parsing on your Dag is executed - if none of the inputs changed - and they should remain stable over time, so they should be independent on the time when they are parsed.
It could also be 2) - but in order to know whether it's 1) or 2) you have to look at the serialized dag table (see https://airflow.apache.org/docs/apache-airflow/stable/database-erd-ref.html ) find two versions of your Dags and find out what's the difference betwen the dags and what kind of "stability" issue in dynamic dag generation causes it. The "compressed" representation in this table is compressed with `.gzip` so you need to retrieve the blobs and decompress them -> you will get .json representation of the dags that you should be able to compare and see where the problem is coming from.
Also it might be easier for you if you review your dynamic dag generation looking at potential stability issues from those described above.
Once you find the reasons - please let us know - it would be interesting to find out what stability issues people like you might have - or maybe you will find some of the problems coming from our serializaiton. So we are looking forward for your investigation.
Converting it into a discussion until you investigate it and provide more information.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069763
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "JoseBueno-plytix (via GitHub)" <gi...@apache.org>.
GitHub user JoseBueno-plytix added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Hello, in relation to this discussion, is it expected behaviour for a serialization to be created eachtime a Dag is run, changing only the task `op_args`. Comapring the serialization of two verions as you suggested, this field `"op_args": "(<airflow.sdk.definitions.param.DagParam object at 0x7fb7bf966750>,)"` seems to be the only difference, and the direction of the DagParam object is what is changing.
I was wondering if this may be cause by a bad written DAG or something else.
Thanks in advance.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14223136
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
It does not do you any good when you pass an object to parameters that is supposed to be string. As simple as that.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14225206
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Maybe even - eventually - we might come up with better tooling to analyse such dynamic dag "stabilty" issues - but I find this case an intersting case study to see what could be the reasons, so that we can write our tooling to be useful to quickly analyse such issues.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069813
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS
> The reality is that is dynamically generated and so it is obvious that changes dynamically.
That's not true in general.
If your intention to generate a different dag every time it is parsed, then yes it will be dynamically changing and. as far as I know - you can disable versioning https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#disable-bundle-versioning for that bundle if your bundle contains DAG that change frequently and it is expected (but this is generally a bad practice)
Dynamically generated dags are NOT supposed to dynamically change every time they are parsed. They MAY change slower or faster - depending on how often the "input" to those dynamic dags change. For example if you have list of countries, and you generate dynamic dags based on that - your parsed dag should change only when you add a new country - every time (providing that your code to generate dynamic dag is well written) , parsing the same Dag **should** produce the same - dynamically generated - parsed representation.
So in your case you likely:
a) your intention is to change your dag every time it is parssed (which is quite strange, but then you should disable versioning for that bundle)
b) there is a problem with your dynamic dag generation code - it **should** produce the same parsed representation, every time Dag is parsed.
There might be two resons for b)
1) your dynamic dag generation has problems and does not produce stable dag object structure or stable parameters. This might change for example when you do not use 'fixed` start date - but use `days_ago` or similar - which is a bad practice, because days_ago() will change every time your dag is parsed. Or you can pass time() or other dynamically generated (and ever changing) content as part of your dag definition. Or you create objects in a random sequence every time you build your dynamic dag. This might happen for example when you run "SELECT * FROM" and do not ORDER the select - because the databases return data in such select in random order in general, and the order might change any time.
2) there is a bug in our Dag serialization code
Most likely it's 1) and you have to fix your dynamic dag generation to be "stable" if you want to version dag. Which means that serialized version of your Dag should be THE SAME if Dag parsing on your Dag is executed - if none of the inputs changed - and they should remain stable over time, so they should be independent on the time when they are parsed.
It could also be 2) - but in order to know whether it's 1) or 2) you have to look at the serialized dag table (see https://airflow.apache.org/docs/apache-airflow/stable/database-erd-ref.html ) find two versions of your Dags and find out what's the difference betwen the dags and what kind of "stability" issue in dynamic dag generation causes it. The "compressed" representation in this table is compressed with `.gzip` so you need to retrieve the blobs and decompress them -> you will get .json representation of the dags that you should be able to compare and see where the problem is coming from.
Also it might be easier for you if you review your dynamic dag generation looking at potential stability issues from those described above.
Once you find the reasons - please let us know - it would be interesting to find out what stability issues people like you might have - or maybe you will find some of the problems coming from our serializaiton. So we are looking forward for your investigation.
Converting it into a discussion until you investigate it and provide more information.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069763
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "JoseBueno-plytix (via GitHub)" <gi...@apache.org>.
GitHub user JoseBueno-plytix added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Using pydantic for assuring the right types, maybe i am missing something, but i don't know why it is supossed to be an string, I define it a as an int
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14225588
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "JoseBueno-plytix (via GitHub)" <gi...@apache.org>.
GitHub user JoseBueno-plytix added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
When you say "any change", do you mean change in the parameters (like adding/removing parameters, which makes perfect sense) or do you refere to the value of said parameters? For example, a simplification of the DAG im currently testing for this, it recives a int as a parameter that then passes to a task to perform some action with that number, even when running the DAG with the same input, it creates a new version.
What i mean by direction is the memory direction where the `DagParam` object is located (`DagParam object at 0x7fb7bf966750`)
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14223831
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
you can - for example - use `mypy` or other type checkers to verify if your code passes the right types of params
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14225211
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
> The reality is that is dynamically generated and so it is obvious that changes dynamically.
That's not true in general.
If your intention to generate a different dag every time it is parsed, then yes it will be dynamically changing and. as far as I know - you can disable versioning https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#disable-bundle-versioning for that bundle if your bundle contains DAG that change frequently and it is expected (but this is generally a bad practice)
Dynamically generated dags are NOT supposed to dynamically change every time they are parsed. They MAY change slower or faster - depending on how often the "input" to those dynamic dags change. For example if you have list of countries, and you generate dynamic dags based on that - your parsed dag should change only when you add a new country - every time (providing that your code to generate dynamic dag is well written) , parsing the same Dag **should** produce the same - dynamically generated - parsed representation.
So in your case you likely:
a) your intention is to change your dag every time it is parssed (which is quite strange, but then you should disable versioning for that bundle)
b) there is a problem with dynamic dag generation code - it **should** produce the same parsed representation, every time Dag is parsed.
There might be two resons for b)
1) your dynamic dag generation has problems and does not produce stable dag object structure or stable parameters. This might change for example when you do not use 'fixed` start date - but use `days_ago` or similar - which is a bad practice, because days_ago() will change every time your dag is parsed. Or you can pass time() or other dynamically generated (and ever changing) content as part of your dag definition. Or you create objects in a random sequence every time you build your dynamic dag. This might happen for example when you run "SELECT * FROM" and do not ORDER the select - because the databases return data in such select in random order in general, and the order might change any time.
2) there is a bug in our Dag serialization code
Most likely it's 1) and you have to fix your dynamic dag generation to be "stable" if you want to version dag. Which means that serialized version of your Dag should be THE SAME if Dag parsing on your Dag is executed - if none of the inputs changed - and they should remain stable over time, so they should be independent on the time when they are parsed.
It could also be 2) - but in order to know whether it's 1) or 2) you have to look at the serialized dag table (see https://airflow.apache.org/docs/apache-airflow/stable/database-erd-ref.html ) find two versions of your Dags and find out what's the difference betwen the dags and what kind of "stability" issue in dynamic dag generation causes it. The "compressed" representation in this table is compressed with `.gzip` so you need to retrieve the blobs and decompress them -> you will get .json representation of the dags that you should be able to compare and see where the problem is coming from.
Also it might be easier for you if you review your dynamic dag generation looking at potential stability issues from those described above.
Once you find the reasons - please let us know - it would be interesting to find out what stability issues people like you might have - or maybe you will find some of the problems coming from our serializaiton. So we are looking forward for your investigation.
Converting it into a discussion until you investigate it and provide more information.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069763
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Thanks for testing it!
> * if you are reasoning like the "git command" this is an Airflow's "subtle" bug.
> * If you are reasoning looking at the final results of the "compiled dynamic code" this is the perfectly expected behaviour.
The answer is easy. Option 2. It's always been the goal of DAG versioning - and you can read more about it in Airflow Improvement proposals (3 of them) describing the idea, implementaiton, goals, and future of Dag versioning - some of the things there are not yet implemented 100% https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning
> Actually if you talk to a developer about "code's versioning" I think that the common mindset is totally "git influenced".
So if we align Airflow to this mindset it is much simpler to explain and document the mechanism... but probably it is not so simple to implement it in the "dag-processor".
No. It's not a question of simpllicity. It's a question of deliberate choice and probably several tens of discussion points and countless hours of talking and agreeing on the direction by those who proposed, discussed, got approved and finally implemented it. This reflects the very "Fabric" of Airflow - where from the very beginning, dynamig Dag generation actually allowed to effectively have different versions of dag (depending on the input parameters). But before Dag versioning it has never been captured and caused all kind of - mostly - ui and rerun / backfill problems when Dynamically generated dags changed.
So - from the very beginnig of the Dag Versioning discussions (and if you look at the AIPs they have linked in discussions at devlist and those discussions refer not only to other discussions but also to previous version of Dag Versioning proposal - rejected and abandoned https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - this was very well discussed and agreed to.
Now - It seems you indeed pointed to a problem - that people might understand it differently. There is no reason or possibility to change it to the "git" approach, but since you now understand and got explained how it works we would be very happy if you could help to make it clearer:
1) Updating our documentation to make it more explicit and clearer. People like you, who got burned and understood things differently (but are smart and understand the reasoning now) are the best to put it in the words and describing things in the way that other people like them are not making the same (wrong) assumptions you did. Simply - after all the years of discussing it, things are "obvious" for maintainers and creators - but they are "non obvious" for newcomers, so adding a PR where you could describe is better - is golden (and fantastic opportunity to contribute back for the free software you get to use).
2) Describing your and similar use-cases to make it easier for people to see and investigate their issues when they hit similar things - even if it is starting "FAQ" or "common mistakes" or smth like that - coming from someone who had, investigated, and found the issues after having potential reasons explained by maintainer - is again golden and fantastic opportunity to contribute a PR
3) More complex and sophisticated - figuring out some ways of automated detection and flagging similar problems in Dags written by people would be a fantastic opportunity to contribute back "code" contribution. There are many ways and many "bad practices" that could be flagged there - it could be based on static code analysis, or detecting situations where indeed every time you parse you have different versions and it keeps on changing continuously .. World is your oyster. Another great opportunity to contribute.
Adding doc PR is super easy "suggest a change on this page" button (bottom-right) opens a PR where you can add documentation - that could be first step, later leading to a code contribution..
That would be fantastic output of that discussion.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14093597
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "dor-bernstein (via GitHub)" <gi...@apache.org>.
GitHub user dor-bernstein added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
@potiuk any chance it will lead to a case where airflow doesn't show the grid of the dag? the graph view works find but the grid is not rendering
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14440099
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "JoseBueno30 (via GitHub)" <gi...@apache.org>.
GitHub user JoseBueno30 deleted a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Using pydantic for assuring the right types, maybe i am missing something, but i don't know why it is supossed to be an string, I define it a as an int
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14225585
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
I also updated the title to properly reflect it - this is not a "huge problem with dag versioning", it's either lack of understanding that "dynamically generated dags" should have "stable" property if you want to version them or some specific bug in serializaiton code.
I hope as result of that discussion maybe you @gbonazzoli -> when you find reasons for that stability issues you can help us to write good documentation describing what "stability" property of such dynamically generated Dags is - or maybe we will even find some bugs in the implementation.
But this is NOT huge problenm with versioning in general, it works as intended, and it **should** work well with dynamically generated Dags - providing of course that those Dags are properly generated and that there are no bugs.
Looking forward to results of your investigation @gbonazzoli
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069799
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "gbonazzoli (via GitHub)" <gi...@apache.org>.
GitHub user gbonazzoli edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
@potiuk
I made some tests and I found that the problem was caused by one clause in the `PythonOperator` definition:
`retry_delay=timedelta(minutes=randrange(5) + 1)`
after having set it up to a static formula:
`retry_delay=timedelta(minutes=1)`
the versioning mechanism is now OK, and Airflow do not increment version number hundreds of times a day.
**My opinion is**:
- if you are reasoning like the "git command" this is an Airflow's "subtle" bug.
- If you are reasoning looking at the final results of the "compiled dynamic code" this is the perfectly expected behaviour.
Actually if you talk to a developer about "code's versioning" I think that the common mindset is totally "git influenced".
So if we align Airflow to this mindset it is much simpler to explain and document the mechanism... but probably it is not so simple to implement it in the "dag-processor".
What do you think ?
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14079190
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "JoseBueno-plytix (via GitHub)" <gi...@apache.org>.
GitHub user JoseBueno-plytix edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Thanks for your fast response.
When you say "any change", do you mean change in the parameters (like adding/removing parameters, which makes perfect sense) or do you refere to the value of said parameters? For example, a simplification of the DAG im currently testing for this, it recives a int as a parameter that then passes to a task to perform some action with that number, even when running the DAG with the same input, it creates a new version.
What i mean by direction is the memory direction where the `DagParam` object is located (`DagParam object at 0x7fb7bf966750`)
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14223831
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
I think if you have problem with grid rendering you should open a new issue.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14666537
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "gbonazzoli (via GitHub)" <gi...@apache.org>.
GitHub user gbonazzoli edited a discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
### Apache Airflow version
3.0.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have a Python Source that generate `PythonOperator` Tasks on a DAG.
It is the load of the Staging Area in a Datawarehouse, so during this phase there are hundreds of tables that we need to load from the transactional database to the analytical one, Airflow limits the load through `max_active_tasks=32` DAG parameter.
The code is:
```python
with TaskGroup(group_id="MOVEX100") as DLTHUB_MOVEX100:
for source_table_list in [SOURCE_MOVEX100_TABLES]:
for source_table_obj in source_table_list:
table_name = source_table_obj["table_name"]
dlt_query = source_table_obj["dlt_query"]
dlt_load_movex100 = PythonOperator(
task_id="dlt_movex100_" + table_name,
python_callable=execute_dlt_movex100,
retries=3, # Retry up to 3 times (original + 3 retries)
retry_delay=timedelta(minutes=randrange(5) + 1), # Wait from 1 to 5 minutes between retries
templates_dict={
"table_name": f"{table_name}",
"dlt_query": f"{dlt_query}",
},
)
Beg_Load_Staging >> dlt_load_movex100 >> End_Load_Staging
```
The problem with this code is that every days Airflow 3.x registers hundreds of different versions for the DAG.
The reality is that is dynamically generated and so it is obvious that changes dynamically.
Is there anyway to limit the problem ?
### What you think should happen instead?
_No response_
### How to reproduce
I think that is a general problem afflicting Airflow 3.x with DAG that are dinamically generated through code.
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.6.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.4
apache-airflow-providers-fab==2.3.1
apache-airflow-providers-git==0.0.5
apache-airflow-providers-postgres==6.2.2
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-salesforce==5.11.2
apache-airflow-providers-smtp==2.1.2
apache-airflow-providers-ssh==4.1.2
apache-airflow-providers-standard==1.5.0
```
### Deployment
Virtualenv installation
### Deployment details
An ubuntu vanilla installation in python 3.12 virtual env.
### Anything else?
no
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/54348
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Any change in the serialized form of the dag. In case you pass an object that is not serializable as parameter that is supposed to the string - it is converted to string which contains address to the object.
Don't do it.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14225201
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Any change that causes serialized form of DAG (i.e. that one that is sent to Scheduler and stored in the DB) causes new version of Dag. Check if what you are looking for is in that json (I believe it is) and if it is, then any change in the serialized form is seen as "version change".
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14423461
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
I also updated the title to properly reflect it - this is not a "huge problem with dag versioning", it's either lack of understanding that "dynamically generated dags" should have "stable" property if you want to version them.
I hope as result of that discussion maybe you @gbonazzoli -> when you find reasons for that stability issues you can help us to write good documentation describing what "stability" property of such dynamically generated Dags is - or maybe we will even find some bugs in the implementation.
But this is NOT huge problenm with versioning in general, it works as intended, and it **should** work well with dynamically generated Dags - providing of course that those Dags are properly generated and that there are no bugs.
Looking forward to results of your investigation @gbonazzoli
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069799
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Thanks for testing it!
> * if you are reasoning like the "git command" this is an Airflow's "subtle" bug.
> * If you are reasoning looking at the final results of the "compiled dynamic code" this is the perfectly expected behaviour.
The answer is easy. Option 2. It's always been the goal of DAG versioning - and you can read more about it in Airflow Improvement proposals (3 of them) describing the idea, implementaiton, goals, and future of Dag versioning - some of the things there are not yet implemented 100% https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning
> Actually if you talk to a developer about "code's versioning" I think that the common mindset is totally "git influenced".
So if we align Airflow to this mindset it is much simpler to explain and document the mechanism... but probably it is not so simple to implement it in the "dag-processor".
No. It's not a question of simpllicity. It's a question of deliberate choice and probably several tens of discussion points and countless hours of talking and agreeing on the direction by those who proposed, discussed, got approved and finally implemented it. This reflects the very "Fabric" of Airflow - where from the very beginning, dynamig Dag generation actually allowed to effectively have different versions of dag (depending on the input parameters). But before Dag versioning it has never been captured and caused all kind of - mostly - ui and rerun / backfill problems when Dynamically generated dags changed.
So - from the very beginnig of the Dag Versioning discussions (and if you look at the AIPs they have linked in discussions at devlist and those discussions refer not only to other discussions but also to previous version of Dag Versioning proposal - rejected and abandoned https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - this was very well discussed and agreed to.
Now - It seems you indeed pointed to a problem - that people might understand it differently. There is no reason or possibility to change it to the "git" approach, but since you now understand and got explained how it works we would be very happy if you could help to make it clearer:
1). Updating our documentation to make it more explicit and clearer. People like you, who got burned and understood things differently (but are smart and understand the reasoning now) are the best to put it in the words and describing things in the way that other people like them are not making the same (wrong) assumptions you did. Simply - after all the years of discussing it, things are "obvious" for maintainers and creators - but they are "non obvious" for newcomers, so adding a PR where you could describe is better - is golden (and fantastic opportunity to contribute back for the free software you get to use).
2) Describing your and similar use-cases to make it easier for people to see and investigate their issues when they hit similar things - even if it is starting "FAQ" or "common mistakes" or smth like that - coming from someone who had, investigated, and found the issues after having potential reasons explained by maintainer - is again golden and fantastic opportunity to contribute a PR
3) More complex and sophisticated - figuring out some ways of automated detection and flagging similar problems in Dags written by people would be a fantastic opportunity to contribute back "code" contribution. There are many ways and many "bad practices" that could be flagged there - it could be based on static code analysis, or detecting situations where indeed every time you parse you have different versions and it keeps on changing continuously .. World is your oyster. Another great opportunity to contribute.
Adding doc PR is super easy "suggest a change on this page" button (bottom-right) opens a PR where you can add documentation - that could be first step, later leading to a code contribution..
That would be fantastic output of that discussion.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14093597
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Thanks for testing it!
> * if you are reasoning like the "git command" this is an Airflow's "subtle" bug.
> * If you are reasoning looking at the final results of the "compiled dynamic code" this is the perfectly expected behaviour.
The answer is easy. Option 2. It's always been the goal of DAG versioning - and you can read more about it in Airflow Improvement proposals (3 of them) describing the idea, implementaiton, goals, and future of Dag versioning - some of the things there are not yet implemented 100% https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning
> Actually if you talk to a developer about "code's versioning" I think that the common mindset is totally "git influenced".
So if we align Airflow to this mindset it is much simpler to explain and document the mechanism... but probably it is not so simple to implement it in the "dag-processor".
No. It's not a question of simpllicity. It's a question of deliberate choice and probably several tens of discussion points and countless hours of talking and agreeing on the direction by those who proposed, discussed, got approved and finally implemented it. This reflects the very "Fabric" of Airflow - where from the very beginning, dynamig Dag generation actually allowed to effectively have different versions of dag (depending on the input parameters). But before Dag versioning it has never been captured and caused all kind of - mostly - ui and rerun / backfill problems when Dynamically generated dags changed.
So - from the very beginnig of the Dag Versioning discussions (and if you look at the AIPs they have linked in discussions at devlist and those discussions refer not only to other discussions but also to previous version of Dag Versioning proposal - rejected and abandoned https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - this was very well discussed and agreed to.
Now - It seems you did point to something that people might understand it differently. There is no reason or possibility to change it to the "git" approach, but since you now understand and got explained how it works we would be very happy if you could help to make it clearer:
1). Updating our documentation to make it more explicit and clearer. People like you, who got burned and understood things differently (but are smart and understand the reasoning now) are the best to put it in the words and describing things in the way that other people like them are not making the same (wrong) assumptions you did. Simply - after all the years of discussing it, things are "obvious" for maintainers and creators - but they are "non obvious" for newcomers, so adding a PR where you could describe is better - is golden (and fantastic opportunity to contribute back for the free software you get to use).
2) Describing your and similar use-cases to make it easier for people to see and investigate their issues when they hit similar things - even if it is starting "FAQ" or "common mistakes" or smth like that - coming from someone who had, investigated, and found the issues after having potential reasons explained by maintainer - is again golden and fantastic opportunity to contribute a PR
3) More complex and sophisticated - figuring out some ways of automated detection and flagging similar problems in Dags written by people would be a fantastic opportunity to contribute back "code" contribution. There are many ways and many "bad practices" that could be flagged there - it could be based on static code analysis, or detecting situations where indeed every time you parse you have different versions and it keeps on changing continuously .. World is your oyster. Another great opportunity to contribute.
Adding doc PR is super easy "suggest a change on this page" button (bottom-right) opens a PR where you can add documentation - that could be first step, later leading to a code contribution..
That would be fantastic output of that discussion.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14093597
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "JoseBueno30 (via GitHub)" <gi...@apache.org>.
GitHub user JoseBueno30 added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Using pydantic for assuring the right types, maybe i am missing something, but i don't know why it is supossed to be an string, I define it a as an int
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14225585
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Yes. Any change in any parameter of the DAG will create a new version. If you want to continuously change some parameter - just disable versioning for this. Not sure what "direction" of the DagParm is. Don't know what your Dag looks like, but you are likely doing it wrong by passing DagParams as op_args, because op_args are likely supposed to be strings, and what you see is a "string" dump of the DagParam (string) which is not what you likely need.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14223202
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "dor-bernstein (via GitHub)" <gi...@apache.org>.
GitHub user dor-bernstein added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
@potiuk when changing something in the executor config for a single task - should I expect a new version of the dag as well?
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14415933
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "dor-bernstein (via GitHub)" <gi...@apache.org>.
GitHub user dor-bernstein added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
got it, tnx!
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14425492
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
GitHub user tirkarthi added a comment to the discussion: Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS
Does using dynamic task mapping help here?
https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html
cc: @jedcunningham @ephraimbuddy
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069762
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Maybe even - eventually - we might come up with better tooling to analyse such dynamic dag "stabilty" issues - byt I find this case an intersting case study to see what could be the reasons, so that we can write our tooling to be useful to quickly analyse such issues.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069813
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Maybe even - eventually - we might come up with better tooling to analyse such dynamic dag "stabilty" issues - but I find this case an interesting case study to see what could be the reasons, so that we can write our tooling to be useful to quickly analyse such issues.
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069813
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Versioning Problem of (likely badly written) Dynamically Created DAGS [airflow]
Posted by "gbonazzoli (via GitHub)" <gi...@apache.org>.
GitHub user gbonazzoli added a comment to the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
@potiuk
I made some tests and I found that the problem was caused by one clause in the `PythonOperator` definition:
`retry_delay=timedelta(minutes=randrange(5) + 1)`
after having set it up to a static formula:
`retry_delay=timedelta(minutes=1)`
the versioning mechanism is now OK, and Airflow do not increment version number hundreds of times a day.
**My opinion is**:
- if you are reasoning like the "git command" this is an Airflow's "subtle" bug.
- If you are reasoning looking at the final results of the "compiled dynamic code" this is the perfectly aspected behaviour.
Actually if you talk to a developer about "code's versioning" I think that the common mindset is totally "git influenced".
So if we align Airflow to this mindset it is much simpler to explain and document the mechanism... but probably it is not so simple to implement it in the "dag-processor".
What do you think ?
GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14079190
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org