You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "gbonazzoli (via GitHub)" <gi...@apache.org> on 2025/08/11 04:52:22 UTC
[I] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
gbonazzoli opened a new issue, #54337:
URL: https://github.com/apache/airflow/issues/54337
### Apache Airflow version
3.0.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have a Python Source that generate `PythonOperator` Tasks on a DAG.
It is the load of the Staging Area in a Datawarehouse, so during this phase there are hundreds of tables that we need to load from the transactional database to the analytical one, Airflow limits the load through `max_active_tasks=32` DAG parameter.
The code is:
```python
with TaskGroup(group_id="MOVEX100") as DLTHUB_MOVEX100:
for source_table_list in [SOURCE_MOVEX100_TABLES]:
for source_table_obj in source_table_list:
table_name = source_table_obj["table_name"]
dlt_query = source_table_obj["dlt_query"]
dlt_load_movex100 = PythonOperator(
task_id="dlt_movex100_" + table_name,
python_callable=execute_dlt_movex100,
retries=3, # Retry up to 3 times (original + 3 retries)
retry_delay=timedelta(minutes=randrange(5) + 1), # Wait from 1 to 5 minutes between retries
templates_dict={
"table_name": f"{table_name}",
"dlt_query": f"{dlt_query}",
},
)
Beg_Load_Staging >> dlt_load_movex100 >> End_Load_Staging
```
The problem with this code is that every days Airflow 3.x registers hundreds of different versions for the DAG.
The reality is that is dynamically generated and so it is obvious that changes dynamically.
Is there anyway to limit the problem ?
### What you think should happen instead?
_No response_
### How to reproduce
I think that is a general problem afflicting Airflow 3.x with DAG that are dinamically generated through code.
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.6.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.4
apache-airflow-providers-fab==2.3.1
apache-airflow-providers-git==0.0.5
apache-airflow-providers-postgres==6.2.2
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-salesforce==5.11.2
apache-airflow-providers-smtp==2.1.2
apache-airflow-providers-ssh==4.1.2
apache-airflow-providers-standard==1.5.0
```
### Deployment
Virtualenv installation
### Deployment details
An ubuntu vanilla installation in python 3.12 virtual env.
### Anything else?
no
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54337:
URL: https://github.com/apache/airflow/issues/54337#issuecomment-3173957792
> The reality is that is dynamically generated and so it is obvious that changes dynamically.
That's not true in general.
If your intention to generate a different dag every time it is parsed, then yes it will be dynamically changing and. as far as I know - you can disable versioning https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#disable-bundle-versioning for that bundle if your bundle contains DAG that change frequently and it is expected (but this is generally a bad practice)
Dynamically generated dags are NOT supposed to dynamically change every time they are parsed. They MAY change slower or faster - depending on how often the "input" to those dynamic dags change. For example if you have list of countries, and you generate dynamic dags based on that - your parsed dag should change only when you add a new country - every time (providing that your code to generate dynamic dag is well written) , parsing the same Dag **should** produce the same - dynamically generated - parsed representation.
So in your case you likely:
a) your intention is to change your dag every time it is parssed (which is quite strange, but then you should disable versioning for that bundle)
b) there is a problem with your dynamic dag generation code - it **should** produce the same parsed representation, every time Dag is parsed.
There might be two resons for b)
1) your dynamic dag generation has problems and does not produce stable dag object structure or stable parameters. This might change for example when you do not use 'fixed` start date - but use `days_ago` or similar - which is a bad practice, because days_ago() will change every time your dag is parsed. Or you can pass time() or other dynamically generated (and ever changing) content as part of your dag definition. Or you create objects in a random sequence every time you build your dynamic dag. This might happen for example when you run "SELECT * FROM" and do not ORDER the select - because the databases return data in such select in random order in general, and the order might change any time.
2) there is a bug in our Dag serialization code
Most likely it's 1) and you have to fix your dynamic dag generation to be "stable" if you want to version dag. Which means that serialized version of your Dag should be THE SAME if Dag parsing on your Dag is executed - if none of the inputs changed - and they should remain stable over time, so they should be independent on the time when they are parsed.
It could also be 2) - but in order to know whether it's 1) or 2) you have to look at the serialized dag table (see https://airflow.apache.org/docs/apache-airflow/stable/database-erd-ref.html ) find two versions of your Dags and find out what's the difference betwen the dags and what kind of "stability" issue in dynamic dag generation causes it. The "compressed" representation in this table is compressed with `.gzip` so you need to retrieve the blobs and decompress them -> you will get .json representation of the dags that you should be able to compare and see where the problem is coming from.
Also it might be easier for you if you review your dynamic dag generation looking at potential stability issues from those described above.
Once you find the reasons - please let us know - it would be interesting to find out what stability issues people like you might have - or maybe you will find some of the problems coming from our serializaiton. So we are looking forward for your investigation.
Converting it into a discussion until you investigate it and provide more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #54337:
URL: https://github.com/apache/airflow/issues/54337#issuecomment-3173955432
Does using dynamic task mapping help here?
https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html
cc: @jedcunningham @ephraimbuddy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #54337: Airflow 3.x Huge Versioning Problem In Dynamically Created DAGS
URL: https://github.com/apache/airflow/issues/54337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org