You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "marcosmartinezfco (via GitHub)" <gi...@apache.org> on 2025/12/15 14:01:20 UTC
[I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
marcosmartinezfco opened a new issue, #59465:
URL: https://github.com/apache/airflow/issues/59465
### Apache Airflow version
Other Airflow 3 version (please specify below)
### If "Other Airflow 3 version" selected, which one?
3.1.4
### What happened?
We are observing **continuous DAG version increments** for DAGs sourced from an S3 DAG bundle (`S3DagBundle`).
Even though **no changes** have been made to the DAG files in S3 for several weeks, Airflow keeps generating new DAG versions—for example, the UI currently shows:
> **Dag Version(s): v2516**
This issue happens **without any DAG code updates**, suggesting that the scheduler or DAG processor is detecting artificial "changes" during bundle sync or serialization.
#### Effects we’re seeing:
- DAG version numbers increase constantly (v1000 → v2000+).
- Metadata DB (`serialized_dag`, `dag_version`) grows unnecessarily.
- UI makes it appear as if DAGs are constantly changing.
- Scheduler and DAG processor logs do **not** indicate code changes.
### What you think should happen instead?
Airflow should only create new DAG versions when:
- The DAG file content changes, or
- Structural changes to the DAG occur.
Since the S3 objects have **unchanged ETags and last-modified timestamps**, new versions should **not** be created.
### How to reproduce
No idea, this only happens with one of my Dags which doesn't have any particular special code or anthing
### Operating System
AlmaLinux 9.5 (Teal Serval)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.18.0
apache-airflow-providers-celery==3.14.0
apache-airflow-providers-cncf-kubernetes==10.11.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.7.0
apache-airflow-providers-common-messaging==2.0.1
apache-airflow-providers-common-sql==1.30.0
apache-airflow-providers-docker==4.5.0
apache-airflow-providers-elasticsearch==6.4.0
apache-airflow-providers-fab==3.0.3
apache-airflow-providers-ftp==3.14.0
apache-airflow-providers-git==0.1.0
apache-airflow-providers-google==19.1.0
apache-airflow-providers-grpc==3.9.0
apache-airflow-providers-hashicorp==4.4.0
apache-airflow-providers-http==5.6.0
apache-airflow-providers-microsoft-azure==12.9.0
apache-airflow-providers-mysql==6.4.0
apache-airflow-providers-odbc==4.11.0
apache-airflow-providers-openlineage==2.9.0
apache-airflow-providers-postgres==6.5.0
apache-airflow-providers-redis==4.4.0
apache-airflow-providers-sendgrid==4.2.0
apache-airflow-providers-sftp==5.5.0
apache-airflow-providers-slack==9.6.0
apache-airflow-providers-smtp==2.4.0
apache-airflow-providers-snowflake==6.7.0
apache-airflow-providers-ssh==3.14.0
apache-airflow-providers-standard==1.10.0
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
## Orchestration
- Docker Compose deployment.
- Scheduler + webserver + DAG processor running on a r7i.8xlarge instance
- Celery workers running on additional EC2 instances.
## Executor
- CeleryExecutor
## Metadata Database
- PostgreSQL Aurora cluster
- Highly available, no failovers detected.
- No write issues or DB-level instability.
## Storage
- DAGs pulled from S3 DAG Bundle (S3DagBundle)
- S3 object ETags and LastModified timestamps confirmed stable.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
marcosmartinezfco commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3705037944
> > > Please check whether any arguments in the DAG or tasks include values that change on each execution, such as `datetime.now()`.
> >
> >
> > yeah we use that kind of stuff a lot.
> > Is there any sort of best practices around this?
>
> We are working on having a warning issued when this happens [#59430](https://github.com/apache/airflow/pull/59430) - you can take a lok at this PR in the meantime and avoid those things that are checked there.
Thank you so much, happy new yeay btw.
Also last question would be whether the increments can cause some sort of weird error in the long term when it overflows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
marcosmartinezfco closed issue #59465: S3DagBundle causes continuous DAG version increments despite no code changes
URL: https://github.com/apache/airflow/issues/59465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3656723186
Can you please share the dag so that it is reproducible? Is it also reproducible using LocalDagBundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
marcosmartinezfco commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3666228854
> Please check whether any arguments in the DAG or tasks include values that change on each execution, such as `datetime.now()`.
yeah we use that kind of stuff a lot. I'm also working on another dag that doesn't fetch the right code from the s3 bucket. The bundle version keep increasing in the ui but then the resolved code in the machine doesn't match the latest code that's in the bucket. Any idea of why this might be broken?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
marcosmartinezfco commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3656849853
> Can you please share the dag so that it is reproducible? Is it also reproducible using LocalDagBundle?
Sorry can't share the dag, but it's basically connecting to bastion host that has access to internet using the ssh operator and then running a script that lives in a shared network filesystem mounted in all our machines.
I haven't managed to reproduce it with the LocalDagBundle. Happy to help with the debugging if there's anything else I can do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3666695662
> > Please check whether any arguments in the DAG or tasks include values that change on each execution, such as `datetime.now()`.
>
> yeah we use that kind of stuff a lot.
>
> Is there any sort of best practices around this?
We are working on having a warning issued when this happens https://github.com/apache/airflow/pull/59430 - you can take a lok at this PR in the meantime and avoid those things that are checked there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
marcosmartinezfco commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3719653877
Closing issue, seems like `3.1.6` will include the fix. Will reopen if persisting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3707281267
> Also last question would be whether the increments can cause some sort of weird error in the long term when it overflows?
Not sure. There are no infinite resources in the world - so any infinite increase might cause problems, archiving this table might be a good idea in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "m8719-github (via GitHub)" <gi...@apache.org>.
m8719-github commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3704392807
Also ran into this issue.
Our use-case: we are injecting labels + annotations into KPO (Kubernetes pod operator) task via templating functions of the form: `fn(context: Context, jinja_env: jinja2.Environment)` and when these get serialized we get this for `labels`: `<function get_labels.<locals>._get_labels at 0x7fe89e4aa7a0>` in `LazyDeserializedDAG.data` field.
The address obviously changes on different calls of `dag-processor` every refresh interval, leading to hash mismatch and a version explosion.
We do this for integration with our K8s observability platform, it's a requirement for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3658338089
You can check the md5 hash to see if it's changing. You can also take a diff of the two serialised dag entries in data column for a given dag_id to see the difference that might help. Please note if the dag constructor has some dynamic elements like changing timestamp in dag.description it could also lead to this.
https://github.com/apache/airflow/blob/d34942ef1fa1b75285ef06f291c9a951cc274783/airflow-core/src/airflow/models/serialized_dag.py#L354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
wjddn279 commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3659222868
Please check whether any arguments in the DAG or tasks include values that change on each execution, such as `datetime.now()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3DagBundle causes continuous DAG version increments despite no code changes [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
marcosmartinezfco commented on issue #59465:
URL: https://github.com/apache/airflow/issues/59465#issuecomment-3655812505
The refresh rate is `30s` so seems like every refresh triggers a new version change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org