[PR] Add atomic versioned deployments to S3 Dag bundles [airflow]

["khuper (via GitHub)" <gi...@apache.org>]

khuper opened a new pull request, #72228:
URL: https://github.com/apache/airflow/pull/72228

   S3 Dag bundles currently sync the latest objects from a prefix, so a Dag run cannot reliably load the exact code it started with. Multi-object deployments can also be observed partway through an update.\n\nThis adds an optional  mode for Airflow 3.4 and later. It:\n\n- pins each release to exact S3 object VersionIds, sizes, and SHA-256 hashes\n- publishes releases through a small current pointer written last\n- verifies downloads in a staging directory before an atomic local rename\n- keeps older generations available for tasks, callbacks, and in-flight parsing\n- leaves the existing unversioned S3 behavior unchanged\n\nThe change also fixes bundle generation lease handling and keeps callback bundle metadata tied to the Dag run version.\n\nTests cover manifest validation, historical versions, partial downloads, integrity failures, concurrent initialization, requester-pays requests, parser and worker leases, stale parse results, and callback version consistency.\n\n---\n\n##### Was
  generative AI tooling used to co-author this PR?\n\n- [X] Yes: Codex (GPT-5)\n\nGenerated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)\n\n---\nDrafted-by: Codex (GPT-5) (no human review before posting)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add atomic versioned deployments to S3 Dag bundles [airflow]

["boring-cyborg[bot] (via GitHub)" <gi...@apache.org>]

boring-cyborg[bot] commented on PR #72228:
URL: https://github.com/apache/airflow/pull/72228#issuecomment-5456264625

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add atomic versioned deployments to S3 Dag bundles [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil commented on PR #72228:
URL: https://github.com/apache/airflow/pull/72228#issuecomment-5942558038

   Closing this. Beyond the S3 `manifest_key` feature, it changes core behavior with no reported problem: four callback paths switch from `ti.dag_version.bundle_version` to `ti.dag_run.bundle_version`, and the comment in `serialized_dag.py` that documents the current behavior is rewritten to say the opposite. It also adds a Cadwyn version change to `DagFileParseRequest`, changes bundle tracking-file semantics, and reorders `BundleVersionLock` in the task runner, across +2399/-235 lines with no linked issue. The description says it was posted without human review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add atomic versioned deployments to S3 Dag bundles [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil closed pull request #72228: Add atomic versioned deployments to S3 Dag bundles
URL: https://github.com/apache/airflow/pull/72228


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org