You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Aditya-XR (via GitHub)" <gi...@apache.org> on 2026/10/12 07:18:48 UTC
[PR] [v3-3-test] Stop treating .jar, .whl and other zip-format files as Dag files [airflow]
Aditya-XR opened a new pull request, #74573:
URL: https://github.com/apache/airflow/pull/74573
On 3.3, Dag discovery decides whether a non-`.py` file is a zipped Dag with `zipfile.is_zipfile()` alone, without looking at the file name. `.jar`, `.whl`, `.egg`, `.docx`, `.xlsx` and plenty of other formats are zip containers, so any of them sitting in a Dags folder gets picked up as a Dag file. The Dag processor then queues it for parsing on every loop, the parse finds no importer for that extension and returns nothing, and the file shows up in parsing stats and `airflow dags report` with 0 Dags. #71125 has a report of this from a real deployment.
This goes straight to `v3-3-test` because there is nothing to cherry-pick: on `main` and `v3-4-test` the importer-based discovery already matches on file extension only.
**Change:** a non-`.py` file is only treated as a zipped Dag when its name ends in `.zip` (any case) and it is a valid zip archive. Applied to both discovery paths:
- `find_dag_file_paths` in `airflow/utils/file.py`, used by the Dag processor via `list_py_file_paths`
- `PythonDagImporter.list_dag_files`, used by `DagBag.collect_dags`
This makes the directory walk agree with the rest of 3.3, where the importer registry already picks an importer by file extension.
**Behaviour notes**
- `.zip` and `.ZIP` archives are discovered exactly as before, and an invalid `.zip` is still skipped.
- A zip archive whose name doesn't end in `.zip` (a `bundle.zip.bak` backup, a file with no extension, ...) is no longer discovered. Dags inside such files couldn't be loaded before either, because no importer is registered for those extensions, so nothing that works today stops working.
- `import_file` and `_get_observed_filelocs` still use `is_zipfile`, but they only ever see files that discovery returned, so I left them alone to keep the patch small.
**Testing**
- New parametrized tests for both discovery paths, each case a real archive containing a Dag-like file: `.zip`, `.ZIP`, `.jar`, `.whl`, `.egg`, `.docx`, `.zip.bak` and no extension. The six non-`.zip` cases fail on current `v3-3-test` and pass with the change.
- `airflow-core/tests/unit/utils/test_file.py` and `airflow-core/tests/unit/dag_processing/`: 466 passed (450 existing + 16 new). The same 450 pass on the base branch.
- `prek` on the changed files passes, including mypy for airflow-core.
- Checked by hand with a Dags folder holding `.zip` and `.ZIP` bundles, a plain `.py` Dag, a directory named `folder.zip/`, a corrupt `.zip`, and a `.jar`, `.whl`, `.docx` and `.zip.bak` that each contain a Dag. Before, discovery returned 8 entries including all four non-`.zip` archives; after, only the `.py`, the two archives and the file inside `folder.zip/`. The set of loaded Dags is the same before and after.
related: #71125
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [v3-3-test] Stop treating .jar, .whl and other zip-format files as Dag files [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #74573:
URL: https://github.com/apache/airflow/pull/74573#issuecomment-6119602326
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org