You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nikolasharing (via GitHub)" <gi...@apache.org> on 2025/12/11 10:41:18 UTC
[I] Airflow can not open path in zipped DAG folder [airflow]
nikolasharing opened a new issue, #59310:
URL: https://github.com/apache/airflow/issues/59310
### Apache Airflow version
3.1.4
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
The bug in the following issue was never solved:
https://github.com/apache/airflow/issues/10160
Jinja2 templating does not work and one can not change the template_searchpath such that the (for example) sql file is recognized.
There is just a workaround using the open_maybe_zipped utils function and using it to open the file manually before giving the path to the Operator, like this:
```
from airflow.utils.file import open_maybe_zipped
open_maybe_zipped("/home/airflow/gcs/dags/test.zip/test_sql/test.sql").read()`
```
### What you think should happen instead?
_No response_
### How to reproduce
In `/home/airflow/gcs/dags/test.zip/test_sql/test.sql`:
```
SELECT column_a FROM test
```
In `/home/airflow/gcs/dags/test.zip/test.py`:
```
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.utils.file import open_maybe_zipped
# Define DAG
with DAG(
dag_id="test_import_path_dag",
schedule_interval=None,
template_searchpath="/home/airflow/gcs/dags/test",
default_args = {"retries": 0}
) as dag:
test = BigQueryInsertJobOperator(
task_id="test_task",
configuration={
"query": {
"query": "test_sql/test.sql" , # PATH HERE IS NOT FOUND WHEN DAGS ARE ZIPPED
"useLegacySql": False,
"destinationTable": {
"projectId": "test",
"datasetId": "test",
"tableId": "test",
},
}
},
)
test
```
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3702128785
Feel free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "soooojinlee (via GitHub)" <gi...@apache.org>.
soooojinlee commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3987684800
I've submitted a PR to address this issue: https://github.com/apache/airflow/pull/60999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "nikolasharing (via GitHub)" <gi...@apache.org>.
nikolasharing commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3654930487
The exact error was:
jinja2.exception.TemplateNotFound: '/home/airflow/gcs/dags/<path_within_zip_file_to_sql_file_here>' not found in search path: '/home/airflow/gcs/dags/<zip_file_name_here>'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3986421963
@soooojinlee We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "nikolasharing (via GitHub)" <gi...@apache.org>.
nikolasharing commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3767365545
I do not exactly know what you mean. I think you can just use the two files I posted under the "How to reproduce" section, put them in a .zip archive and use them with your airflow version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "nikolasharing (via GitHub)" <gi...@apache.org>.
nikolasharing commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3647033812
I get a jinja2.TemplateNotFoundError.
I will share the exact error beginning of next week, when I am back at my PC.
________________________________
Von: Jake Roach ***@***.***>
Gesendet: Thursday, December 11, 2025 5:08:39 PM
An: apache/airflow ***@***.***>
Cc: Nikolas Haring ***@***.***>; Mention ***@***.***>
Betreff: Re: [apache/airflow] Airflow can not open path in zipped DAG folder (Issue #59310)
[https://avatars.githubusercontent.com/u/116606359?s=20&v=4]jroachgolf84 left a comment (apache/airflow#59310)<https://github.com/apache/airflow/issues/59310#issuecomment-3642651684>
@nikolasharing<https://github.com/nikolasharing>, thanks for creating this issue. Can you share the error that you're seeing?
—
Reply to this email directly, view it on GitHub<https://github.com/apache/airflow/issues/59310#issuecomment-3642651684>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKTBGN3LQKAGH57HUGRA3WD4BGJIPAVCNFSM6AAAAACOXACJUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNBSGY2TCNRYGQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3641313384
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "Nataneljpwd (via GitHub)" <gi...@apache.org>.
Nataneljpwd commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3765735286
Hello, do you mind posting some minimal reproduction steps? what dag bundle setup did you use?
it would help in reproducing, so that I will be able to try and solve the issue, and debug what causes it.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "soooojinlee (via GitHub)" <gi...@apache.org>.
soooojinlee commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3794459950
I treated this as a Jinja2 loader issue rather than a file‑open issue, so I added a zip‑aware loader in the task SDK. Regular paths keep `FileSystemLoader` behavior, and zip paths now use the same `split_template_path` validation to keep safety and behavior consistent. I also made sure searchpath order is preserved.
I consolidated the tests into `task-sdk/tests/task_sdk/definitions/test_dag.py` to keep them in the SDK boundary. Coverage includes zipped DAG template loading, mixed searchpaths, ordering precedence, and zip path traversal blocking. Local tests passed.
`open_maybe_zipped` lives in airflow‑core and is just a file‑open helper; this bug is about Jinja2’s template resolution path, so it wasn’t a good fit to reuse it. That’s why I fixed it in the loader instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "soooojinlee (via GitHub)" <gi...@apache.org>.
soooojinlee commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3701305032
Hi! I’m interested in working on this issue. I’ll investigate and open a draft PR if I find a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3642651684
@nikolasharing, thanks for creating this issue. Can you share the error that you're seeing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3775594128
> [@Nataneljpwd](https://github.com/Nataneljpwd) I do not exactly know what you mean. I think you can just use the two files I posted under the "How to reproduce" section, put them in a .zip archive and use them with your airflow version?
Usually the more exact the reproduction steps are - for example prepare the exact zip file that causede you a problem that you might drop into a "start-airflow" dags folder - the better. Otherwise if it's just your specific issue, those who try to reproduce it, migh need to remotely diagnose your setup without having access to it.
Generally speaking if you want volunteers to take a look at your issue, making it super-easy and 100% reproducible with very limited effort of the one who wants to help you, donating their own time to solve your problem is a good ide @nikolasharing
- that's how open-source work. You increase chances of your problem being quickly fixed by making it easy easy as possible to reproduce it.
I'd say it's in your best interest to respond and provide as straightforward and easy way to reproduce your issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow can not open path in zipped DAG folder [airflow]
Posted by "nikolasharing (via GitHub)" <gi...@apache.org>.
nikolasharing commented on issue #59310:
URL: https://github.com/apache/airflow/issues/59310#issuecomment-3777422545
@potiuk @Nataneljpwd Sorry, I did not mean to make the impression that I do not want to help here. If I can help in any way, I'm happy to.
Here is the zip file:
[test.zip](https://github.com/user-attachments/files/24764065/test.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org