You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uranusjr (via GitHub)" <gi...@apache.org> on 2025/05/09 09:23:47 UTC
[PR] Add bundle root to sys.path in dag processor [airflow]
uranusjr opened a new pull request, #50385:
URL: https://github.com/apache/airflow/pull/50385
Instead of adding the dags folder into `sys.path` globally, each dag file parser now _only_ adds the dag bundle’s root path to `sys.path` for that specific parsing job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add bundle root to sys.path in dag processor [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #50385:
URL: https://github.com/apache/airflow/pull/50385#discussion_r2083945634
##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -87,6 +85,11 @@ def _parse_file_entrypoint():
task_runner.SUPERVISOR_COMMS = comms_decoder
log = structlog.get_logger(logger_name="task")
+ # Put bundle root on sys.path if needed. This allows the dag bundle to add
+ # code in util modules to be shared between files within the same bundle.
+ if (bundle_root := os.fspath(msg.bundle_path)) not in sys.path:
+ sys.path.append(bundle_root)
Review Comment:
Should this be append or prepend? What did we do with DAGs folder in 2.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add bundle root to sys.path in dag processor [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #50385:
URL: https://github.com/apache/airflow/pull/50385#discussion_r2085281481
##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -87,6 +85,11 @@ def _parse_file_entrypoint():
task_runner.SUPERVISOR_COMMS = comms_decoder
log = structlog.get_logger(logger_name="task")
+ # Put bundle root on sys.path if needed. This allows the dag bundle to add
+ # code in util modules to be shared between files within the same bundle.
+ if (bundle_root := os.fspath(msg.bundle_path)) not in sys.path:
+ sys.path.append(bundle_root)
Review Comment:
Keeping it the same is fine imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add bundle root to sys.path in dag processor [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50385:
URL: https://github.com/apache/airflow/pull/50385#issuecomment-2873695644
### Backport successfully created: v3-0-test
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-0-test</td>
<td><a href="https://github.com/apache/airflow/pull/50509"><img src="https://img.shields.io/badge/PR-50509-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add bundle root to sys.path in dag processor [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #50385:
URL: https://github.com/apache/airflow/pull/50385#discussion_r2083970006
##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -87,6 +85,11 @@ def _parse_file_entrypoint():
task_runner.SUPERVISOR_COMMS = comms_decoder
log = structlog.get_logger(logger_name="task")
+ # Put bundle root on sys.path if needed. This allows the dag bundle to add
+ # code in util modules to be shared between files within the same bundle.
+ if (bundle_root := os.fspath(msg.bundle_path)) not in sys.path:
+ sys.path.append(bundle_root)
Review Comment:
We do append with the dags folder. I was skeptical about this too but decided to just copy the behaviour for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add bundle root to sys.path in dag processor [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham merged PR #50385:
URL: https://github.com/apache/airflow/pull/50385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add bundle root to sys.path in dag processor [airflow]
Posted by "codenamelxl (via GitHub)" <gi...@apache.org>.
codenamelxl commented on PR #50385:
URL: https://github.com/apache/airflow/pull/50385#issuecomment-2928561100
I port this fix to our Airflow 3.0.1 installation, it works for Dag Processor but fail with same error for worker. Created #51282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org