You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "simi (via GitHub)" <gi...@apache.org> on 2025/09/19 15:47:59 UTC

[PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

simi opened a new pull request, #55894:
URL: https://github.com/apache/airflow/pull/55894

   - removes ad-hoc bundle_path addition and moves it to one central place in DagBag parsing logic
   - when DagBag is initialized with bundle_path, bundle_path is temporarily added into sys.path during parsing (and removed after)
     - if already present, nothing happens
   - fixes https://github.com/apache/airflow/issues/53617
   - makes docs at https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html#built-in-pythonpath-entries-in-airflow up-to-date again (could be worth to update how this process works, since it is different, but result - our dag folder is in sys.path when needed - is the same)
   
   /cc @potiuk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3325554623

   Sorry, thought I'd replied to the task thing, I assume thats what you meant? Shout if you have other questions.
   
   (Missed rc2 though)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3801391550

   FYI @AutomationDev85 I think you reported the same problem to me, correct?
   
   One question though: Is this only respecting the path at Dag parse time or also at execution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410008662

   The only kind of cleanup I want is no cleanup code at all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3328605794

   :information_source: rebased and squashed after https://github.com/apache/airflow/pull/55139 merge. @potiuk would you mind to check also?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3424669686

   Sounds good to me. I’d say the function could be a method on the dagbag instead, but that’s just personal preference; either is fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3424866934

   > @potiuk I was actually trying to achieve the same, just with the opposite thinking. Do not do changes and annoy others with review until it is at least agreed on the direction. I'm happy to rewrite until mergable, no worries.
   
   Proposed changes that you can review and comment on are **FAR** less annyoing that trying to imagine the same changes described in words and following a discussion that might be prone to interpretation, mis-interpretation, and simply not taking into account all edge cases and practicalities of such change.
   
   I'd say discussing on an actual change proposed is always several orders of magnitude more effective than discussion on "future described change' - and with small changes like that one, the effort to make a proposal is really negligible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3423617538

   @potiuk good to go and update? WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410426692

   > @potiuk can you be more precise? Which of the suggested approaches would fit that? Special DagBag class, explicit `append_bundle_to_syspath` argument or even any other?
   
   The additional helper method that I mentioned above replacing original duplicated code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3728748934

   > Good to squash now and update the commit message?
   
   No need. We use "squash & merge" as the only way to merge PRs so they are always squashed into single commit when we merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3327906873

   > Those are literal `...`, or `pass` - you can have a dag without tasks :) (the 4 dots was a typo, sorry)
   
   Ahh, I had no idea bout `...`, I recognize `pass`. Thanks for sharing, I'm still relatively new to Python :snake: world.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3325815730

   @jedcunningham I'm not sure I do follow `...` and `....` comments, but I did my best to fix styling and the append.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3374650170

   @eladkal I think there's consensus this is actually improvement, not bugfix. Bug was in 3.0.1, fixed in 3.0.2 and this is addition to it to provide safe default (to be closer to Airflow 2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410113452

   @potiuk that's what I mean by wrapped `DagBag`, special case adding to path with no cleanup. Currently it does add to path actually only when `bundle_path` is used which happens only in those cases:
   
   ```
   [retro@retro2  airflow (dagbag-syspath %=)]❤ ag DagBag -A 10 | grep bundle_path | sort
   airflow-core/src/airflow/cli/commands/dag_command.py:378:                    dagbag = DagBag(bundle.path, bundle_path=bundle.path)
   airflow-core/src/airflow/cli/commands/dag_command.py:475:                    dagbag = DagBag(bundle.path, bundle_path=bundle.path)
   airflow-core/src/airflow/cli/commands/dag_command.py:527:        dagbag = DagBag(bundle.path, bundle_path=bundle.path, include_examples=False)
   airflow-core/src/airflow/cli/commands/dag_command.py:691:        dag_bag = DagBag(bundle.path, bundle_path=bundle.path, include_examples=False)
   airflow-core/src/airflow/dag_processing/processor.py:189-        bundle_path=msg.bundle_path,
   airflow-core/src/airflow/utils/cli.py:283-                dag_folder=dagfile_path or bundle.path, bundle_path=bundle.path, include_examples=False
   airflow-core/src/airflow/utils/cli.py:293-                dag_folder=dagfile_path or bundle.path, bundle_path=bundle.path, include_examples=False
   airflow-core/src/airflow/utils/cli.py:330:        dagbag = DagBag(dag_folder=bundle.path, bundle_path=bundle.path)
   airflow-core/tests/unit/dag_processing/test_dagbag.py:1010:        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=None, include_examples=False)
   airflow-core/tests/unit/dag_processing/test_dagbag.py:951:        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
   airflow-core/tests/unit/dag_processing/test_dagbag.py:986:        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
   airflow-core/tests/unit/dag_processing/test_dagbag.py:991-    def test_bundle_path_none_no_syspath_manipulation(self, tmp_path, caplog):
   airflow-core/tests/unit/dag_processing/test_dagbag.py:993-        Test that no sys.path manipulation occurs when bundle_path is None.
   airflow-core/tests/unit/dag_processing/test_manager.py:469-            bundle_path=test_dag_path.bundle_path,
   airflow-core/tests/unit/models/test_dag.py:2124:        bag = DagBag(dag_folder=file_path, bundle_path=bundle_path)
   task-sdk/src/airflow/sdk/execution_time/task_runner.py:620-        bundle_path=bundle_instance.path,
   ```
   
   Wouldn't be enough to rename this argument (not sure it is that simple for backward compatibility), to make it clear it modifies `sys.path` or add additional argument `append_bundle_to_syspath` or similar?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410209869

   > Wouldn't be enough to rename this argument (not sure it is that simple for backward compatibility), to make it clear it modifies sys.path or add additional argument append_bundle_to_syspath or similar?
   
   As mentioned above - that opens up for potential side-effects in places where the path should not be added - so I think (@uranusjr ?) that explicitly adding the path in the places where we know we are about to exit right after is likely a better approach. In this case path modification is really tied with the place where it is used, not the DagBag itself. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3409863947

   > @uranusjr can you explain what you refer to by the "original code" here?
   
   I believe what @uranusjr meant is that now adding the path happens inside DagBag, and potentially DagBag might be used in a number of places, so adding a path for those cases (even if it is removed) can have some side effects (modules loaded from a path that is no longer available on sys.path. 
   
   The "orignal" way (if I interpret it correctly) addded the sys.path entry not in the "DagBag" - but outside - in the code that we "knew" we don't care about side-effects (for example in dag file processor parsing is done in a forked interpreter, and the interpreter exists after the Dag is serialized).
   
   Again - if I am right - maybe a better solution that is both DRY and side-effect frree for regular DagBag usage, might be another class (`DagBagWithExtraBundlePath` - I know, bad name) which would extend from the DagBag, do the try/finally around DagBag parse in it's own parse - and use that class specifically in those places that need to extend sys.path.
   
   Is that good interpretation and proposal @uranusjr ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3409995573

   You need to be aware @simi  - that with our velocity and variety of contributors, anything that makes it more difficult to make accidental mistake by a new person or missed during review might have a huge impact on all the 100s of contributors. DagBag is one of very, very core classes we use (look at how many times it is used in tests) - so any side effects in it should be avoided. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2365623343


##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -398,12 +398,23 @@ def handler(signum, frame):
 
         def parse(mod_name, filepath):
             try:
-                loader = importlib.machinery.SourceFileLoader(mod_name, filepath)
-                spec = importlib.util.spec_from_loader(mod_name, loader)
-                new_module = importlib.util.module_from_spec(spec)
-                sys.modules[spec.name] = new_module
-                loader.exec_module(new_module)
-                return [new_module]
+                # Add bundle path to sys.path if we have one
+                bundle_path_added = False
+                if self.bundle_path and str(self.bundle_path) not in sys.path:
+                    sys.path.insert(0, str(self.bundle_path))

Review Comment:
   No idea what's best practice. Happy to change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2365624011


##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -398,12 +398,23 @@ def handler(signum, frame):
 
         def parse(mod_name, filepath):
             try:
-                loader = importlib.machinery.SourceFileLoader(mod_name, filepath)
-                spec = importlib.util.spec_from_loader(mod_name, loader)
-                new_module = importlib.util.module_from_spec(spec)
-                sys.modules[spec.name] = new_module
-                loader.exec_module(new_module)
-                return [new_module]
+                # Add bundle path to sys.path if we have one
+                bundle_path_added = False
+                if self.bundle_path and str(self.bundle_path) not in sys.path:
+                    sys.path.insert(0, str(self.bundle_path))

Review Comment:
   Just looking at the diff, we were tossing it on the end previously. I think that makes sense too, lower priority than everything else.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3723362023

   @uranusjr update. Not sure if the warning is not too much. I did it this way to keep the kw args to BundleDagBag the same as DagBag. Looking at the usage, maybe it is not worth it and it will be better to strip only to currently used kw params and provide safe defaults for the others.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3407638360

   > _🙏 I hope this is last rebase I need to do, the test_dagbag.py seems pretty tubulent these days. :)_
   
   :eyes: 
   
   Yeah.. With our velocity (picking up after the Summit) conflicts are bound to happen unfortunately. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3715233549

   @uranusjr got idea to create separate BundleDagBag to make it clear it will update `sys.path` and updated all CLI related places. Yes, there's chance it will be missed, but IMHO low chance it will be used when not needed (aka instead of DagBag). I need to finish my testing but looks promising to me. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3409389954

   @uranusjr can you explain what you refer to by the "original code" here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410525599

   Good, @uranusjr would that work for you also?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3329918694

   updated @jedcunningham 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #55894:
URL: https://github.com/apache/airflow/pull/55894


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3726827710

   Good to squash now and update the commit message?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2664725541


##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -235,6 +235,13 @@ def __init__(
         self.bundle_path = bundle_path
         self.bundle_name = bundle_name
 
+        # Add bundle path to sys.path if provided.
+        # This allows DAG files to import modules from their bundle directory.
+        # No cleanup is performed - this is intentional for ephemeral processes
+        # (dag processor, task runner, CLI) where the process exits after use.
+        if bundle_path and str(bundle_path) not in sys.path:
+            sys.path.append(str(bundle_path))

Review Comment:
   Yes, I have moved this into draft. I'm working on better sollution.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2671391877


##########
airflow-core/src/airflow/cli/commands/dag_command.py:
##########
@@ -526,7 +530,9 @@ def dag_report(args) -> None:
         if bundle.name not in bundles_to_reserialize:
             continue
         bundle.initialize()
-        dagbag = DagBag(bundle.path, bundle_name=bundle.name, include_examples=False)
+        dagbag = BundleDagBag(
+            bundle.path, bundle_path=bundle.path, bundle_name=bundle.name, include_examples=False

Review Comment:
   `include_examples` should always be False when loading from a bundle (because example dags are not in the bundle… obviously), so this should be handled in BundleDagBag’s `__init__` instead of repeated in every instantiation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3409977179

   > @potiuk What would be the difference? sys.path will get mangled anyway (if used in non-forked env). Are those 3 cases handled in this PR this kind of "fire and forget the environment" situation?
   
   Yes. All 3. Which means that if we specifically use it in those 3 places, the "sys.path" mangling is only possible there - rather than in random places where DagBag is used - accidentally.  Again - if I am not mistaken about intention, the thing is that DagBag is used in so many places (for example in tests where interpreter is used by multiple tests)  that adding a potentially breaking parameter. It's purely "prevent accidental errors" difference.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410055884

   > DagBag is one of very, very core classes we use (look at how many times it is used in tests) - so any side effects in it should be avoided.
   
   @potiuk you're totally right, I have realized this during writting tests, since the loaded DAGs seems to be persisted across tests, if I remeber well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2373452126


##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task

Review Comment:
   These are more important though, import everything from sdk not the old locations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2376039076


##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,95 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow.sdk import DAG
+                from airflow.operators.empty import EmptyOperator
+
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    EmptyOperator(task_id="mytask")
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow.sdk import DAG
+                from airflow.operators.empty import EmptyOperator

Review Comment:
   ```suggestion
                   from airflow.providers.standard.operators.empty import EmptyOperator
   ```



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,95 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow.sdk import DAG
+                from airflow.operators.empty import EmptyOperator
+
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    EmptyOperator(task_id="mytask")
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow.sdk import DAG
+                from airflow.operators.empty import EmptyOperator
+
+                import bundle_utils # typo
+
+                with DAG('test_import'):
+                    EmptyOperator(task_id="mytask")
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        assert dagbag.import_errors  # Check import failed
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_none_no_syspath_manipulation(self, tmp_path, caplog):
+        """
+        Test that no sys.path manipulation occurs when bundle_path is None.
+        """
+        dag_file = tmp_path / "simple_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow.sdk import DAG
+                from airflow.operators.empty import EmptyOperator

Review Comment:
   ```suggestion
                   from airflow.providers.standard.operators.empty import EmptyOperator
   ```



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,95 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow.sdk import DAG
+                from airflow.operators.empty import EmptyOperator

Review Comment:
   ```suggestion
                   from airflow.providers.standard.operators.empty import EmptyOperator
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2365724096


##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -398,12 +398,23 @@ def handler(signum, frame):
 
         def parse(mod_name, filepath):
             try:
-                loader = importlib.machinery.SourceFileLoader(mod_name, filepath)
-                spec = importlib.util.spec_from_loader(mod_name, loader)
-                new_module = importlib.util.module_from_spec(spec)
-                sys.modules[spec.name] = new_module
-                loader.exec_module(new_module)
-                return [new_module]
+                # Add bundle path to sys.path if we have one
+                bundle_path_added = False
+                if self.bundle_path and str(self.bundle_path) not in sys.path:
+                    sys.path.insert(0, str(self.bundle_path))

Review Comment:
   I would go with append, since it was used previously.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2365648670


##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()

Review Comment:
   Should I remove the task?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3407529660

   _:pray: I hope this is last rebase I need to do, the test_dagbag.py seems pretty tubulent these days. :)_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410049789

   @potiuk @uranusjr good, would wrapped `DagBag` carefully applied in those 3 cases be considered improvement over current state and also useful extension of dag command?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2671602322


##########
airflow-core/src/airflow/models/__init__.py:
##########


Review Comment:
   I'll remove, just wondering, is there plan to move CLI commands to use DBDabBags also? Or what is plan on those?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2671396075


##########
airflow-core/src/airflow/models/__init__.py:
##########


Review Comment:
   This is for backward compatibility. Let’s not add the new class here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3729030437

   Happy BDay ! Glad we could make it better!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3326267965

   Those are literal `...`, or `pass` - you can have a dag without tasks :) (the 4 dots was a typo, sorry)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3314861739

   This is also affecting 3.0 right? (That is, this isn't a new regression in 3.1?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3314958396

   3.0.0/3.0.1 was arguably worse in the sense it naively added the AF2 dags dir instead of bundle root, even if it wasnt configured.
   
   But yes, not a regression. Meaning, it should go in rc2 if merged and we have an rc2, but not enough to warrant an rc2 on its own.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3314914078

   > This is also affecting 3.0 right? (That is, this isn't a new regression in 3.1?)
   
   IMHO this was missed in 3.0.2 during some Task API refactorings (moving to DAG bundles). 3.0.1 works the same as 3.1 with this patch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3409963103

   > Again - if I am right - maybe a better solution that is both DRY and side-effect frree for regular DagBag usage, might be another class (DagBagWithExtraBundlePath - I know, bad name) which would extend from the DagBag, do the try/finally around DagBag parse in it's own parse - and use that class specifically in those places that need to extend sys.path.
   
   > The "orignal" way (if I interpret it correctly) addded the sys.path entry not in the "DagBag" - but outside - in the code that we "knew" we don't care about side-effects (for example in dag file processor parsing is done in a forked interpreter, and the interpreter exists after the Dag is serialized).
   
   What would be the difference? `sys.path` will get mangled anyway (if used in non-forked env). Are those 3 cases handled in this PR this kind of "fire and forget the environment" situation?
   
   - `dag_command.py` seems yes to me, it does the work and exit
   - `processor.py` not sure
   - `task_runner.py` not sure
   
   IMHO Approach in this PR even makes those places having the original append only better, since it includes the cleanup also now.
   
   > cleanup code is also messier than I’d like
   
   @uranusjr please suggest better cleanup strategy, happy to improve


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410244724

   @potiuk can you be more precise? Which of the suggested approaches would fit that? Special DagBag class, explicit `append_bundle_to_syspath` argument or even any other?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410510415

   
   > @potiuk like this at given places?
   > 
   > ```
   > dagbag = DagBag(bundle.path, bundle_path=bundle.path, include_examples=False)
   > add_bundle_path_ephemeral_no_cleanup(dagbag) # or better pass bundle or even bundle.path?
   > # + make warn/info line inside the method saying it is permantently changing sys.path
   > ```
   
   Smth like that. I would rather explicitly use bundle.path and do it before DagBag instantiation (this is what happens currently in task runner for example.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3325499653

   @jedcunningham would you mind to help me resolve the one comment? I have no idea what's requested there and rc2 is around the corner. 🙏 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3314961943

   > t configured.
   > 
   > But yes, not a regression. Meaning, it should go in rc2 if merged and we have an rc2, but not enough to warrant an rc2 on its own.
   
   That's my understanding also. 3.0.0 and 3.0.1 was actually wrong and this is actually improvement to bundle path to be appended when needed automatically so you doesn't need to do it manually on all obvious cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3424639521

   > @potiuk good to go and update? WDYT?
   
   Go ahead. 
   
   
   But I think some clarity is needed here. 
   
   In fact, I think in Airflow we prefer to "do" stuff rather than make sure that what we "do" is not going to be commented on and asked to be changed. While it might seem like over-burdening of contributors with potential re-work, this is in fact targeted to optimise the committer's time (which is far more scarce of a resource). That also includes committers proposing changes that other committers review. I often redo and rewrite my PRs multiple times and that's OK. 
   
   It's OK to get later comments that were not received before, and also it's not **guaranteed** that if you got a "go ahed" it means that no more changes will be requested (by the same maintainer or other maintainers). This is simply because a) things change, b) people have more time to think c) before ideas are materialized in a PR, you might not realise somethign d) new evidences show up  ... or e) the comments are suggestion were unclear, ambiguous or misuderstood or  e) simply committer changes their mind seeing the proposed PR.
   
   So - just to be very clear about expectations here - if you (or anyone else - it's not you in particular) get a "go ahead" after comments where you were asked to make a change, it does not automatically mean it will be merged if you follow the comments. This is not "given". It might well be when you rework your change proposal, you will get other comments or even that maintainers who will see it will tell you "no, after seeing it, the original proposal was actually better with that simple thing" - or that you won't get conflicting comments and proposals from another maintainer (even someone who had not been involved so far). This all **might** happen and is even expected to happen quite often.
   
   That's why I suggest to anyone reacting to comment - just do. Make the changes, and then ask to re-review, rather than wait for review, try to get 100% clarity on the changes and only after that - do it. This is far more efficient, optimizes maintainer's time, energy and focus way better, even if it might be that the contributor's individual time is not optimally used. Yes - it's not, and it's ok and pretty much conscious choice of maintainers here on what we are (even without thinking) optimising for, also often that contributor's time / energy and focus is often not "lost" - resulting discussions, learnings, conflicts of opinions are often way better result for the community.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3424678966

   @potiuk I was actually trying to achieve the same, just with the opposite thinking. Do not do changes and annoy others with review until it is at least agreed on the direction. I'm happy to rewrite until mergable, no worries.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3726669022

   Agree with your latest comment, but we’ll need to do this for compat reasons. Feel free to submit refactoring PRs after this if you feel like reworking it; we can discuss what needs to be done there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2664707910


##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -235,6 +235,13 @@ def __init__(
         self.bundle_path = bundle_path
         self.bundle_name = bundle_name
 
+        # Add bundle path to sys.path if provided.
+        # This allows DAG files to import modules from their bundle directory.
+        # No cleanup is performed - this is intentional for ephemeral processes
+        # (dag processor, task runner, CLI) where the process exits after use.
+        if bundle_path and str(bundle_path) not in sys.path:
+            sys.path.append(str(bundle_path))

Review Comment:
   It is extremely counter-intuitive that creating a DagBag object has a system-wide side effect. I would argue this is worse than the previous approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2671395424


##########
airflow-core/src/airflow/models/__init__.py:
##########
@@ -98,6 +99,7 @@ def __getattr__(name):
     "BaseOperator": "airflow.sdk",
     "BaseOperatorLink": "airflow.sdk",
     "BaseXCom": "airflow.sdk.bases.xcom",
+    "BundleDagBag": "airflow.dag_processing.dagbag",

Review Comment:
   This is for backward compatibility. Let’s not add the new class here.



##########
airflow-core/src/airflow/models/__init__.py:
##########
@@ -98,6 +99,7 @@ def __getattr__(name):
     "BaseOperator": "airflow.sdk",
     "BaseOperatorLink": "airflow.sdk",
     "BaseXCom": "airflow.sdk.bases.xcom",
+    "BundleDagBag": "airflow.dag_processing.dagbag",

Review Comment:
   This is for backward compatibility. Let’s not add the new class here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3722774345

   A couple of things we should fix, but otherwise I think this is good!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2671602322


##########
airflow-core/src/airflow/models/__init__.py:
##########


Review Comment:
   I'll remove, just wondering, is there plan to move CLI commands to use DBDagBags also? Or what is plan on those?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2365619889


##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task

Review Comment:
   ```suggestion
                   from airflow.sdk import DAG
   ```



##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -398,12 +398,23 @@ def handler(signum, frame):
 
         def parse(mod_name, filepath):
             try:
-                loader = importlib.machinery.SourceFileLoader(mod_name, filepath)
-                spec = importlib.util.spec_from_loader(mod_name, loader)
-                new_module = importlib.util.module_from_spec(spec)
-                sys.modules[spec.name] = new_module
-                loader.exec_module(new_module)
-                return [new_module]
+                # Add bundle path to sys.path if we have one
+                bundle_path_added = False
+                if self.bundle_path and str(self.bundle_path) not in sys.path:
+                    sys.path.insert(0, str(self.bundle_path))

Review Comment:
   Why not on the back with append?



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import bundle_utils # typo
+
+                with DAG('test_import'):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        assert dagbag.import_errors  # Check import failed
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_none_no_syspath_manipulation(self, tmp_path, caplog):
+        """
+        Test that no sys.path manipulation occurs when bundle_path is None.
+        """
+        dag_file = tmp_path / "simple_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+            from datetime import datetime
+            from airflow.sdk import DAG, BaseOperator

Review Comment:
   ```suggestion
               from airflow.sdk import DAG
   ```



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()

Review Comment:
   ```suggestion
                       ....
   ```



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import bundle_utils # typo
+
+                with DAG('test_import'):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()

Review Comment:
   ```suggestion
                       ...
   ```



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import bundle_utils # typo
+
+                with DAG('test_import'):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        assert dagbag.import_errors  # Check import failed
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_none_no_syspath_manipulation(self, tmp_path, caplog):
+        """
+        Test that no sys.path manipulation occurs when bundle_path is None.
+        """
+        dag_file = tmp_path / "simple_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+            from datetime import datetime
+            from airflow.sdk import DAG, BaseOperator
+            import sys
+
+            dag = DAG("simple_dag", start_date=datetime(2021, 1, 1), schedule=None, description=f"DAG with sys.path: {sys.path}")
+            t1 = BaseOperator(task_id="test_task", dag=dag)

Review Comment:
   ```suggestion
               with DAG("simple_dag", description=f"DAG with sys.path: {sys.path}"):
                   ...
   ```



##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()
+                """
+            )
+        )
+        syspath_before = deepcopy(sys.path)
+        assert str(tmp_path) not in sys.path  # bundle path is not in sys.path before bundle parse
+        dagbag = DagBag(dag_folder=str(dag_file), bundle_path=tmp_path, include_examples=False)
+
+        # Check import was successful
+        assert len(dagbag.dags) == 1
+        assert not dagbag.import_errors
+
+        dag = dagbag.get_dag("test_import")
+        assert dag is not None
+        assert str(tmp_path) in dag.description  # sys.path is enhanced during parse
+
+        assert str(tmp_path) not in sys.path  # bundle path is not preserved in sys.path
+        assert sys.path == syspath_before  # sys.path doesn't change
+
+    def test_bundle_path_syspath_is_cleanup_on_failure(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task

Review Comment:
   ```suggestion
                   from airflow.sdk import DAG
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2373450473


##########
airflow-core/tests/unit/models/test_dagbag.py:
##########
@@ -866,6 +866,98 @@ def mytask():
             DagBag(dag_folder=os.fspath(tmp_path), include_examples=False)
             assert "SIGSEGV signal handler registration failed. Not in the main thread" in caplog.text
 
+    def test_bundle_path_is_in_syspath_during_parse(self, tmp_path):
+        """
+        Test that bundle_path is correctly added to and removed from sys.path during DAG import.
+        """
+        util_file = tmp_path / "bundle_util.py"
+        util_file.write_text('def get_message(): return "Hello from bundle!"')
+
+        dag_file = tmp_path / "test_dag.py"
+        dag_file.write_text(
+            textwrap.dedent(
+                """\
+                from airflow import DAG
+                from airflow.decorators import task
+                import sys
+                import bundle_util
+
+                with DAG('test_import', description=f"DAG with sys.path: {sys.path}"):
+                    @task
+                    def mytask():
+                        print(util.get_message())
+                    mytask()

Review Comment:
   Stylistic ultimately, but I'd remove them or use EmptyOperator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3409286678

   I don’t like this affects the global process and needs a cleanup (and not to mention the cleanup code is also messier than I’d like). The original code did not need cleanup because it’s only run inside a dag processing process and would not affect the parent process. That is a lot cleaner IMO.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410069438

   > @potiuk @uranusjr good, would wrapped `DagBag` carefully applied in those 3 cases be considered improvement over current state and also useful extension of dag command?
   
   I think @uranusjr point about "no cleanup at all" is a good one. The try/finally gives you false sense of "being properly cleaned up". 
   
   So I'd would even say we might simply add the path and not clean it deliberately (maybe a helper method of some sorts). And I think it should be plain obvious in that code that:
   
   1) the lack of cleanup is not a misteke but deliberte choice
   2) that it is only used because of Ephemeral interpreter that is going away soon.
   
   Maybe some `addBundlePathForEphemeralInterpreter` as a helper method (yes - I am bad in naming). 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3410473028

   @potiuk like this at given places?
   
   ```
   dagbag = DagBag(bundle.path, bundle_path=bundle.path, include_examples=False)
   add_bundle_path_ephemeral_no_cleanup(dagbag)
   # + make warn/info line inside the method saying it is permantently changing sys.path
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3395513713

   @potiuk finally time to check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3402239884

   Sure - looks good but have some conflict resolution in the meantime.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on code in PR #55894:
URL: https://github.com/apache/airflow/pull/55894#discussion_r2671631778


##########
airflow-core/src/airflow/models/__init__.py:
##########


Review Comment:
   Ahh, I think I do follow now. The mappings are for backward compatibility. 🧠 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3728782867

   :partying_face: Thanks everyone for a guidance and patience!
   
   _It is actually my :birthday: today, this is nice accidental :gift:!_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add bundle_path temporarily to sys.path during DagBag parsing. [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55894:
URL: https://github.com/apache/airflow/pull/55894#issuecomment-3678236274

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org