You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/09 16:05:17 UTC

[GitHub] [airflow] ashb opened a new pull request, #23592: Automatically register DAGs that are used in a context manager

ashb opened a new pull request, #23592:
URL: https://github.com/apache/airflow/pull/23592

   This has caused a bit of confusion to committers (missing off the `as dag`) and is just more user friendly
   
   This is on by default but can be disabled by passing `auto_register=False` do a DAG
   
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] kaxil commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
kaxil commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1121358430

   >Tying registration of DAGs to with is perhaps a bit confusing in this sense – also, you can use with on a particular dag instance multiple times!
   
   @malthe Is it the param name that is confusing? I do not understand I follow what you are saying. The use-case for this PR is that we don't need to needlessly do a `with DAG(..) as dag` as there is no need to have the `as dag` in it unless someone wants to actually use the `dag` variable itself
   
   >What about DAGs declared in functions?
   Isn't it the same behaviour as the one with `with DAG(..) as dag` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r935383612


##########
tests/models/test_dagbag.py:
##########
@@ -937,8 +953,7 @@ def test_get_dag_with_dag_serialization(self):
     def test_collect_dags_from_db(self):
         """DAGs are collected from Database"""
         db.clear_db_dags()
-        example_dags_folder = airflow.example_dags.__path__[0]
-        dagbag = DagBag(example_dags_folder)
+        dagbag = DagBag(str(example_dags_folder))

Review Comment:
   Yeah, that was my thought -- it was orthogonal to this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1206212365

   I still need to update the docs/tutorial etc to reflect this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r947370929


##########
airflow/example_dags/tutorial.py:
##########
@@ -97,7 +97,7 @@
     """
     )
 
-    dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG
+    dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG; OR

Review Comment:
   ```suggestion
       dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG
   ```
   
   accidental?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1212932788

   Reminder to self: finish this before 2.4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r934173133


##########
tests/models/test_dagbag.py:
##########
@@ -937,8 +953,7 @@ def test_get_dag_with_dag_serialization(self):
     def test_collect_dags_from_db(self):
         """DAGs are collected from Database"""
         db.clear_db_dags()
-        example_dags_folder = airflow.example_dags.__path__[0]
-        dagbag = DagBag(example_dags_folder)
+        dagbag = DagBag(str(example_dags_folder))

Review Comment:
   Maybe we should just allow `DagBag` to take `Path`, but this doesn’t need to be done now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r964312268


##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -133,13 +134,16 @@ As long as a ``DAG`` object in ``globals()`` is created, Airflow will load it.
 
             print_message(config["message"])
 
-        globals()[dag_id] = dynamic_generated_dag()
+        dynamic_generated_dag()
 
 The code below will generate a DAG for each config: ``dynamic_generated_dag_config1`` and ``dynamic_generated_dag_config2``.
 Each of them can run separately with related configuration

Review Comment:
   ```suggestion
   Each of them can run separately with related configuration.
   
   If you do not wish to have DAGs auto-registered, you can disable the behavior by setting ``auto_register=False`` on your DAG.
   ```



##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -107,8 +107,9 @@ the meta-data file in your DAG easily. The location of the file to read can be f
     # Configuration dict is available here
 
 
-Dynamic DAGs with ``globals()``
-...............................
+Registering dynamic DAGs
+........................
+
 You can dynamically generate DAGs by working with ``globals()``.
 As long as a ``DAG`` object in ``globals()`` is created, Airflow will load it.

Review Comment:
   Maybe something like this instead?
   
   ```suggestion
   You can dynamically generate DAGs when using the ``@dag`` decorator or the ``with DAG(..)`` context manager.
   Airflow will automatically register them.
   ```



##########
newsfragments/23592.significant.rst:
##########
@@ -0,0 +1,24 @@
+DAGS used in a context manager no longer need to be assigned to a module variable

Review Comment:
   This should also cover the `@dag` decorator, no?



##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -133,13 +134,16 @@ As long as a ``DAG`` object in ``globals()`` is created, Airflow will load it.
 
             print_message(config["message"])
 
-        globals()[dag_id] = dynamic_generated_dag()
+        dynamic_generated_dag()
 
 The code below will generate a DAG for each config: ``dynamic_generated_dag_config1`` and ``dynamic_generated_dag_config2``.

Review Comment:
   ```suggestion
   The code above will generate a DAG for each config: ``dynamic_generated_dag_config1`` and ``dynamic_generated_dag_config2``.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1121293812

   This works for the current example sub-dags we have in the tests -- is there any other case I should test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r964884461


##########
airflow/models/dagbag.py:
##########
@@ -306,6 +314,8 @@ def _load_modules_from_file(self, filepath, safe_mode):
         if mod_name in sys.modules:
             del sys.modules[mod_name]
 
+        DagContext.current_autoregister_module_name = mod_name

Review Comment:
   Yeah, I agree, This is essentially used as a global variable.
   
   It's not very easy to refactor given it's use across three or so different functions, so a context manager would be tricky. (I.e. I can't see an easy refactor to make it work right now)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r935695857


##########
airflow/models/dag.py:
##########
@@ -312,6 +316,7 @@ class DAG(LoggingMixin):
         ``Environment`` is used to render templates as string values.
     :param tags: List of tags to help filtering DAGs in the UI.
     :param schedule_on: List of upstream datasets if for use in triggering DAG runs.
+    :param auto_register: Automatically register this DAG when it is used in a ``with`` block

Review Comment:
   It's only relevant to context manager yes, but this is the only place you can add a flag to change the behavoiur of _one_ dag.
   
   I think we could add a new config flag (default to True) that globally sets the auto_register behaviour, but I think we still need the way to change the behaviour for a single dag.
   
   The other option I can think of is this:
   
   ```python
   with DAG(dag_id=...).with_autoregister(False):
       ...
   ```
   
   I don't have a strong preference either way



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r947578085


##########
airflow/example_dags/tutorial.py:
##########
@@ -97,7 +97,7 @@
     """
     )
 
-    dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG
+    dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG; OR

Review Comment:
   Oh, this needs to be read with a next lines as one sentence. Got it now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r937851593


##########
airflow/models/dag.py:
##########
@@ -312,6 +316,7 @@ class DAG(LoggingMixin):
         ``Environment`` is used to render templates as string values.
     :param tags: List of tags to help filtering DAGs in the UI.
     :param schedule_on: List of upstream datasets if for use in triggering DAG runs.
+    :param auto_register: Automatically register this DAG when it is used in a ``with`` block

Review Comment:
   Yeah. I like it the way it is now. with X.with_something() does not look good, and the current way is more explicit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r964103714


##########
airflow/models/dagbag.py:
##########
@@ -306,6 +314,8 @@ def _load_modules_from_file(self, filepath, safe_mode):
         if mod_name in sys.modules:
             del sys.modules[mod_name]
 
+        DagContext.current_autoregister_module_name = mod_name

Review Comment:
   This in-place operation feels a bit weird to me… Perhaps a context manager would be easier to understand and maintain? Something like
   
   ```python
   with DagContext.enable_autoregister(mod_name):
       parse(...)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1129060336

   We could extend the auto-register -- but that makes it slightly more complex for SubDag (we'd have to "unregister" a dag when it gets turned in to a sub dag) but we could do it just as easily if we want to.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] feluelle commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
feluelle commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r935505540


##########
airflow/models/dag.py:
##########
@@ -312,6 +316,7 @@ class DAG(LoggingMixin):
         ``Environment`` is used to render templates as string values.
     :param tags: List of tags to help filtering DAGs in the UI.
     :param schedule_on: List of upstream datasets if for use in triggering DAG runs.
+    :param auto_register: Automatically register this DAG when it is used in a ``with`` block

Review Comment:
   If this is only applicable to context manager calls I think this is not the correct place to pass it or? Wouldn't it make more sense to define this on config level for all DAGs using context manager i.e. in the config?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb merged pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb merged PR #23592:
URL: https://github.com/apache/airflow/pull/23592


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r947572910


##########
airflow/example_dags/tutorial.py:
##########
@@ -97,7 +97,7 @@
     """
     )
 
-    dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG
+    dag.doc_md = __doc__  # providing that you have a docstring at the beginning of the DAG; OR

Review Comment:
   No, intentional but probably shouldn't be in this PR. (The tutorial is showing multiple ways of setting `dag.doc` -- you wouldn't want to do all of them.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r964103714


##########
airflow/models/dagbag.py:
##########
@@ -306,6 +314,8 @@ def _load_modules_from_file(self, filepath, safe_mode):
         if mod_name in sys.modules:
             del sys.modules[mod_name]
 
+        DagContext.current_autoregister_module_name = mod_name

Review Comment:
   This in-place operation feels a bit weird to me… Perhaps a context manager would be easier to understand and maintain? Something like
   
   ```python
   with DagContext.enable_autoregister(mod_name):
       parse(...)
   ```
   
   This probably needs some refactoring to work though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] feluelle commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
feluelle commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r938611987


##########
airflow/models/dag.py:
##########
@@ -312,6 +316,7 @@ class DAG(LoggingMixin):
         ``Environment`` is used to render templates as string values.
     :param tags: List of tags to help filtering DAGs in the UI.
     :param schedule_on: List of upstream datasets if for use in triggering DAG runs.
+    :param auto_register: Automatically register this DAG when it is used in a ``with`` block

Review Comment:
   Yeah if we only have these two options, I would also keep it like it is. 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] malthe commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
malthe commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1121493040

   @kaxil I understand the use-case – what I am somewhat hesitant to vote for is the conflation of the two orthogonal concerns which are both suggested for `with` here:
   
   1. As a scope for tasks created which automatically get the `dag` parameter set;
   2. As a mechanism to automatically register a DAG with the loading mechanism.
   
   I think you could argue then that _any_ DAG instance created during the loading of a module should be automatically registered – after all, why would a user create a DAG instance only to throw it away?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] feluelle commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
feluelle commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r935467143


##########
newsfragments/23592.significant.rst:
##########
@@ -0,0 +1,24 @@
+DAGS used in a context manager no longer need to be assigned to a module variable
+
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this

Review Comment:
   ```suggestion
   DAGs used in a context manager no longer need to be assigned to a module variable.
   
   Previously you had to assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
   ```



##########
newsfragments/23592.significant.rst:
##########
@@ -0,0 +1,24 @@
+DAGS used in a context manager no longer need to be assigned to a module variable
+
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
+
+.. code-block::
+
+   with DAG(dag_id="example") as dag:
+      ...
+
+can become
+
+.. code-block::
+
+   with DAG(dag_id="example"):
+      ...
+
+
+If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag::

Review Comment:
   ```suggestion
   If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] malthe commented on pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
malthe commented on PR #23592:
URL: https://github.com/apache/airflow/pull/23592#issuecomment-1121317236

   @ashb there are at least two things to consider here which I stumbled upon playing around with this:
   
   1. There's nothing special about `with` except that tasks are automatically assigned to this dag (i.e., `dag=dag`). Tying registration of DAGs to `with` is perhaps a bit confusing in this sense – also, you can use `with` on a particular dag instance multiple times!
   
   2. What about DAGs declared in functions?
   
       ```python
       def test():
           with DAG():
               ...
       test()
       ```
   
       I don't mind but it is a little magic perhaps, that this automatically _declares_ the DAG.
   
   I ended up instead with this pattern:
   ```python
   dag = DAG(...)
   
   with dag:
       ...
   ```
   – as a recommended pattern, simply because then you can't forget to give the DAG instance a top-level variable.
   
   And then as I mention in https://lists.apache.org/thread/xndt3zklxbzo5nlmjz10dtfm2pp4t4wq, we could instead warn the user if they'd created a DAG which isn't exposed/registered.
   
   (The gist of that linked reply is that there are two different ways to do that.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r900027678


##########
airflow/models/dag.py:
##########
@@ -3022,24 +3029,28 @@ class DagContext:
 
     """
 
-    _context_managed_dag: Optional[DAG] = None
-    _previous_context_managed_dags: List[DAG] = []
+    _context_managed_dags: Deque[DAG] = deque()

Review Comment:
   No particular reason no -- mostly thinking that we only ever need to access the head value, or push a new head value, and deque is more optimized for that case.
   
   But it hardly matters as this is not on the critical path for performance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #23592: Automatically register DAGs that are used in a context manager

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #23592:
URL: https://github.com/apache/airflow/pull/23592#discussion_r881282355


##########
airflow/models/dag.py:
##########
@@ -3022,24 +3029,28 @@ class DagContext:
 
     """
 
-    _context_managed_dag: Optional[DAG] = None
-    _previous_context_managed_dags: List[DAG] = []
+    _context_managed_dags: Deque[DAG] = deque()

Review Comment:
   Any particular reason we need a deque here? From what I can tell a plain list is enough (with the ordering reversed, i.e. we append to / pop from the end and use -1 to access).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org