You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sanederchik (via GitHub)" <gi...@apache.org> on 2025/04/30 12:27:45 UTC

[I] Another way for reusing DAGs not stated in docs [airflow]

sanederchik opened a new issue, #50020:
URL: https://github.com/apache/airflow/issues/50020

   ### What do you see as an issue?
   
   [In official docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html) there have been stated several ways for dynamic DAGs generation. I found out another way which looks very convenient, so I'd like to get your opinion of it.
   
   ### Solving the problem
   
   ```
   # reusable_dag.py
   
   import datetime
   
   from airflow.decorators import task
   from airflow import DAG
   
   DEFAULT_DAG_CONFIG = {
       'schedule_interval': None,
       'catchup': False,
       'start_date': datetime.datetime.fromisoformat('2025-04-29'),
       'default_args': {
           'depends_on_past': False
       },
       'max_active_tasks': 1,
       'max_active_runs': 1
   }
   
   def reusable_dag_generator(
           dag_obj: DAG,
           printable_msg: str
   ):
      """Reusable dag generator used as a template for DAG generation"""
   
       with dag_obj:
   
           @task()
           def start_dag():
               pass
   
           @task()
           def print_msg():
               print(printable_msg)
   
           @task()
           def finish_dag():
               pass
   
           start_dag() >> print_msg() >> finish_dag()
       return dag_obj
   
   # dag0.py
   
   from airflow import DAG
   from reusable_dag import reusable_dag_generator, DEFAULT_DAG_CONFIG
   
   dag = reusable_dag_generator(
       dag_obj=DAG(
           dag_id='test0',
           **DEFAULT_DAG_CONFIG
       ),
       printable_msg='Hello DAG0!'
   )
   
   # dag1.py
   
   from airflow import DAG
   from reusable_dag import reusable_dag_generator, DEFAULT_DAG_CONFIG
   
   dag = reusable_dag_generator(
       dag_obj=DAG(
           dag_id='test1',
           **DEFAULT_DAG_CONFIG
       ),
       printable_msg='Hello DAG1!'
   )
   ```
   
   ### Anything else
   
   If provided way looks convenient, I would be glad to make a PR for docs enhancements.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-2865696989

   There are many ways you can write such generation - If you think you can add is as yet another example in a clear and conscious way and explain why it's different - feel free to add another example to the page linked above - but it has to be clear why it's different and concisely explain what you get from it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-6410499716

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-2841814238

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "sanederchik (via GitHub)" <gi...@apache.org>.
sanederchik commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-2850615737

   I think you misunderstood a little bit my point, so I'd like to clarify, sorry if I had missled you before.
   
   Your example from docs shows dynamic DAG generation where all DAG config and creation process is provided within single DAG file. And it is very good for creating several DAGs from one script file with the same structure but different configs.
   
   However in production environment it may be convenient to add an additional layer of abstraction by moving DAG generation config, tasks, etc. to separate modules. And then reuse them either in dynamic DAGs generation, like in example in docs, or in single DAG generation like in my example. 
   
   Additionally, it becomes easier to cover a "reusable DAG" with tests.
   
   So the main point of my issue is about creating `reusable DAG` modules, not about only dynamic DAG generation (but of course it's the main way to use it).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-3447963949

   @potiuk shouldn't this be closed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "gyli (via GitHub)" <gi...@apache.org>.
gyli commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-2864946146

   I get your point that you want to wrap the DAG orchestration in a function to make it reusable. However, I just don't see how much it differs from the way under [Generating Python code with embedded meta-data](https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html#generating-python-code-with-embedded-meta-data), only other than parameterizing `dag_obj`.
   
   I am happy to hear other opinions though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Another way for reusing DAGs not stated in docs [airflow]

Posted by "gyli (via GitHub)" <gi...@apache.org>.
gyli commented on issue #50020:
URL: https://github.com/apache/airflow/issues/50020#issuecomment-2849809553

   Hi @sanederchik, since you are willing to submit a PR, can I assign this issue to you?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org