You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "respecht (via GitHub)" <gi...@apache.org> on 2023/10/16 19:05:53 UTC
[I] Dag File Processor Does Not Detect Dynamically Generated Dags [airflow]
respecht opened a new issue, #34978:
URL: https://github.com/apache/airflow/issues/34978
### Apache Airflow version
2.7.2
### What happened
If a DAG is dynamically generated, with the code for that generation in a separate file, the Dag File Processor will not pick up the file where the DAG-generating function is called. See reproduction for an example.
A workaround for this is to import airflow.decorators.dag at the top of the file where the DAG is called. Although this is an unused import, the Dag File Processor will process the file and recognize the DAG.
### What you think should happen instead
Airflow should pick up valid DAGs in the dags directory regardless of how they are instantiated.
### How to reproduce
DAG factory at dags/dag_factory.py:
```
import time
from airflow.decorators import dag, task
from airflow.utils.dates import days_ago
def generate_dag(name):
@dag(name, start_date=days_ago(10), tags=['test'])
def job_dag():
@task
def add_ids_to_queue():
time.sleep(5)
print('Task complete!')
add_ids_to_queue()
return job_dag()
```
dags/test_no_import_dag.py (does not show up in DAG list)
```
from dag_factory import generate_dag
generate_dag('test_no_import')
```
dags/test_with_import_dag.py (shows up in DAG list)
```
from dag_factory import generate_dag
from airflow.decorators import dag
generate_dag('test_has_import')
```
Although the import `from airflow.decorators import dag` is unused, it is a workaround for having Airflow pick up these dynamically generated DAGs.
### Operating System
Mac OSX Ventura 13.5.1
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.7.2
apache-airflow-providers-ftp==3.5.2
apache-airflow-providers-http==4.5.2
apache-airflow-providers-imap==3.3.2
apache-airflow-providers-sqlite==3.4.3
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I created a clean Airflow deployment following the instructions at https://airflow.apache.org/docs/apache-airflow/stable/start.html
### Anything else
Latest logs from logs/dag_processor_manager/dag_processor_manager.log:
```
[2023-10-16T15:03:48.829-0400] {manager.py:887} INFO -
================================================================================
DAG File Processing Stats
File Path PID Runtime # DAGs # Errors Last Runtime Last Run
-------------------------------------------------------------------------------------------------------------------------- ----- --------- -------- ---------- -------------- -------------------
/Users/RSpecht/airflow/dags/dag_factory.py 0 0 0.57s 2023-10-16T19:03:23
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py 0 0 0.58s 2023-10-16T19:02:57
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api_virtualenv.py 0 0 0.58s 2023-10-16T19:03:30
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_kubernetes_executor.py 0 0 0.57s 2023-10-16T19:03:01
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/plugins/workday.py 0 0 0.58s 2023-10-16T19:03:26
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/plugins/event_listener.py 0 0 0.58s 2023-10-16T19:03:29
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/plugins/listener_plugin.py 0 0 0.63s 2023-10-16T19:03:28
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/subdags/subdag.py 0 0 0.56s 2023-10-16T19:03:25
/Users/RSpecht/airflow/dags/test_with_import_dag.py 1 0 0.62s 2023-10-16T19:03:22
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_latest_only.py 1 0 0.63s 2023-10-16T19:03:00
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_task_group_decorator.py 1 0 0.68s 2023-10-16T19:03:39
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_complex.py 1 0 0.64s 2023-10-16T19:03:10
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_python_operator.py 1 0 0.65s 2023-10-16T19:02:34
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_params_trigger_ui.py 1 0 0.65s 2023-10-16T19:02:39
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_bash_operator.py 1 0 0.61s 2023-10-16T19:03:21
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_short_circuit_operator.py 1 0 0.64s 2023-10-16T19:03:47
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_setup_teardown.py 1 0 0.63s 2023-10-16T19:02:30
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_operator.py 1 0 0.62s 2023-10-16T19:03:14
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_labels.py 1 0 0.62s 2023-10-16T19:03:16
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py 1 0 0.63s 2023-10-16T19:03:32
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_task_group.py 1 0 0.69s 2023-10-16T19:03:41
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial.py 1 0 0.64s 2023-10-16T19:03:34
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_sla_dag.py 1 0 0.67s 2023-10-16T19:03:44
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_passing_params_via_test_command.py 1 0 0.64s 2023-10-16T19:02:36
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_latest_only_with_trigger.py 1 0 0.62s 2023-10-16T19:02:59
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_time_delta_sensor_async.py 1 0 0.64s 2023-10-16T19:03:38
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_dag_decorator.py 1 0 0.82s 2023-10-16T19:03:08
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_xcom.py 1 0 0.62s 2023-10-16T19:02:54
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_operator_decorator.py 1 0 0.65s 2023-10-16T19:03:13
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_day_of_week_operator.py 1 0 0.63s 2023-10-16T19:03:17
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_skip_dag.py 1 0 0.63s 2023-10-16T19:03:45
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_short_circuit_decorator.py 1 0 0.72s 2023-10-16T19:03:48
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial_dag.py 1 0 0.61s 2023-10-16T19:03:33
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_trigger_target_dag.py 1 0 0.64s 2023-10-16T19:02:55
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_sensor_decorator.py 1 0 0.63s 2023-10-16T19:02:32
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_python_dop_operator_3.py 1 0 0.63s 2023-10-16T19:03:12
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_sensors.py 1 0 0.64s 2023-10-16T19:02:31
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_setup_teardown_taskflow.py 1 0 0.63s 2023-10-16T19:02:28
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_nested_branch_dag.py 1 0 0.68s 2023-10-16T19:02:41
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py 1 0 0.62s 2023-10-16T19:03:04
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_trigger_controller_dag.py 1 0 0.63s 2023-10-16T19:03:36
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_params_ui_tutorial.py 1 0 0.67s 2023-10-16T19:02:37
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_dynamic_task_mapping.py 1 0 0.61s 2023-10-16T19:03:05
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_external_task_marker_dag.py 2 0 0.59s 2023-10-16T19:03:02
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_xcomargs.py 2 0 0.63s 2023-10-16T19:02:52
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_datetime_operator.py 3 0 0.61s 2023-10-16T19:03:19
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_subdag_operator.py 3 0 0.66s 2023-10-16T19:03:43
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py 6 0 0.62s 2023-10-16T19:03:06
================================================================================
```
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag File Processor Does Not Detect Dynamically Generated Dags [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34978:
URL: https://github.com/apache/airflow/issues/34978#issuecomment-1765111534
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag File Processor Does Not Detect Dynamically Generated Dags [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #34978: Dag File Processor Does Not Detect Dynamically Generated Dags
URL: https://github.com/apache/airflow/issues/34978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org