You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/19 05:09:33 UTC
[GitHub] [airflow] jedcunningham commented on a diff in pull request #25121: Add "Optimizing" chapter to dynamic-dags section
jedcunningham commented on code in PR #25121:
URL: https://github.com/apache/airflow/pull/25121#discussion_r924055943
##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -140,3 +140,74 @@ Each of them can run separately with related configuration
.. warning::
Using this practice, pay attention to "late binding" behaviour in Python loops. See `that GitHub discussion <https://github.com/apache/airflow/discussions/21278#discussioncomment-2103559>`_ for more details
+
+
+Optimizing DAG parsing delays during execution
+----------------------------------------------
+
+Sometimes when you generate a lot of Dynamic DAGs from a single DAG file, it might cause unnecessary delays
+when the DAG file is parsed during task execution. This is visible as delay before a task starts
+and when it gets executed. Sometimes the delays might be minutes.
+
+Why is this happening ?
+
+You might not be aware but when your task is executed just before execution, airflow parses the Python
+file the dag comes from. When tasks are executed, parsing time of dynamically generated DAGs
+when task are executed can be optimized. This optimization is most effective when the
+number of generated DAGs is high.
Review Comment:
```suggestion
Why is this happening? You might not be aware but just before your task is executed, airflow parses
the Python file the dag comes from.
```
I think these last 2 sentences are here too early in this section.
##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -140,3 +140,74 @@ Each of them can run separately with related configuration
.. warning::
Using this practice, pay attention to "late binding" behaviour in Python loops. See `that GitHub discussion <https://github.com/apache/airflow/discussions/21278#discussioncomment-2103559>`_ for more details
+
+
+Optimizing DAG parsing delays during execution
+----------------------------------------------
+
+Sometimes when you generate a lot of Dynamic DAGs from a single DAG file, it might cause unnecessary delays
+when the DAG file is parsed during task execution. This is visible as delay before a task starts
+and when it gets executed. Sometimes the delays might be minutes.
+
+Why is this happening ?
+
+You might not be aware but when your task is executed just before execution, airflow parses the Python
+file the dag comes from. When tasks are executed, parsing time of dynamically generated DAGs
+when task are executed can be optimized. This optimization is most effective when the
+number of generated DAGs is high.
+
+The Airflow Scheduler (or DAG Processor) requires loading of a complete DAG file to process all metadata.
+However, task execution requires only a single DAG object to execute a task. Knowing this, allows us to
+skip the generation of unnecessary DAG objects when task is executed, shortening the parsing time.
+Upon evaluation of a DAG file, command line arguments are supplied which we can use to determine which
+airflow component performs parsing:
+
+* Scheduler/DAG Processor args: ``["scheduler"]`` or ``["dag-processor"]``
+* Task execution args: ``["airflow", "tasks", "run", "dag_id", "task_id", ...]``
+
+However, depending on the executor used and forking model, those args might be available via ``sys.args``
+or via name of the process running. Airflow either executes tasks via running a new Python interpreter or
+sets the name of the process as "airflow task supervisor: {ARGS}".
+
+Upon iterating over the collection of things to generate DAGs for, you can use these arguments to determine
+whether you need to generate all DAG objects (when parsing in the DAG File processor), or to generate only
+a single DAG object (when executing the task).
+
+
+.. code-block:: python
+ :emphasize-lines: 1,2,3,7,8
+
+ import sys
+ import ast
+
+ current_dag = None
+ if len(sys.argv) > 3 and sys.argv[1] == "tasks":
+ current_dag = sys.argv[3]
+ else:
+ try:
+ PROCTITLE_PREFIX = "airflow task supervisor: "
+ proctitle = str(setproctitle.getproctitle())
+ if proctitle.startswith(PROCTITLE_PREFIX):
+ args_string = proctitle[len(PROCTITLE_PREFIX) :]
+ args = ast.literal_eval(args_string)
+ if len(args) > 3 and args[1] == "tasks":
+ current_dag = args[3]
+ except:
+ pass
+
+ for thing in list_of_things:
+ dag_id = f"generated_dag_{thing}"
+ if current_dag is not None and current_dag != dag_id:
+ continue # skip generation of non-selected DAG
+
+ dag = DAG(dag_id=dag_id, ...)
+ globals()[dag_id] = dag
+
+
+Checking if the second argument is ``tasks`` also allows to optimize ``airflow tasks test`` command
Review Comment:
```suggestion
Checking if the second argument is ``tasks`` also allows us to optimize the ``airflow tasks test`` command
```
##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -140,3 +140,74 @@ Each of them can run separately with related configuration
.. warning::
Using this practice, pay attention to "late binding" behaviour in Python loops. See `that GitHub discussion <https://github.com/apache/airflow/discussions/21278#discussioncomment-2103559>`_ for more details
+
+
+Optimizing DAG parsing delays during execution
+----------------------------------------------
+
+Sometimes when you generate a lot of Dynamic DAGs from a single DAG file, it might cause unnecessary delays
+when the DAG file is parsed during task execution. This is visible as delay before a task starts
+and when it gets executed. Sometimes the delays might be minutes.
+
+Why is this happening ?
+
+You might not be aware but when your task is executed just before execution, airflow parses the Python
+file the dag comes from. When tasks are executed, parsing time of dynamically generated DAGs
+when task are executed can be optimized. This optimization is most effective when the
+number of generated DAGs is high.
+
+The Airflow Scheduler (or DAG Processor) requires loading of a complete DAG file to process all metadata.
+However, task execution requires only a single DAG object to execute a task. Knowing this, allows us to
+skip the generation of unnecessary DAG objects when task is executed, shortening the parsing time.
+Upon evaluation of a DAG file, command line arguments are supplied which we can use to determine which
Review Comment:
```suggestion
This optimization is most effective when the number of generated DAGs is high.
Upon evaluation of a DAG file, command line arguments are supplied which we can use to determine which
```
This is a better place for it I think.
##########
docs/apache-airflow/howto/dynamic-dag-generation.rst:
##########
@@ -140,3 +140,74 @@ Each of them can run separately with related configuration
.. warning::
Using this practice, pay attention to "late binding" behaviour in Python loops. See `that GitHub discussion <https://github.com/apache/airflow/discussions/21278#discussioncomment-2103559>`_ for more details
+
+
+Optimizing DAG parsing delays during execution
+----------------------------------------------
+
+Sometimes when you generate a lot of Dynamic DAGs from a single DAG file, it might cause unnecessary delays
+when the DAG file is parsed during task execution. This is visible as delay before a task starts
+and when it gets executed. Sometimes the delays might be minutes.
+
+Why is this happening ?
+
+You might not be aware but when your task is executed just before execution, airflow parses the Python
+file the dag comes from. When tasks are executed, parsing time of dynamically generated DAGs
+when task are executed can be optimized. This optimization is most effective when the
+number of generated DAGs is high.
+
+The Airflow Scheduler (or DAG Processor) requires loading of a complete DAG file to process all metadata.
+However, task execution requires only a single DAG object to execute a task. Knowing this, allows us to
Review Comment:
```suggestion
However, task execution requires only a single DAG object to execute a task. Knowing this, we can
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org