[GitHub] [airflow] eladkal commented on a diff in pull request #26400: Create a more efficient airflow dag test command that also has better local logging

[GitBox <gi...@apache.org>]

eladkal commented on code in PR #26400:
URL: https://github.com/apache/airflow/pull/26400#discussion_r982244700


##########
docs/apache-airflow/executor/debug.rst:
##########
@@ -15,11 +15,59 @@
     specific language governing permissions and limitations
     under the License.
 
+Testing DAGs with dag.test()
+=============================
+
+To debug DAGs in an IDE, you can set up the ``dag.test`` command in your dag file and run through your DAG in a single
+serialized python process.
+
+This approach can be used with any supported database (including a local SQLite database) and will
+*fail fast* as all tasks run in a single process.
+
+To set up ``dag.test``, add these two lines to the bottom of your dag file:
+
+.. code-block:: python
+
+  if __name__ == "__main__":
+      dag.test()
+
+and that's it! You can add argument such as ``execution_date`` if you want to test argument-specific dagruns, but otherwise
+you can run or debug DAGs as needed.
+
+Comparison with DebugExecutor
+*****************************
+
+The ``dag.test`` command has the following benefits over the :class:`~airflow.executors.debug_executor.DebugExecutor`
+class, which is now deprecated:
+
+1. It does not require running an executor at all. Tasks are run one at a time with no executor or scheduler logs.
+2. It is significantly faster than running code with a DebugExecutor as it does not need to go through a scheduler loop.
+3. It does not perform a backfill.
+
+
+Debugging Airflow DAGs on the command line
+==========================================
+
+With the same two line addition as mentioned in the above section, you can now easily debug a DAG using pdb as well.
+Run ``python -m pdb <path to dag file>.py`` for an interactive debugging experience on the command line.
+
+.. code-block:: bash
+
+  root@ef2c84ad4856:/opt/airflow# python -m pdb airflow/example_dags/example_bash_operator.py
+  > /opt/airflow/airflow/example_dags/example_bash_operator.py(18)<module>()
+  -> """Example DAG demonstrating the usage of the BashOperator."""
+  (Pdb) b 45
+  Breakpoint 1 at /opt/airflow/airflow/example_dags/example_bash_operator.py:45
+  (Pdb) c
+  > /opt/airflow/airflow/example_dags/example_bash_operator.py(45)<module>()
+  -> bash_command='echo 1',
+  (Pdb) run_this_last
+  <Task(EmptyOperator): run_this_last>
 
 .. _executor:DebugExecutor:
 
-Debug Executor
-==================
+Debug Executor (deprecated)

Review Comment:
   Does this mean we want to remove DebugExecutor in Airflow 3?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org