[GitHub] [airflow] sharmaTanmay opened a new issue #7858: ShortCircuitOperator Ignoring Trigger Rules for Downstream Task and Skipping all Tasks

[GitBox <gi...@apache.org>]

sharmaTanmay opened a new issue #7858: ShortCircuitOperator Ignoring Trigger Rules for Downstream Task and Skipping all Tasks
URL: https://github.com/apache/airflow/issues/7858
 
 
   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   <!--
   
   IMPORTANT!!!
   
   Please complete the next sections or the issue will be closed.
   This questions are the first thing we need to know to understand the context.
   
   -->
   
   **Apache Airflow version**: 1.10.3
   
   **What happened**: I'm trying to use a ShortCircuitOperator with a two downstream tasks, one of which has a `trigger_rule` set as `all_done`. But despite the trigger_rule, both of these tasks are getting skipped. As per documentation,  skipped tasks should not cascade through with a `trigger_rule` set as `all_done`. https://airflow.apache.org/docs/stable/concepts.html?highlight=trigger
   
   **What you expected to happen**: That the task with `trigger_rule` should execute the piece of code inside it.
   
   **How to reproduce it**:
   
   from airflow import DAG
   from airflow.operators.python_operator import PythonOperator, ShortCircuitOperator
   
   dag = DAG(
       dag_id='dag_name',
       orientation="LR",
       schedule_interval=None,
   )
   
   def shortcircuit_func(**context):
       return False
   
   
   shortcircuit = ShortCircuitOperator(
       task_id='shortcircuit',
       dag=dag,
       python_callable=shortcircuit_func
   )
   
   
   def task1_func():
       print("hello world")
   
   
   task1 = PythonOperator(
       task_id='task1',
       dag=dag,
       python_callable=task1_func
   )
   
   def task3_func():
       for x in range(10):
           print("ALL DONE", x)
   
   
   task3 = PythonOperator(
       task_id='task3',
       dag=dag,
       python_callable=task3_func,
       trigger_rule='all_done',
       depends_on_past=True
   )
   
   
   def task2_func():
       print("task2_func")
   
   
   task2 = PythonOperator(
       task_id='task2',
       dag=dag,
       python_callable=task2_func,
   )
   
   task1 >> shortcircuit >> task2 >> task3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] eladkal commented on issue #7858: ShortCircuitOperator Ignoring Trigger Rules for Downstream Task and Skipping all Tasks

[GitBox <gi...@apache.org>]

eladkal commented on issue #7858: ShortCircuitOperator Ignoring Trigger Rules for Downstream Task and Skipping all Tasks
URL: https://github.com/apache/airflow/issues/7858#issuecomment-615863490
 
 
   When the callable of `ShortCircuitOperator` is evaluated to `False` it short-circuits the entire downstream branch. This means that all downstream tasks are set to `SKIP` immediately. This is done by calling `skip` method of `SkipMixin`.
   The `ShortCircuitOperator` is more for the usage of "Stop here. don't continue no matter what."
   
   The docs seems to be written by the spirit of `BranchPythonOperator` where it uses `skip_all_except` method of `SkipMixin`. It's not quite the same because it checks for dependency between tasks rather than trigger rules but at it uses a less violent approach.
   
   I'm not sure if this this is a bug in the `ShortCircuitOperator`. I for once have workflows that need the functionality exactly as it is now so I think this is more case of wrong documentation issue rather than a bug.
   
   On the other hand maybe the `ShortCircuitOperator` can be enhanced to have two modes: `hard` & `soft`. Where `hard` is the same behavior as now while `soft` will check if SKIP should cascade further.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services