You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/10 14:10:32 UTC

[GitHub] [airflow] darwinyip opened a new issue #10275: Trigger rule for "always"

darwinyip opened a new issue #10275:
URL: https://github.com/apache/airflow/issues/10275


   **Description**
   
   Trigger rule for `always`.
   
   **Use case / motivation**
   
   This comes in handy when there is a need to clean up resources at the end of the DAG run no matter whether it fails or succeeds.
   
   For example:
   ```
   allocate >> a >> b >> c >> d >> cleanup
   ```
   
   The biggest issue is that if an error occurs before `d`, `cleanup` will never get executed.
   
   Example use case is using BigQuery's Reservation API. We want to reserve slots at the beginning of the run and deallocate at the end. Not deallocating would mean a rampant bill at the end of the month.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] darwinyip commented on issue #10275: Trigger rule for "always"

Posted by GitBox <gi...@apache.org>.
darwinyip commented on issue #10275:
URL: https://github.com/apache/airflow/issues/10275#issuecomment-671598774


   Current workaround
   
   ```
   with DAG(DAG_NAME, schedule_interval='@daily', default_args=default_args) as dag:
       task1 = DummyOperator(task_id='task1')
       task2 = BashOperator(task_id='task2', bash_command='exit 1')
       task3 = DummyOperator(task_id='task3')
   ​
       task1 >> task2 >> task3
   ​
       sensor = ExternalTaskSensor(task_id='sensor',
                                   external_dag_id=DAG_NAME,
                                   external_task_id='task3',
                                   allowed_states=[State.SUCCESS, State.FAILED, State.UPSTREAM_FAILED],
                                   mode='reschedule')
       last = DummyOperator(task_id='last')
   ​
       sensor >> last
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #10275: Trigger rule for "always"

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10275:
URL: https://github.com/apache/airflow/issues/10275#issuecomment-671701631


   I have similar scenario:
   create machine -> run steps -> terminate machine
   
   For termination I use trigger rule `all_done` and it works great.
   Whether upstream succeeded or failed the termination task is executed.
   Won't it solve your use case as well?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] darwinyip commented on issue #10275: Trigger rule for "always"

Posted by GitBox <gi...@apache.org>.
darwinyip commented on issue #10275:
URL: https://github.com/apache/airflow/issues/10275#issuecomment-672091385


   That's true. Closing issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] darwinyip closed issue #10275: Trigger rule for "always"

Posted by GitBox <gi...@apache.org>.
darwinyip closed issue #10275:
URL: https://github.com/apache/airflow/issues/10275


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org