Re: [DISCUSS] AIP-52 Automatic setup and tear down tasks

["Ferruzzi, Dennis" <fe...@amazon.com.INVALID>]

I love the idea.  You've clearly put a lot of thought into this so maybe you've already considered and discarded these ideas, but I have two suggestions, for what they are worth.  One improvement would be to automagically append/prepend them to the task list.  If a task is tagged @setup, then it should be prepended as the first task in the task list.  The other suggestion would be that any task decorated with @setup or @teardown (or whatever you decide to name them) would get added to a new setup or teardown taskgroup.  The benefit there is that when you get tot he future work of multiple steps, it would make the GUI DAG view cleaner by collapsing all of the setup and teardown tasks by default if there are multiple tasks in either category.



________________________________
From: Ash Berlin-Taylor <as...@apache.org>
Sent: Friday, December 16, 2022 8:04 AM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [DISCUSS] AIP-52 Automatic setup and tear down tasks

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hi everyone,

I'd like to start a discussion about a new feature we'd like to add to
Airflow we call "setup and tear down tasks"

https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks

Lets start with a code example, that if you are familiar with
unittest.TestCase I hope will be familiar and you can guess what the
effect would be:

```
from airflow import DAG, task, setup, teardown


with DAG(dag_id='test'):
    @setup
    def create_cluster():
        ...
        return cluster_id

    @task
    def load(ti):
        # Example:
        cluster_id = ti.xcom_pull(task_id="create_cluster")

    def summarize():
        ...

    @teardown(on_failure_fail_dagrun=False)
    def teardown_cluster():
        ...
       cluster_id = ti.xcom_pull(task_id="create_cluster")

    create_cluster()
    load() >> summarize()
    teardown_cluster()
```

(This has been an itch of mine that I've had since I first started using
Airflow back in 2017!)

We go in to quite a bit of detail about the semantics and behaviours of
these new task types, but the tl;dr (copied from the AIP doc)

Add a new syntax for marking tasks as setup/teardown that:

 * Teardown tasks will still run even if the upstream tasks have
   otherwise failed
 * Teardown tasks failing don't always result in the DagRun being marked
   as failed (up to DAG author to choose failure mode)
 * Automatically clear setup/teardown tasks when clearing a dependent
   task

I'm not sure when we'll start a vote on this due to the holiday season,
but we'd like to start working on this in January.

-ash