You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ash Berlin-Taylor <as...@apache.org> on 2023/01/09 16:27:37 UTC
[VOTE] AIP-52 Automatic setup and teardown tasks
Hello everyone,
I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
There haven't been any notable changes to the original document, mostly just clairifications to the proposal.
This is my +1, and the vote will last until 10am (UTC) on Monday 16th January.
Thanks,
Ash
(To remind people what this is about, here is the example from the first dicuss email)
```
from airflow import DAG, task, setup, teardown
with DAG(dag_id='test'):
@setup
def create_cluster():
...
return cluster_id
@task
def load(ti):
# Example:
cluster_id = ti.xcom_pull(task_id="create_cluster")
def summarize():
...
@teardown(on_failure_fail_dagrun=False)
def teardown_cluster():
...
cluster_id = ti.xcom_pull(task_id="create_cluster")
create_cluster()
load() >> summarize()
teardown_cluster()
```
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Kaxil Naik <ka...@gmail.com>.
+1 binding
On Fri, 13 Jan 2023 at 05:01, Jed Cunningham <je...@apache.org>
wrote:
> +1 (binding)
>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Jed Cunningham <je...@apache.org>.
+1 (binding)
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Pankaj Koti <pa...@astronomer.io.INVALID>.
+1 (non-binding)
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time (IST)
Email: pankaj.koti@astronomer.io
Mobile: +91 9730079985
On Tue, Jan 10, 2023 at 5:22 AM Oliveira, Niko <on...@amazon.com.invalid>
wrote:
> +1 (binding)
>
>
> Cheers,
> Niko
> ------------------------------
> *From:* Kevin Yang <yr...@gmail.com>
> *Sent:* Monday, January 9, 2023 1:46:00 PM
> *To:* dev@airflow.apache.org
> *Subject:* RE: [EXTERNAL][VOTE] AIP-52 Automatic setup and teardown tasks
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
> +1 (binding)
>
> On Mon, Jan 9, 2023 at 1:41 PM Felix Uellendall <fe...@pm.me.invalid>
> wrote:
>
>> +1 (binding)
>>
>> -Felix
>>
>>
>> Sent from Proton Mail for iOS
>>
>>
>> On Mon, Jan 9, 2023 at 22:05, Pierre Jeambrun <pi...@gmail.com>
>> wrote:
>>
>> +1 (binding)
>>
>> Le lun. 9 janv. 2023 à 21:12, Vikram Koka <vi...@astronomer.io.invalid>
>> a écrit :
>>
>>> +1 binding
>>>
>>> Vikram
>>>
>>>
>>> On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang <pi...@umich.edu> wrote:
>>>
>>>> +1 binding
>>>>
>>>> Thanks,
>>>>
>>>> Ping
>>>>
>>>>
>>>> On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <
>>>> ephraimanierobi@gmail.com> wrote:
>>>>
>>>>> +1 binding
>>>>>
>>>>> On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell
>>>>>> <jo...@astronomer.io.invalid> wrote:
>>>>>>
>>>>>>> +1 binding
>>>>>>>
>>>>>>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl
>>>>>>> <dr...@astronomer.io.invalid> wrote:
>>>>>>>
>>>>>>>> +1 (non-binding)
>>>>>>>>
>>>>>>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>>>>>>
>>>>>>>> +1 (binding)
>>>>>>>>
>>>>>>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 (binding)
>>>>>>>>>
>>>>>>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>>>>>>>> <fe...@amazon.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> +1 non-binding
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>>>>>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>>>>>>>> *To:* dev@airflow.apache.org
>>>>>>>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown
>>>>>>>>>> tasks
>>>>>>>>>>
>>>>>>>>>> *CAUTION*: This email originated from outside of the
>>>>>>>>>> organization. Do not click links or open attachments unless you can confirm
>>>>>>>>>> the sender and know the content is safe.
>>>>>>>>>>
>>>>>>>>>> Hello everyone,
>>>>>>>>>>
>>>>>>>>>> I am calling for a vote on AIP-52
>>>>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>>>>>>> There haven't been any notable changes to the original document,
>>>>>>>>>> mostly just clairifications to the proposal.
>>>>>>>>>>
>>>>>>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday
>>>>>>>>>> 16th January.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ash
>>>>>>>>>> (To remind people what this is about, here is the example from
>>>>>>>>>> the first dicuss email)
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> from airflow import DAG, task, setup, teardown
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> with DAG(dag_id='test'):
>>>>>>>>>> @setup
>>>>>>>>>> def create_cluster():
>>>>>>>>>> ...
>>>>>>>>>> return cluster_id
>>>>>>>>>>
>>>>>>>>>> @task
>>>>>>>>>> def load(ti):
>>>>>>>>>> # Example:
>>>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>>>
>>>>>>>>>> def summarize():
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>>>>>>> def teardown_cluster():
>>>>>>>>>> ...
>>>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>>>
>>>>>>>>>> create_cluster()
>>>>>>>>>> load() >> summarize()
>>>>>>>>>> teardown_cluster()
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Charles Frank Cash
>>>>>> https://github.com/frankcash
>>>>>> https://keybase.io/frankcash
>>>>>> <https://github.com/frankcash>
>>>>>>
>>>>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
+1 (binding)
Cheers,
Niko
________________________________
From: Kevin Yang <yr...@gmail.com>
Sent: Monday, January 9, 2023 1:46:00 PM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL][VOTE] AIP-52 Automatic setup and teardown tasks
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
+1 (binding)
On Mon, Jan 9, 2023 at 1:41 PM Felix Uellendall <fe...@pm.me.invalid> wrote:
+1 (binding)
-Felix
Sent from Proton Mail for iOS
On Mon, Jan 9, 2023 at 22:05, Pierre Jeambrun <pi...@gmail.com>> wrote:
+1 (binding)
Le lun. 9 janv. 2023 à 21:12, Vikram Koka <vi...@astronomer.io.invalid> a écrit :
+1 binding
Vikram
On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang <pi...@umich.edu>> wrote:
+1 binding
Thanks,
Ping
On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <ep...@gmail.com>> wrote:
+1 binding
On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com>> wrote:
+1 (non-binding)
On Mon, Jan 9, 2023 at 1:34 PM Josh Fell <jo...@astronomer.io.invalid> wrote:
+1 binding
On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl <dr...@astronomer.io.invalid> wrote:
+1 (non-binding)
On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org>> wrote:
+1 (binding)
On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com>> wrote:
+1 (binding)
On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis <fe...@amazon.com.invalid> wrote:
+1 non-binding
________________________________
From: Ash Berlin-Taylor <as...@apache.org>>
Sent: Monday, January 9, 2023 8:27 AM
To: dev@airflow.apache.org<ma...@airflow.apache.org>
Subject: [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello everyone,
I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
There haven't been any notable changes to the original document, mostly just clairifications to the proposal.
This is my +1, and the vote will last until 10am (UTC) on Monday 16th January.
Thanks,
Ash
(To remind people what this is about, here is the example from the first dicuss email)
```
from airflow import DAG, task, setup, teardown
with DAG(dag_id='test'):
@setup
def create_cluster():
...
return cluster_id
@task
def load(ti):
# Example:
cluster_id = ti.xcom_pull(task_id="create_cluster")
def summarize():
...
@teardown(on_failure_fail_dagrun=False)
def teardown_cluster():
...
cluster_id = ti.xcom_pull(task_id="create_cluster")
create_cluster()
load() >> summarize()
teardown_cluster()
```
--
Charles Frank Cash
https://github.com/frankcash
https://keybase.io/frankcash
<https://github.com/frankcash>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Kevin Yang <yr...@gmail.com>.
+1 (binding)
On Mon, Jan 9, 2023 at 1:41 PM Felix Uellendall <fe...@pm.me.invalid>
wrote:
> +1 (binding)
>
> -Felix
>
>
> Sent from Proton Mail for iOS
>
>
> On Mon, Jan 9, 2023 at 22:05, Pierre Jeambrun <pi...@gmail.com>
> wrote:
>
> +1 (binding)
>
> Le lun. 9 janv. 2023 à 21:12, Vikram Koka <vi...@astronomer.io.invalid>
> a écrit :
>
>> +1 binding
>>
>> Vikram
>>
>>
>> On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang <pi...@umich.edu> wrote:
>>
>>> +1 binding
>>>
>>> Thanks,
>>>
>>> Ping
>>>
>>>
>>> On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <
>>> ephraimanierobi@gmail.com> wrote:
>>>
>>>> +1 binding
>>>>
>>>> On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell
>>>>> <jo...@astronomer.io.invalid> wrote:
>>>>>
>>>>>> +1 binding
>>>>>>
>>>>>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl
>>>>>> <dr...@astronomer.io.invalid> wrote:
>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 (binding)
>>>>>>>>
>>>>>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>>>>>>> <fe...@amazon.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> +1 non-binding
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>>>>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>>>>>>> *To:* dev@airflow.apache.org
>>>>>>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown
>>>>>>>>> tasks
>>>>>>>>>
>>>>>>>>> *CAUTION*: This email originated from outside of the
>>>>>>>>> organization. Do not click links or open attachments unless you can confirm
>>>>>>>>> the sender and know the content is safe.
>>>>>>>>>
>>>>>>>>> Hello everyone,
>>>>>>>>>
>>>>>>>>> I am calling for a vote on AIP-52
>>>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>>>>>> There haven't been any notable changes to the original document,
>>>>>>>>> mostly just clairifications to the proposal.
>>>>>>>>>
>>>>>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday
>>>>>>>>> 16th January.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ash
>>>>>>>>> (To remind people what this is about, here is the example from the
>>>>>>>>> first dicuss email)
>>>>>>>>>
>>>>>>>>> ```
>>>>>>>>> from airflow import DAG, task, setup, teardown
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> with DAG(dag_id='test'):
>>>>>>>>> @setup
>>>>>>>>> def create_cluster():
>>>>>>>>> ...
>>>>>>>>> return cluster_id
>>>>>>>>>
>>>>>>>>> @task
>>>>>>>>> def load(ti):
>>>>>>>>> # Example:
>>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>>
>>>>>>>>> def summarize():
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>>>>>> def teardown_cluster():
>>>>>>>>> ...
>>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>>
>>>>>>>>> create_cluster()
>>>>>>>>> load() >> summarize()
>>>>>>>>> teardown_cluster()
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Charles Frank Cash
>>>>> https://github.com/frankcash
>>>>> https://keybase.io/frankcash
>>>>> <https://github.com/frankcash>
>>>>>
>>>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Felix Uellendall <fe...@pm.me.INVALID>.
+1 (binding)
-Felix
Sent from Proton Mail for iOS
On Mon, Jan 9, 2023 at 22:05, Pierre Jeambrun <pi...@gmail.com> wrote:
> +1 (binding)
>
> Le lun. 9 janv. 2023 à 21:12, Vikram Koka <vi...@astronomer.io.invalid> a écrit :
>
>> +1 binding
>>
>> Vikram
>>
>> On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang <pi...@umich.edu> wrote:
>>
>>> +1 binding
>>>
>>> Thanks,
>>>
>>> Ping
>>>
>>> On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <ep...@gmail.com> wrote:
>>>
>>>> +1 binding
>>>>
>>>> On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell <jo...@astronomer.io.invalid> wrote:
>>>>>
>>>>>> +1 binding
>>>>>>
>>>>>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl <dr...@astronomer.io.invalid> wrote:
>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>>>>>>
>>>>>>>> +1 (binding)
>>>>>>>>
>>>>>>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>>
>>>>>>>>> +1 (binding)
>>>>>>>>>
>>>>>>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis <fe...@amazon.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> +1 non-binding
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> From: Ash Berlin-Taylor <as...@apache.org>
>>>>>>>>>> Sent: Monday, January 9, 2023 8:27 AM
>>>>>>>>>> To: dev@airflow.apache.org
>>>>>>>>>> Subject: [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>>>>>>>>>>
>>>>>>>>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>>>>>>>>
>>>>>>>>>> Hello everyone,
>>>>>>>>>>
>>>>>>>>>> I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>>>>>>> There haven't been any notable changes to the original document, mostly just clairifications to the proposal.
>>>>>>>>>>
>>>>>>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th January.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ash
>>>>>>>>>> (To remind people what this is about, here is the example from the first dicuss email)
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> from airflow import DAG, task, setup, teardown
>>>>>>>>>>
>>>>>>>>>> with DAG(dag_id='test'):
>>>>>>>>>> @setup
>>>>>>>>>> def create_cluster():
>>>>>>>>>> ...
>>>>>>>>>> return cluster_id
>>>>>>>>>>
>>>>>>>>>> @task
>>>>>>>>>> def load(ti):
>>>>>>>>>> # Example:
>>>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>>>
>>>>>>>>>> def summarize():
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>>>>>>> def teardown_cluster():
>>>>>>>>>> ...
>>>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>>>
>>>>>>>>>> create_cluster()
>>>>>>>>>> load() >> summarize()
>>>>>>>>>> teardown_cluster()
>>>>>>>>>> ```
>>>>>
>>>>> --
>>>>>
>>>>> Charles Frank Cash
>>>>> https://github.com/frankcash[https://keybase.io/frankcash](https://github.com/frankcash)
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Pierre Jeambrun <pi...@gmail.com>.
+1 (binding)
Le lun. 9 janv. 2023 à 21:12, Vikram Koka <vi...@astronomer.io.invalid> a
écrit :
> +1 binding
>
> Vikram
>
>
> On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang <pi...@umich.edu> wrote:
>
>> +1 binding
>>
>> Thanks,
>>
>> Ping
>>
>>
>> On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <
>> ephraimanierobi@gmail.com> wrote:
>>
>>> +1 binding
>>>
>>> On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell
>>>> <jo...@astronomer.io.invalid> wrote:
>>>>
>>>>> +1 binding
>>>>>
>>>>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl
>>>>> <dr...@astronomer.io.invalid> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>>>>>> <fe...@amazon.com.invalid> wrote:
>>>>>>>
>>>>>>>> +1 non-binding
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>>>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>>>>>> *To:* dev@airflow.apache.org
>>>>>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown
>>>>>>>> tasks
>>>>>>>>
>>>>>>>>
>>>>>>>> *CAUTION*: This email originated from outside of the organization.
>>>>>>>> Do not click links or open attachments unless you can confirm the sender
>>>>>>>> and know the content is safe.
>>>>>>>>
>>>>>>>> Hello everyone,
>>>>>>>>
>>>>>>>> I am calling for a vote on AIP-52
>>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>>>>> There haven't been any notable changes to the original document,
>>>>>>>> mostly just clairifications to the proposal.
>>>>>>>>
>>>>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday
>>>>>>>> 16th January.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ash
>>>>>>>> (To remind people what this is about, here is the example from the
>>>>>>>> first dicuss email)
>>>>>>>>
>>>>>>>> ```
>>>>>>>> from airflow import DAG, task, setup, teardown
>>>>>>>>
>>>>>>>>
>>>>>>>> with DAG(dag_id='test'):
>>>>>>>> @setup
>>>>>>>> def create_cluster():
>>>>>>>> ...
>>>>>>>> return cluster_id
>>>>>>>>
>>>>>>>> @task
>>>>>>>> def load(ti):
>>>>>>>> # Example:
>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>
>>>>>>>> def summarize():
>>>>>>>> ...
>>>>>>>>
>>>>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>>>>> def teardown_cluster():
>>>>>>>> ...
>>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>>
>>>>>>>> create_cluster()
>>>>>>>> load() >> summarize()
>>>>>>>> teardown_cluster()
>>>>>>>> ```
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Charles Frank Cash
>>>> https://github.com/frankcash
>>>> https://keybase.io/frankcash
>>>> <https://github.com/frankcash>
>>>>
>>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
+1 binding
Vikram
On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang <pi...@umich.edu> wrote:
> +1 binding
>
> Thanks,
>
> Ping
>
>
> On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <
> ephraimanierobi@gmail.com> wrote:
>
>> +1 binding
>>
>> On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell
>>> <jo...@astronomer.io.invalid> wrote:
>>>
>>>> +1 binding
>>>>
>>>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl
>>>> <dr...@astronomer.io.invalid> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>>>
>>>>> +1 (binding)
>>>>>
>>>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>>>>> <fe...@amazon.com.invalid> wrote:
>>>>>>
>>>>>>> +1 non-binding
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>>>>> *To:* dev@airflow.apache.org
>>>>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown
>>>>>>> tasks
>>>>>>>
>>>>>>>
>>>>>>> *CAUTION*: This email originated from outside of the organization.
>>>>>>> Do not click links or open attachments unless you can confirm the sender
>>>>>>> and know the content is safe.
>>>>>>>
>>>>>>> Hello everyone,
>>>>>>>
>>>>>>> I am calling for a vote on AIP-52
>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>>>> There haven't been any notable changes to the original document,
>>>>>>> mostly just clairifications to the proposal.
>>>>>>>
>>>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday
>>>>>>> 16th January.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ash
>>>>>>> (To remind people what this is about, here is the example from the
>>>>>>> first dicuss email)
>>>>>>>
>>>>>>> ```
>>>>>>> from airflow import DAG, task, setup, teardown
>>>>>>>
>>>>>>>
>>>>>>> with DAG(dag_id='test'):
>>>>>>> @setup
>>>>>>> def create_cluster():
>>>>>>> ...
>>>>>>> return cluster_id
>>>>>>>
>>>>>>> @task
>>>>>>> def load(ti):
>>>>>>> # Example:
>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>
>>>>>>> def summarize():
>>>>>>> ...
>>>>>>>
>>>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>>>> def teardown_cluster():
>>>>>>> ...
>>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>>
>>>>>>> create_cluster()
>>>>>>> load() >> summarize()
>>>>>>> teardown_cluster()
>>>>>>> ```
>>>>>>>
>>>>>>>
>>>>>
>>>
>>> --
>>> Charles Frank Cash
>>> https://github.com/frankcash
>>> https://keybase.io/frankcash
>>> <https://github.com/frankcash>
>>>
>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Ping Zhang <pi...@umich.edu>.
+1 binding
Thanks,
Ping
On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi <ep...@gmail.com>
wrote:
> +1 binding
>
> On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell
>> <jo...@astronomer.io.invalid> wrote:
>>
>>> +1 binding
>>>
>>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl
>>> <dr...@astronomer.io.invalid> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>>
>>>> +1 (binding)
>>>>
>>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> +1 (binding)
>>>>>
>>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>>>> <fe...@amazon.com.invalid> wrote:
>>>>>
>>>>>> +1 non-binding
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>>>> *To:* dev@airflow.apache.org
>>>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown
>>>>>> tasks
>>>>>>
>>>>>>
>>>>>> *CAUTION*: This email originated from outside of the organization.
>>>>>> Do not click links or open attachments unless you can confirm the sender
>>>>>> and know the content is safe.
>>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> I am calling for a vote on AIP-52
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>>> There haven't been any notable changes to the original document,
>>>>>> mostly just clairifications to the proposal.
>>>>>>
>>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>>>>>> January.
>>>>>>
>>>>>> Thanks,
>>>>>> Ash
>>>>>> (To remind people what this is about, here is the example from the
>>>>>> first dicuss email)
>>>>>>
>>>>>> ```
>>>>>> from airflow import DAG, task, setup, teardown
>>>>>>
>>>>>>
>>>>>> with DAG(dag_id='test'):
>>>>>> @setup
>>>>>> def create_cluster():
>>>>>> ...
>>>>>> return cluster_id
>>>>>>
>>>>>> @task
>>>>>> def load(ti):
>>>>>> # Example:
>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>
>>>>>> def summarize():
>>>>>> ...
>>>>>>
>>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>>> def teardown_cluster():
>>>>>> ...
>>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>>
>>>>>> create_cluster()
>>>>>> load() >> summarize()
>>>>>> teardown_cluster()
>>>>>> ```
>>>>>>
>>>>>>
>>>>
>>
>> --
>> Charles Frank Cash
>> https://github.com/frankcash
>> https://keybase.io/frankcash
>> <https://github.com/frankcash>
>>
>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Ephraim Anierobi <ep...@gmail.com>.
+1 binding
On Mon, 9 Jan 2023 at 19:55, Frank Cash <ca...@gmail.com> wrote:
> +1 (non-binding)
>
> On Mon, Jan 9, 2023 at 1:34 PM Josh Fell <jo...@astronomer.io.invalid>
> wrote:
>
>> +1 binding
>>
>> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl <dr...@astronomer.io.invalid>
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>>
>>> +1 (binding)
>>>
>>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>>> <fe...@amazon.com.invalid> wrote:
>>>>
>>>>> +1 non-binding
>>>>>
>>>>> ------------------------------
>>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>>> *To:* dev@airflow.apache.org
>>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>>>>>
>>>>>
>>>>> *CAUTION*: This email originated from outside of the organization. Do
>>>>> not click links or open attachments unless you can confirm the sender and
>>>>> know the content is safe.
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> I am calling for a vote on AIP-52
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>>> There haven't been any notable changes to the original document,
>>>>> mostly just clairifications to the proposal.
>>>>>
>>>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>>>>> January.
>>>>>
>>>>> Thanks,
>>>>> Ash
>>>>> (To remind people what this is about, here is the example from the
>>>>> first dicuss email)
>>>>>
>>>>> ```
>>>>> from airflow import DAG, task, setup, teardown
>>>>>
>>>>>
>>>>> with DAG(dag_id='test'):
>>>>> @setup
>>>>> def create_cluster():
>>>>> ...
>>>>> return cluster_id
>>>>>
>>>>> @task
>>>>> def load(ti):
>>>>> # Example:
>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>
>>>>> def summarize():
>>>>> ...
>>>>>
>>>>> @teardown(on_failure_fail_dagrun=False)
>>>>> def teardown_cluster():
>>>>> ...
>>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>>
>>>>> create_cluster()
>>>>> load() >> summarize()
>>>>> teardown_cluster()
>>>>> ```
>>>>>
>>>>>
>>>
>
> --
> Charles Frank Cash
> https://github.com/frankcash
> https://keybase.io/frankcash
> <https://github.com/frankcash>
>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Frank Cash <ca...@gmail.com>.
+1 (non-binding)
On Mon, Jan 9, 2023 at 1:34 PM Josh Fell <jo...@astronomer.io.invalid>
wrote:
> +1 binding
>
> On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl <dr...@astronomer.io.invalid>
> wrote:
>
>> +1 (non-binding)
>>
>> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>>
>> +1 (binding)
>>
>> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> +1 (binding)
>>>
>>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>>> <fe...@amazon.com.invalid> wrote:
>>>
>>>> +1 non-binding
>>>>
>>>> ------------------------------
>>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>>> *To:* dev@airflow.apache.org
>>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>>>>
>>>>
>>>> *CAUTION*: This email originated from outside of the organization. Do
>>>> not click links or open attachments unless you can confirm the sender and
>>>> know the content is safe.
>>>>
>>>> Hello everyone,
>>>>
>>>> I am calling for a vote on AIP-52
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>> There haven't been any notable changes to the original document, mostly
>>>> just clairifications to the proposal.
>>>>
>>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>>>> January.
>>>>
>>>> Thanks,
>>>> Ash
>>>> (To remind people what this is about, here is the example from the
>>>> first dicuss email)
>>>>
>>>> ```
>>>> from airflow import DAG, task, setup, teardown
>>>>
>>>>
>>>> with DAG(dag_id='test'):
>>>> @setup
>>>> def create_cluster():
>>>> ...
>>>> return cluster_id
>>>>
>>>> @task
>>>> def load(ti):
>>>> # Example:
>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>
>>>> def summarize():
>>>> ...
>>>>
>>>> @teardown(on_failure_fail_dagrun=False)
>>>> def teardown_cluster():
>>>> ...
>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>
>>>> create_cluster()
>>>> load() >> summarize()
>>>> teardown_cluster()
>>>> ```
>>>>
>>>>
>>
--
Charles Frank Cash
https://github.com/frankcash
https://keybase.io/frankcash
<https://github.com/frankcash>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Josh Fell <jo...@astronomer.io.INVALID>.
+1 binding
On Mon, Jan 9, 2023 at 12:51 PM Drew Hubl <dr...@astronomer.io.invalid>
wrote:
> +1 (non-binding)
>
> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>
> +1 (binding)
>
> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> +1 (binding)
>>
>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
>> <fe...@amazon.com.invalid> wrote:
>>
>>> +1 non-binding
>>>
>>> ------------------------------
>>> *From:* Ash Berlin-Taylor <as...@apache.org>
>>> *Sent:* Monday, January 9, 2023 8:27 AM
>>> *To:* dev@airflow.apache.org
>>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>>>
>>>
>>> *CAUTION*: This email originated from outside of the organization. Do
>>> not click links or open attachments unless you can confirm the sender and
>>> know the content is safe.
>>>
>>> Hello everyone,
>>>
>>> I am calling for a vote on AIP-52
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>> There haven't been any notable changes to the original document, mostly
>>> just clairifications to the proposal.
>>>
>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>>> January.
>>>
>>> Thanks,
>>> Ash
>>> (To remind people what this is about, here is the example from the first
>>> dicuss email)
>>>
>>> ```
>>> from airflow import DAG, task, setup, teardown
>>>
>>>
>>> with DAG(dag_id='test'):
>>> @setup
>>> def create_cluster():
>>> ...
>>> return cluster_id
>>>
>>> @task
>>> def load(ti):
>>> # Example:
>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>
>>> def summarize():
>>> ...
>>>
>>> @teardown(on_failure_fail_dagrun=False)
>>> def teardown_cluster():
>>> ...
>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>
>>> create_cluster()
>>> load() >> summarize()
>>> teardown_cluster()
>>> ```
>>>
>>>
>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Drew Hubl <dr...@astronomer.io.INVALID>.
+1 (non-binding)
> On Jan 9, 2023, at 10:10 AM, Elad Kalif <el...@apache.org> wrote:
>
> +1 (binding)
>
> On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
>> +1 (binding)
>>
>> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis <fe...@amazon.com.invalid> wrote:
>>> +1 non-binding
>>>
>>>
>>> From: Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>>
>>> Sent: Monday, January 9, 2023 8:27 AM
>>> To: dev@airflow.apache.org <ma...@airflow.apache.org>
>>> Subject: [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>
>>>
>>> Hello everyone,
>>>
>>> I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>> There haven't been any notable changes to the original document, mostly just clairifications to the proposal.
>>>
>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th January.
>>>
>>> Thanks,
>>> Ash
>>> (To remind people what this is about, here is the example from the first dicuss email)
>>>
>>> ```
>>> from airflow import DAG, task, setup, teardown
>>>
>>>
>>> with DAG(dag_id='test'):
>>> @setup
>>> def create_cluster():
>>> ...
>>> return cluster_id
>>>
>>> @task
>>> def load(ti):
>>> # Example:
>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>
>>> def summarize():
>>> ...
>>>
>>> @teardown(on_failure_fail_dagrun=False)
>>> def teardown_cluster():
>>> ...
>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>
>>> create_cluster()
>>> load() >> summarize()
>>> teardown_cluster()
>>> ```
>>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Elad Kalif <el...@apache.org>.
+1 (binding)
On Mon, Jan 9, 2023 at 7:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> +1 (binding)
>
> On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
>
>> +1 non-binding
>>
>> ------------------------------
>> *From:* Ash Berlin-Taylor <as...@apache.org>
>> *Sent:* Monday, January 9, 2023 8:27 AM
>> *To:* dev@airflow.apache.org
>> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>>
>>
>> *CAUTION*: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>> Hello everyone,
>>
>> I am calling for a vote on AIP-52
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>> There haven't been any notable changes to the original document, mostly
>> just clairifications to the proposal.
>>
>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>> January.
>>
>> Thanks,
>> Ash
>> (To remind people what this is about, here is the example from the first
>> dicuss email)
>>
>> ```
>> from airflow import DAG, task, setup, teardown
>>
>>
>> with DAG(dag_id='test'):
>> @setup
>> def create_cluster():
>> ...
>> return cluster_id
>>
>> @task
>> def load(ti):
>> # Example:
>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>
>> def summarize():
>> ...
>>
>> @teardown(on_failure_fail_dagrun=False)
>> def teardown_cluster():
>> ...
>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>
>> create_cluster()
>> load() >> summarize()
>> teardown_cluster()
>> ```
>>
>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 (binding)
On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis <fe...@amazon.com.invalid>
wrote:
> +1 non-binding
>
> ------------------------------
> *From:* Ash Berlin-Taylor <as...@apache.org>
> *Sent:* Monday, January 9, 2023 8:27 AM
> *To:* dev@airflow.apache.org
> *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
> Hello everyone,
>
> I am calling for a vote on AIP-52
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
> There haven't been any notable changes to the original document, mostly
> just clairifications to the proposal.
>
> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
> January.
>
> Thanks,
> Ash
> (To remind people what this is about, here is the example from the first
> dicuss email)
>
> ```
> from airflow import DAG, task, setup, teardown
>
>
> with DAG(dag_id='test'):
> @setup
> def create_cluster():
> ...
> return cluster_id
>
> @task
> def load(ti):
> # Example:
> cluster_id = ti.xcom_pull(task_id="create_cluster")
>
> def summarize():
> ...
>
> @teardown(on_failure_fail_dagrun=False)
> def teardown_cluster():
> ...
> cluster_id = ti.xcom_pull(task_id="create_cluster")
>
> create_cluster()
> load() >> summarize()
> teardown_cluster()
> ```
>
>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
+1 non-binding
________________________________
From: Ash Berlin-Taylor <as...@apache.org>
Sent: Monday, January 9, 2023 8:27 AM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello everyone,
I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
There haven't been any notable changes to the original document, mostly just clairifications to the proposal.
This is my +1, and the vote will last until 10am (UTC) on Monday 16th January.
Thanks,
Ash
(To remind people what this is about, here is the example from the first dicuss email)
```
from airflow import DAG, task, setup, teardown
with DAG(dag_id='test'):
@setup
def create_cluster():
...
return cluster_id
@task
def load(ti):
# Example:
cluster_id = ti.xcom_pull(task_id="create_cluster")
def summarize():
...
@teardown(on_failure_fail_dagrun=False)
def teardown_cluster():
...
cluster_id = ti.xcom_pull(task_id="create_cluster")
create_cluster()
load() >> summarize()
teardown_cluster()
```
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Dennis Akpenyi <de...@gmail.com>.
+1 non-binding
On Sun, Jan 15, 2023 at 9:28 PM Robert Karish <ro...@gmail.com>
wrote:
> +1 (non-binding)
>
> Would be very convenient not having to clear out create_cluster and
> terminate_cluster tasks when needing to re-run one task that does a
> spark-submit.
>
> On Sun, Jan 15, 2023 at 3:37 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> The key thing that isn't possible is the automatic clearing.
>>
>> If I notice a problem in the data after the cluster has been torn down
>> and I want to the rerun a single task then you have to manually clear the
>> create cluster task and the task you want to run AND the teardown task.
>> (This is not a hypothetical but a real issue I have had)
>>
>> More generally this is part of the trend of giving DAG authors the
>> ability to specify "directly" the constructs and flows that they currently
>> have to hack around.
>>
>>
>> -a
>>
>> On 15 January 2023 00:26:31 GMT, Alexander Shorin <kx...@gmail.com>
>> wrote:
>>>
>>> Hi!
>>>
>>> This looks like sugar without any practical difference from explicitly
>>> defined setup/teardown operators in dag. And the example doesn't add any
>>> automatic'sation except to set you free from defining explicit relations
>>> between setup, teardown and regular operators, but why all of this when you
>>> already could do this explicitly and reliable? Looks like a feature just
>>> for a feature.
>>>
>>> If that example of implementation could hide within a context details
>>> about ti.xcom_pull(task_id="create_cluster") and hide the need of
>>> explicit call setup/teardown operators - that would be a bit helpful, but
>>> there is no such a thing.
>>>
>>> --
>>> ,,,^..^,,,
>>>
>>>
>>> On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I am calling for a vote on AIP-52
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>>> There haven't been any notable changes to the original document, mostly
>>>> just clairifications to the proposal.
>>>>
>>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>>>> January.
>>>>
>>>> Thanks,
>>>> Ash
>>>> (To remind people what this is about, here is the example from the
>>>> first dicuss email)
>>>>
>>>> ```
>>>> from airflow import DAG, task, setup, teardown
>>>>
>>>>
>>>> with DAG(dag_id='test'):
>>>> @setup
>>>> def create_cluster():
>>>> ...
>>>> return cluster_id
>>>>
>>>> @task
>>>> def load(ti):
>>>> # Example:
>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>
>>>> def summarize():
>>>> ...
>>>>
>>>> @teardown(on_failure_fail_dagrun=False)
>>>> def teardown_cluster():
>>>> ...
>>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>>
>>>> create_cluster()
>>>> load() >> summarize()
>>>> teardown_cluster()
>>>> ```
>>>>
>>>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Robert Karish <ro...@gmail.com>.
+1 (non-binding)
Would be very convenient not having to clear out create_cluster and
terminate_cluster tasks when needing to re-run one task that does a
spark-submit.
On Sun, Jan 15, 2023 at 3:37 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> The key thing that isn't possible is the automatic clearing.
>
> If I notice a problem in the data after the cluster has been torn down and
> I want to the rerun a single task then you have to manually clear the
> create cluster task and the task you want to run AND the teardown task.
> (This is not a hypothetical but a real issue I have had)
>
> More generally this is part of the trend of giving DAG authors the ability
> to specify "directly" the constructs and flows that they currently have to
> hack around.
>
>
> -a
>
> On 15 January 2023 00:26:31 GMT, Alexander Shorin <kx...@gmail.com>
> wrote:
>>
>> Hi!
>>
>> This looks like sugar without any practical difference from explicitly
>> defined setup/teardown operators in dag. And the example doesn't add any
>> automatic'sation except to set you free from defining explicit relations
>> between setup, teardown and regular operators, but why all of this when you
>> already could do this explicitly and reliable? Looks like a feature just
>> for a feature.
>>
>> If that example of implementation could hide within a context details
>> about ti.xcom_pull(task_id="create_cluster") and hide the need of
>> explicit call setup/teardown operators - that would be a bit helpful, but
>> there is no such a thing.
>>
>> --
>> ,,,^..^,,,
>>
>>
>> On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>>> Hello everyone,
>>>
>>> I am calling for a vote on AIP-52
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>>> There haven't been any notable changes to the original document, mostly
>>> just clairifications to the proposal.
>>>
>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>>> January.
>>>
>>> Thanks,
>>> Ash
>>> (To remind people what this is about, here is the example from the first
>>> dicuss email)
>>>
>>> ```
>>> from airflow import DAG, task, setup, teardown
>>>
>>>
>>> with DAG(dag_id='test'):
>>> @setup
>>> def create_cluster():
>>> ...
>>> return cluster_id
>>>
>>> @task
>>> def load(ti):
>>> # Example:
>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>
>>> def summarize():
>>> ...
>>>
>>> @teardown(on_failure_fail_dagrun=False)
>>> def teardown_cluster():
>>> ...
>>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>>
>>> create_cluster()
>>> load() >> summarize()
>>> teardown_cluster()
>>> ```
>>>
>>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Ash Berlin-Taylor <as...@apache.org>.
The key thing that isn't possible is the automatic clearing.
If I notice a problem in the data after the cluster has been torn down and I want to the rerun a single task then you have to manually clear the create cluster task and the task you want to run AND the teardown task. (This is not a hypothetical but a real issue I have had)
More generally this is part of the trend of giving DAG authors the ability to specify "directly" the constructs and flows that they currently have to hack around.
-a
On 15 January 2023 00:26:31 GMT, Alexander Shorin <kx...@gmail.com> wrote:
>Hi!
>
>This looks like sugar without any practical difference from explicitly
>defined setup/teardown operators in dag. And the example doesn't add any
>automatic'sation except to set you free from defining explicit relations
>between setup, teardown and regular operators, but why all of this when you
>already could do this explicitly and reliable? Looks like a feature just
>for a feature.
>
>If that example of implementation could hide within a context details about
>ti.xcom_pull(task_id="create_cluster") and hide the need of explicit call
>setup/teardown operators - that would be a bit helpful, but there is no
>such a thing.
>
>--
>,,,^..^,,,
>
>
>On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Hello everyone,
>>
>> I am calling for a vote on AIP-52
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
>> There haven't been any notable changes to the original document, mostly
>> just clairifications to the proposal.
>>
>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
>> January.
>>
>> Thanks,
>> Ash
>> (To remind people what this is about, here is the example from the first
>> dicuss email)
>>
>> ```
>> from airflow import DAG, task, setup, teardown
>>
>>
>> with DAG(dag_id='test'):
>> @setup
>> def create_cluster():
>> ...
>> return cluster_id
>>
>> @task
>> def load(ti):
>> # Example:
>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>
>> def summarize():
>> ...
>>
>> @teardown(on_failure_fail_dagrun=False)
>> def teardown_cluster():
>> ...
>> cluster_id = ti.xcom_pull(task_id="create_cluster")
>>
>> create_cluster()
>> load() >> summarize()
>> teardown_cluster()
>> ```
>>
>>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by Alexander Shorin <kx...@gmail.com>.
Hi!
This looks like sugar without any practical difference from explicitly
defined setup/teardown operators in dag. And the example doesn't add any
automatic'sation except to set you free from defining explicit relations
between setup, teardown and regular operators, but why all of this when you
already could do this explicitly and reliable? Looks like a feature just
for a feature.
If that example of implementation could hide within a context details about
ti.xcom_pull(task_id="create_cluster") and hide the need of explicit call
setup/teardown operators - that would be a bit helpful, but there is no
such a thing.
--
,,,^..^,,,
On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> Hello everyone,
>
> I am calling for a vote on AIP-52
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
> There haven't been any notable changes to the original document, mostly
> just clairifications to the proposal.
>
> This is my +1, and the vote will last until 10am (UTC) on Monday 16th
> January.
>
> Thanks,
> Ash
> (To remind people what this is about, here is the example from the first
> dicuss email)
>
> ```
> from airflow import DAG, task, setup, teardown
>
>
> with DAG(dag_id='test'):
> @setup
> def create_cluster():
> ...
> return cluster_id
>
> @task
> def load(ti):
> # Example:
> cluster_id = ti.xcom_pull(task_id="create_cluster")
>
> def summarize():
> ...
>
> @teardown(on_failure_fail_dagrun=False)
> def teardown_cluster():
> ...
> cluster_id = ti.xcom_pull(task_id="create_cluster")
>
> create_cluster()
> load() >> summarize()
> teardown_cluster()
> ```
>
>
Re: [VOTE] AIP-52 Automatic setup and teardown tasks
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Would love to see this one going through !
+1 (non binding)
From: Ash Berlin-Taylor <as...@apache.org>
Reply-To: "dev@airflow.apache.org" <de...@airflow.apache.org>
Date: Monday, January 9, 2023 at 11:27 AM
To: "dev@airflow.apache.org" <de...@airflow.apache.org>
Subject: [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello everyone,
I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks
There haven't been any notable changes to the original document, mostly just clairifications to the proposal.
This is my +1, and the vote will last until 10am (UTC) on Monday 16th January.
Thanks,
Ash
(To remind people what this is about, here is the example from the first dicuss email)
```
from airflow import DAG, task, setup, teardown
with DAG(dag_id='test'):
@setup
def create_cluster():
...
return cluster_id
@task
def load(ti):
# Example:
cluster_id = ti.xcom_pull(task_id="create_cluster")
def summarize():
...
@teardown(on_failure_fail_dagrun=False)
def teardown_cluster():
...
cluster_id = ti.xcom_pull(task_id="create_cluster")
create_cluster()
load() >> summarize()
teardown_cluster()
```