You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Ace Haidrey <ah...@pinterest.com.INVALID> on 2020/10/16 16:21:57 UTC

Airflow Feature Design Feedback

Hi team,

My name is Ace, I’m a data engineer at Pinterest, and I wanted to get some feedback from the community on how they’d propose designing a feature that has been asked for us multiple times, that we’d like to implement in-house.

The ask is the following:
- Users want to be able to add an optional reason/description when they click on the Mark Success/Failed for both task level in graph views or dag level in tree views. This is to clearly state why the actions were taken on higher priority flows for when others are stepping in to look in it.

For us the feature makes sense to have, and I was wondering if this is something the greater group would like to have upstream as well. And if so then we wanted to know what was preferred.

1. We can add this information in the UI for the confirmation view and an optional textbook to add details similar to how users can add conf variables when triggering a dag option is selected in the UI.

Then after this, storing this information is the question. Do we want to store this information in the action logging extra blob and then add a view in the dag view where it has actions just pertaining to this dag and its tasks. (We plan to add this personally anyways for ourselves as going to the big list of actions for all dags and searching is not convenient always).

Or do we want to add a new column in the task Instance model and dag run model to store the descriptions and retrieve the information from there.

Tradeoffs here include the addition of a schema change, where the new column would generally be sparse, and the data is more resultant from an action vs necessarily being stored as part of the task run/dag run.

Please let me know on any feedback, concerns, ideas, and more!

Cheers,
Ace

Re: Airflow Feature Design Feedback

Posted by Tomasz Urbaszek <tu...@apache.org>.

Hi Ace,

Any update on this one? I think there's a Github issue proposing similar functionality:
https://github.com/apache/airflow/issues/12199

It would be good to coordinate the effort.

Best,
Tomek

On 2020/10/18 13:09:01, Kaxil Naik <ka...@gmail.com> wrote: 
> The Log table can be used at Dag Level too if we pass task&instance=None
> 
> For example: We log all the CLI actions in the Log table where we pass
> task_instance as None. Code:
> https://github.com/apache/airflow/blob/4e32546faf227a6497ce8b282fff7450cae6f665/airflow/utils/cli.py#L133
> 
> - Kaxil
> 
> 
> On Sun, Oct 18, 2020, 13:49 Deng Xiaodong <xd...@gmail.com> wrote:
> 
> > I have the same concern as Yu Qian does, and a bit more elaborations
> > follow below:
> >
> > - "log" table
> > <https://github.com/apache/airflow/blob/master/airflow/models/log.py> has
> > fields: *id*, *dttm*, *dag_id*, *task_id*, *event*, *execution_date*,
> > *owner*, *extra*. Seems to me only "*event*" or "*extra*" can provide
> > some room for this use case. But does it support more complicated cases
> > well? For example, for the same task instance, there may be back and forth
> > tries, and I may mark it Success/Failure for multiple times back and forth,
> > with multiple comments.
> >
> > - As demonstrated by the field list of the "log" table, it's at "task"
> > level (it has field "taks_id"). Ace has mentioned that "*Users want to be
> > able to add an optional reason/description when they click on the Mark
> > Success/Failed for both task level in graph views or dag level in tree
> > views*". So my question would be how the "dag level" should be supported
> > if we use the "log" table to implement this feature?
> >
> >
> > XD
> >
> > On Sun, Oct 18, 2020 at 11:13 AM Yu Qian <yu...@gmail.com> wrote:
> >
> >> Did anyone consider having a more generic comments feature? The feature
> >> Ace described is an example of a comment being useful. E.g. when a user
> >> clears a task or marks it success/failed, he probably wants to include a
> >> comment for his action when he clicks OK. If a task failed and the user
> >> decided to do nothing about it, he may also wants to leave a comment with
> >> the reason.
> >>
> >> It'll be nice if these comments can optionally be displayed on the UI
> >> somewhere. To do that, it's likely cleaner to create a dedicated comments
> >> table where each comment has a foreign key to associate it with the
> >> TaskInstance.
> >>
> >> On Sat, Oct 17, 2020 at 1:30 AM Vikram Koka <vi...@astronomer.io> wrote:
> >>
> >>> +1 for feature. Strong preference for storing it in the log table,
> >>> rather than in the task instance.
> >>>
> >>>
> >>>
> >>> On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <tu...@apache.org>
> >>> wrote:
> >>>
> >>>> +1, I know a customer who was requesting it too
> >>>>
> >>>> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <ka...@gmail.com> wrote:
> >>>>
> >>>>> +1
> >>>>>
> >>>>> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
> >>>>> wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <
> >>>>>> sumeet.manit@gmail.com> wrote:
> >>>>>>
> >>>>>>> +1 for the feature. Looking at the schema of the log table, I think
> >>>>>>> it's perfect to store such kind of information.
> >>>>>>>
> >>>>>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
> >>>>>>> daniel.imberman@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> This could be pretty valuable for future audits. I’d personally
> >>>>>>>> rather avoid adding fields to the DB in general. Could we store it wherever
> >>>>>>>> we store normal failure messages?
> >>>>>>>>
> >>>>>>>> via Newton Mail
> >>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
> >>>>>>>>
> >>>>>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
> >>>>>>>> <ah...@pinterest.com.invalid> wrote:
> >>>>>>>>
> >>>>>>>> Hi team,
> >>>>>>>>
> >>>>>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to
> >>>>>>>> get some feedback from the community on how they’d propose designing a
> >>>>>>>> feature that has been asked for us multiple times, that we’d like to
> >>>>>>>> implement in-house.
> >>>>>>>>
> >>>>>>>> The ask is the following:
> >>>>>>>> - Users want to be able to add an optional reason/description when
> >>>>>>>> they click on the Mark Success/Failed for both task level in graph views or
> >>>>>>>> dag level in tree views. This is to clearly state why the actions were
> >>>>>>>> taken on higher priority flows for when others are stepping in to look in
> >>>>>>>> it.
> >>>>>>>>
> >>>>>>>> For us the feature makes sense to have, and I was wondering if this
> >>>>>>>> is something the greater group would like to have upstream as well. And if
> >>>>>>>> so then we wanted to know what was preferred.
> >>>>>>>>
> >>>>>>>> 1. We can add this information in the UI for the confirmation view
> >>>>>>>> and an optional textbook to add details similar to how users can add conf
> >>>>>>>> variables when triggering a dag option is selected in the UI.
> >>>>>>>>
> >>>>>>>> Then after this, storing this information is the question. Do we
> >>>>>>>> want to store this information in the action logging extra blob and then
> >>>>>>>> add a view in the dag view where it has actions just pertaining to this dag
> >>>>>>>> and its tasks. (We plan to add this personally anyways for ourselves as
> >>>>>>>> going to the big list of actions for all dags and searching is not
> >>>>>>>> convenient always).
> >>>>>>>>
> >>>>>>>> Or do we want to add a new column in the task Instance model and
> >>>>>>>> dag run model to store the descriptions and retrieve the information from
> >>>>>>>> there.
> >>>>>>>>
> >>>>>>>> Tradeoffs here include the addition of a schema change, where the
> >>>>>>>> new column would generally be sparse, and the data is more resultant from
> >>>>>>>> an action vs necessarily being stored as part of the task run/dag run.
> >>>>>>>>
> >>>>>>>> Please let me know on any feedback, concerns, ideas, and more!
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Ace
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Jarek Potiuk
> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>>
> >>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>
> >>>>>>
>

Re: Airflow Feature Design Feedback

Posted by Kaxil Naik <ka...@gmail.com>.

The Log table can be used at Dag Level too if we pass task&instance=None

For example: We log all the CLI actions in the Log table where we pass
task_instance as None. Code:
https://github.com/apache/airflow/blob/4e32546faf227a6497ce8b282fff7450cae6f665/airflow/utils/cli.py#L133

- Kaxil


On Sun, Oct 18, 2020, 13:49 Deng Xiaodong <xd...@gmail.com> wrote:

> I have the same concern as Yu Qian does, and a bit more elaborations
> follow below:
>
> - "log" table
> <https://github.com/apache/airflow/blob/master/airflow/models/log.py> has
> fields: *id*, *dttm*, *dag_id*, *task_id*, *event*, *execution_date*,
> *owner*, *extra*. Seems to me only "*event*" or "*extra*" can provide
> some room for this use case. But does it support more complicated cases
> well? For example, for the same task instance, there may be back and forth
> tries, and I may mark it Success/Failure for multiple times back and forth,
> with multiple comments.
>
> - As demonstrated by the field list of the "log" table, it's at "task"
> level (it has field "taks_id"). Ace has mentioned that "*Users want to be
> able to add an optional reason/description when they click on the Mark
> Success/Failed for both task level in graph views or dag level in tree
> views*". So my question would be how the "dag level" should be supported
> if we use the "log" table to implement this feature?
>
>
> XD
>
> On Sun, Oct 18, 2020 at 11:13 AM Yu Qian <yu...@gmail.com> wrote:
>
>> Did anyone consider having a more generic comments feature? The feature
>> Ace described is an example of a comment being useful. E.g. when a user
>> clears a task or marks it success/failed, he probably wants to include a
>> comment for his action when he clicks OK. If a task failed and the user
>> decided to do nothing about it, he may also wants to leave a comment with
>> the reason.
>>
>> It'll be nice if these comments can optionally be displayed on the UI
>> somewhere. To do that, it's likely cleaner to create a dedicated comments
>> table where each comment has a foreign key to associate it with the
>> TaskInstance.
>>
>> On Sat, Oct 17, 2020 at 1:30 AM Vikram Koka <vi...@astronomer.io> wrote:
>>
>>> +1 for feature. Strong preference for storing it in the log table,
>>> rather than in the task instance.
>>>
>>>
>>>
>>> On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <tu...@apache.org>
>>> wrote:
>>>
>>>> +1, I know a customer who was requesting it too
>>>>
>>>> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <
>>>>>> sumeet.manit@gmail.com> wrote:
>>>>>>
>>>>>>> +1 for the feature. Looking at the schema of the log table, I think
>>>>>>> it's perfect to store such kind of information.
>>>>>>>
>>>>>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
>>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>>
>>>>>>>> This could be pretty valuable for future audits. I’d personally
>>>>>>>> rather avoid adding fields to the DB in general. Could we store it wherever
>>>>>>>> we store normal failure messages?
>>>>>>>>
>>>>>>>> via Newton Mail
>>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>>>>>>>
>>>>>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>>>>>>>> <ah...@pinterest.com.invalid> wrote:
>>>>>>>>
>>>>>>>> Hi team,
>>>>>>>>
>>>>>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to
>>>>>>>> get some feedback from the community on how they’d propose designing a
>>>>>>>> feature that has been asked for us multiple times, that we’d like to
>>>>>>>> implement in-house.
>>>>>>>>
>>>>>>>> The ask is the following:
>>>>>>>> - Users want to be able to add an optional reason/description when
>>>>>>>> they click on the Mark Success/Failed for both task level in graph views or
>>>>>>>> dag level in tree views. This is to clearly state why the actions were
>>>>>>>> taken on higher priority flows for when others are stepping in to look in
>>>>>>>> it.
>>>>>>>>
>>>>>>>> For us the feature makes sense to have, and I was wondering if this
>>>>>>>> is something the greater group would like to have upstream as well. And if
>>>>>>>> so then we wanted to know what was preferred.
>>>>>>>>
>>>>>>>> 1. We can add this information in the UI for the confirmation view
>>>>>>>> and an optional textbook to add details similar to how users can add conf
>>>>>>>> variables when triggering a dag option is selected in the UI.
>>>>>>>>
>>>>>>>> Then after this, storing this information is the question. Do we
>>>>>>>> want to store this information in the action logging extra blob and then
>>>>>>>> add a view in the dag view where it has actions just pertaining to this dag
>>>>>>>> and its tasks. (We plan to add this personally anyways for ourselves as
>>>>>>>> going to the big list of actions for all dags and searching is not
>>>>>>>> convenient always).
>>>>>>>>
>>>>>>>> Or do we want to add a new column in the task Instance model and
>>>>>>>> dag run model to store the descriptions and retrieve the information from
>>>>>>>> there.
>>>>>>>>
>>>>>>>> Tradeoffs here include the addition of a schema change, where the
>>>>>>>> new column would generally be sparse, and the data is more resultant from
>>>>>>>> an action vs necessarily being stored as part of the task run/dag run.
>>>>>>>>
>>>>>>>> Please let me know on any feedback, concerns, ideas, and more!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Ace
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>>>

Re: Airflow Feature Design Feedback

Posted by Deng Xiaodong <xd...@gmail.com>.

I have the same concern as Yu Qian does, and a bit more elaborations follow
below:

- "log" table
<https://github.com/apache/airflow/blob/master/airflow/models/log.py> has
fields: *id*, *dttm*, *dag_id*, *task_id*, *event*, *execution_date*,
*owner*, *extra*. Seems to me only "*event*" or "*extra*" can provide some
room for this use case. But does it support more complicated cases well?
For example, for the same task instance, there may be back and forth tries,
and I may mark it Success/Failure for multiple times back and forth, with
multiple comments.

- As demonstrated by the field list of the "log" table, it's at "task"
level (it has field "taks_id"). Ace has mentioned that "*Users want to be
able to add an optional reason/description when they click on the Mark
Success/Failed for both task level in graph views or dag level in tree
views*". So my question would be how the "dag level" should be supported if
we use the "log" table to implement this feature?


XD

On Sun, Oct 18, 2020 at 11:13 AM Yu Qian <yu...@gmail.com> wrote:

> Did anyone consider having a more generic comments feature? The feature
> Ace described is an example of a comment being useful. E.g. when a user
> clears a task or marks it success/failed, he probably wants to include a
> comment for his action when he clicks OK. If a task failed and the user
> decided to do nothing about it, he may also wants to leave a comment with
> the reason.
>
> It'll be nice if these comments can optionally be displayed on the UI
> somewhere. To do that, it's likely cleaner to create a dedicated comments
> table where each comment has a foreign key to associate it with the
> TaskInstance.
>
> On Sat, Oct 17, 2020 at 1:30 AM Vikram Koka <vi...@astronomer.io> wrote:
>
>> +1 for feature. Strong preference for storing it in the log table, rather
>> than in the task instance.
>>
>>
>>
>> On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <tu...@apache.org>
>> wrote:
>>
>>> +1, I know a customer who was requesting it too
>>>
>>> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <
>>>>> sumeet.manit@gmail.com> wrote:
>>>>>
>>>>>> +1 for the feature. Looking at the schema of the log table, I think
>>>>>> it's perfect to store such kind of information.
>>>>>>
>>>>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>
>>>>>>> This could be pretty valuable for future audits. I’d personally
>>>>>>> rather avoid adding fields to the DB in general. Could we store it wherever
>>>>>>> we store normal failure messages?
>>>>>>>
>>>>>>> via Newton Mail
>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>>>>>>
>>>>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>>>>>>> <ah...@pinterest.com.invalid> wrote:
>>>>>>>
>>>>>>> Hi team,
>>>>>>>
>>>>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to
>>>>>>> get some feedback from the community on how they’d propose designing a
>>>>>>> feature that has been asked for us multiple times, that we’d like to
>>>>>>> implement in-house.
>>>>>>>
>>>>>>> The ask is the following:
>>>>>>> - Users want to be able to add an optional reason/description when
>>>>>>> they click on the Mark Success/Failed for both task level in graph views or
>>>>>>> dag level in tree views. This is to clearly state why the actions were
>>>>>>> taken on higher priority flows for when others are stepping in to look in
>>>>>>> it.
>>>>>>>
>>>>>>> For us the feature makes sense to have, and I was wondering if this
>>>>>>> is something the greater group would like to have upstream as well. And if
>>>>>>> so then we wanted to know what was preferred.
>>>>>>>
>>>>>>> 1. We can add this information in the UI for the confirmation view
>>>>>>> and an optional textbook to add details similar to how users can add conf
>>>>>>> variables when triggering a dag option is selected in the UI.
>>>>>>>
>>>>>>> Then after this, storing this information is the question. Do we
>>>>>>> want to store this information in the action logging extra blob and then
>>>>>>> add a view in the dag view where it has actions just pertaining to this dag
>>>>>>> and its tasks. (We plan to add this personally anyways for ourselves as
>>>>>>> going to the big list of actions for all dags and searching is not
>>>>>>> convenient always).
>>>>>>>
>>>>>>> Or do we want to add a new column in the task Instance model and dag
>>>>>>> run model to store the descriptions and retrieve the information from
>>>>>>> there.
>>>>>>>
>>>>>>> Tradeoffs here include the addition of a schema change, where the
>>>>>>> new column would generally be sparse, and the data is more resultant from
>>>>>>> an action vs necessarily being stored as part of the task run/dag run.
>>>>>>>
>>>>>>> Please let me know on any feedback, concerns, ideas, and more!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Ace
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>

Re: Airflow Feature Design Feedback

Posted by Yu Qian <yu...@gmail.com>.

Did anyone consider having a more generic comments feature? The feature Ace
described is an example of a comment being useful. E.g. when a user clears
a task or marks it success/failed, he probably wants to include a comment
for his action when he clicks OK. If a task failed and the user decided to
do nothing about it, he may also wants to leave a comment with the reason.

It'll be nice if these comments can optionally be displayed on the UI
somewhere. To do that, it's likely cleaner to create a dedicated comments
table where each comment has a foreign key to associate it with the
TaskInstance.

On Sat, Oct 17, 2020 at 1:30 AM Vikram Koka <vi...@astronomer.io> wrote:

> +1 for feature. Strong preference for storing it in the log table, rather
> than in the task instance.
>
>
>
> On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <tu...@apache.org>
> wrote:
>
>> +1, I know a customer who was requesting it too
>>
>> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <
>>>> sumeet.manit@gmail.com> wrote:
>>>>
>>>>> +1 for the feature. Looking at the schema of the log table, I think
>>>>> it's perfect to store such kind of information.
>>>>>
>>>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
>>>>> daniel.imberman@gmail.com> wrote:
>>>>>
>>>>>> This could be pretty valuable for future audits. I’d personally
>>>>>> rather avoid adding fields to the DB in general. Could we store it wherever
>>>>>> we store normal failure messages?
>>>>>>
>>>>>> via Newton Mail
>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>>>>>
>>>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>>>>>> <ah...@pinterest.com.invalid> wrote:
>>>>>>
>>>>>> Hi team,
>>>>>>
>>>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get
>>>>>> some feedback from the community on how they’d propose designing a feature
>>>>>> that has been asked for us multiple times, that we’d like to implement
>>>>>> in-house.
>>>>>>
>>>>>> The ask is the following:
>>>>>> - Users want to be able to add an optional reason/description when
>>>>>> they click on the Mark Success/Failed for both task level in graph views or
>>>>>> dag level in tree views. This is to clearly state why the actions were
>>>>>> taken on higher priority flows for when others are stepping in to look in
>>>>>> it.
>>>>>>
>>>>>> For us the feature makes sense to have, and I was wondering if this
>>>>>> is something the greater group would like to have upstream as well. And if
>>>>>> so then we wanted to know what was preferred.
>>>>>>
>>>>>> 1. We can add this information in the UI for the confirmation view
>>>>>> and an optional textbook to add details similar to how users can add conf
>>>>>> variables when triggering a dag option is selected in the UI.
>>>>>>
>>>>>> Then after this, storing this information is the question. Do we want
>>>>>> to store this information in the action logging extra blob and then add a
>>>>>> view in the dag view where it has actions just pertaining to this dag and
>>>>>> its tasks. (We plan to add this personally anyways for ourselves as going
>>>>>> to the big list of actions for all dags and searching is not convenient
>>>>>> always).
>>>>>>
>>>>>> Or do we want to add a new column in the task Instance model and dag
>>>>>> run model to store the descriptions and retrieve the information from
>>>>>> there.
>>>>>>
>>>>>> Tradeoffs here include the addition of a schema change, where the new
>>>>>> column would generally be sparse, and the data is more resultant from an
>>>>>> action vs necessarily being stored as part of the task run/dag run.
>>>>>>
>>>>>> Please let me know on any feedback, concerns, ideas, and more!
>>>>>>
>>>>>> Cheers,
>>>>>> Ace
>>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>

Re: Airflow Feature Design Feedback

Posted by Ace Haidrey <ah...@pinterest.com.INVALID>.

Well thank you all for the feedback so quickly! Really appreciate it. We will get started in the coming week internally and hope within two weeks to have a pr for you all to review! I.e. our fabulous engineer Euccas Chen will be :)

Thanks for the notes everyone - 

Ace

> On Oct 16, 2020, at 10:30 AM, Vikram Koka <vi...@astronomer.io> wrote:
> 
> +1 for feature. Strong preference for storing it in the log table, rather than in the task instance. 
> 
> 
> 
> On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <turbaszek@apache.org <ma...@apache.org>> wrote:
> +1, I know a customer who was requesting it too
> 
> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <kaxilnaik@gmail.com <ma...@gmail.com>> wrote:
> +1
> 
> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Jarek.Potiuk@polidea.com <ma...@polidea.com>> wrote:
> +1
> 
> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <sumeet.manit@gmail.com <ma...@gmail.com>> wrote:
> +1 for the feature. Looking at the schema of the log table, I think it's perfect to store such kind of information. 
> 
> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <daniel.imberman@gmail.com <ma...@gmail.com>> wrote:
> This could be pretty valuable for future audits. I’d personally rather avoid adding fields to the DB in general. Could we store it wherever we store normal failure messages?
> 
> via Newton Mail <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey <ah...@pinterest.com.invalid> wrote:
> Hi team,
> 
> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get some feedback from the community on how they’d propose designing a feature that has been asked for us multiple times, that we’d like to implement in-house. 
> 
> The ask is the following:
> - Users want to be able to add an optional reason/description when they click on the Mark Success/Failed for both task level in graph views or dag level in tree views. This is to clearly state why the actions were taken on higher priority flows for when others are stepping in to look in it.
> 
> For us the feature makes sense to have, and I was wondering if this is something the greater group would like to have upstream as well. And if so then we wanted to know what was preferred.
> 
> 1. We can add this information in the UI for the confirmation view and an optional textbook to add details similar to how users can add conf variables when triggering a dag option is selected in the UI.
> 
> Then after this, storing this information is the question. Do we want to store this information in the action logging extra blob and then add a view in the dag view where it has actions just pertaining to this dag and its tasks. (We plan to add this personally anyways for ourselves as going to the big list of actions for all dags and searching is not convenient always).
> 
> Or do we want to add a new column in the task Instance model and dag run model to store the descriptions and retrieve the information from there. 
> 
> Tradeoffs here include the addition of a schema change, where the new column would generally be sparse, and the data is more resultant from an action vs necessarily being stored as part of the task run/dag run.
> 
> Please let me know on any feedback, concerns, ideas, and more!
> 
> Cheers,
> Ace
> 
> 
> -- 
> 	
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
> M: +48 660 796 129 <tel:+48660796129>
>  <https://www.polidea.com/>

Re: Airflow Feature Design Feedback

Posted by Vikram Koka <vi...@astronomer.io>.

+1 for feature. Strong preference for storing it in the log table, rather
than in the task instance.



On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <tu...@apache.org>
wrote:

> +1, I know a customer who was requesting it too
>
> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> +1
>>
>> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <su...@gmail.com>
>>> wrote:
>>>
>>>> +1 for the feature. Looking at the schema of the log table, I think
>>>> it's perfect to store such kind of information.
>>>>
>>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
>>>> daniel.imberman@gmail.com> wrote:
>>>>
>>>>> This could be pretty valuable for future audits. I’d personally rather
>>>>> avoid adding fields to the DB in general. Could we store it wherever we
>>>>> store normal failure messages?
>>>>>
>>>>> via Newton Mail
>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>>>>
>>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>>>>> <ah...@pinterest.com.invalid> wrote:
>>>>>
>>>>> Hi team,
>>>>>
>>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get
>>>>> some feedback from the community on how they’d propose designing a feature
>>>>> that has been asked for us multiple times, that we’d like to implement
>>>>> in-house.
>>>>>
>>>>> The ask is the following:
>>>>> - Users want to be able to add an optional reason/description when
>>>>> they click on the Mark Success/Failed for both task level in graph views or
>>>>> dag level in tree views. This is to clearly state why the actions were
>>>>> taken on higher priority flows for when others are stepping in to look in
>>>>> it.
>>>>>
>>>>> For us the feature makes sense to have, and I was wondering if this is
>>>>> something the greater group would like to have upstream as well. And if so
>>>>> then we wanted to know what was preferred.
>>>>>
>>>>> 1. We can add this information in the UI for the confirmation view and
>>>>> an optional textbook to add details similar to how users can add conf
>>>>> variables when triggering a dag option is selected in the UI.
>>>>>
>>>>> Then after this, storing this information is the question. Do we want
>>>>> to store this information in the action logging extra blob and then add a
>>>>> view in the dag view where it has actions just pertaining to this dag and
>>>>> its tasks. (We plan to add this personally anyways for ourselves as going
>>>>> to the big list of actions for all dags and searching is not convenient
>>>>> always).
>>>>>
>>>>> Or do we want to add a new column in the task Instance model and dag
>>>>> run model to store the descriptions and retrieve the information from
>>>>> there.
>>>>>
>>>>> Tradeoffs here include the addition of a schema change, where the new
>>>>> column would generally be sparse, and the data is more resultant from an
>>>>> action vs necessarily being stored as part of the task run/dag run.
>>>>>
>>>>> Please let me know on any feedback, concerns, ideas, and more!
>>>>>
>>>>> Cheers,
>>>>> Ace
>>>>>
>>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>

Re: Airflow Feature Design Feedback

Posted by Tomasz Urbaszek <tu...@apache.org>.

+1, I know a customer who was requesting it too

On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <ka...@gmail.com> wrote:

> +1
>
> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> +1
>>
>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>
>>> +1 for the feature. Looking at the schema of the log table, I think it's
>>> perfect to store such kind of information.
>>>
>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
>>> daniel.imberman@gmail.com> wrote:
>>>
>>>> This could be pretty valuable for future audits. I’d personally rather
>>>> avoid adding fields to the DB in general. Could we store it wherever we
>>>> store normal failure messages?
>>>>
>>>> via Newton Mail
>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>>>
>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>>>> <ah...@pinterest.com.invalid> wrote:
>>>>
>>>> Hi team,
>>>>
>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get
>>>> some feedback from the community on how they’d propose designing a feature
>>>> that has been asked for us multiple times, that we’d like to implement
>>>> in-house.
>>>>
>>>> The ask is the following:
>>>> - Users want to be able to add an optional reason/description when they
>>>> click on the Mark Success/Failed for both task level in graph views or dag
>>>> level in tree views. This is to clearly state why the actions were taken on
>>>> higher priority flows for when others are stepping in to look in it.
>>>>
>>>> For us the feature makes sense to have, and I was wondering if this is
>>>> something the greater group would like to have upstream as well. And if so
>>>> then we wanted to know what was preferred.
>>>>
>>>> 1. We can add this information in the UI for the confirmation view and
>>>> an optional textbook to add details similar to how users can add conf
>>>> variables when triggering a dag option is selected in the UI.
>>>>
>>>> Then after this, storing this information is the question. Do we want
>>>> to store this information in the action logging extra blob and then add a
>>>> view in the dag view where it has actions just pertaining to this dag and
>>>> its tasks. (We plan to add this personally anyways for ourselves as going
>>>> to the big list of actions for all dags and searching is not convenient
>>>> always).
>>>>
>>>> Or do we want to add a new column in the task Instance model and dag
>>>> run model to store the descriptions and retrieve the information from
>>>> there.
>>>>
>>>> Tradeoffs here include the addition of a schema change, where the new
>>>> column would generally be sparse, and the data is more resultant from an
>>>> action vs necessarily being stored as part of the task run/dag run.
>>>>
>>>> Please let me know on any feedback, concerns, ideas, and more!
>>>>
>>>> Cheers,
>>>> Ace
>>>>
>>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>

Re: Airflow Feature Design Feedback

Posted by Kaxil Naik <ka...@gmail.com>.

+1

On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <Ja...@polidea.com>
wrote:

> +1
>
> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
>> +1 for the feature. Looking at the schema of the log table, I think it's
>> perfect to store such kind of information.
>>
>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <
>> daniel.imberman@gmail.com> wrote:
>>
>>> This could be pretty valuable for future audits. I’d personally rather
>>> avoid adding fields to the DB in general. Could we store it wherever we
>>> store normal failure messages?
>>>
>>> via Newton Mail
>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>>
>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>>> <ah...@pinterest.com.invalid> wrote:
>>>
>>> Hi team,
>>>
>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get
>>> some feedback from the community on how they’d propose designing a feature
>>> that has been asked for us multiple times, that we’d like to implement
>>> in-house.
>>>
>>> The ask is the following:
>>> - Users want to be able to add an optional reason/description when they
>>> click on the Mark Success/Failed for both task level in graph views or dag
>>> level in tree views. This is to clearly state why the actions were taken on
>>> higher priority flows for when others are stepping in to look in it.
>>>
>>> For us the feature makes sense to have, and I was wondering if this is
>>> something the greater group would like to have upstream as well. And if so
>>> then we wanted to know what was preferred.
>>>
>>> 1. We can add this information in the UI for the confirmation view and
>>> an optional textbook to add details similar to how users can add conf
>>> variables when triggering a dag option is selected in the UI.
>>>
>>> Then after this, storing this information is the question. Do we want to
>>> store this information in the action logging extra blob and then add a view
>>> in the dag view where it has actions just pertaining to this dag and its
>>> tasks. (We plan to add this personally anyways for ourselves as going to
>>> the big list of actions for all dags and searching is not convenient
>>> always).
>>>
>>> Or do we want to add a new column in the task Instance model and dag run
>>> model to store the descriptions and retrieve the information from there.
>>>
>>> Tradeoffs here include the addition of a schema change, where the new
>>> column would generally be sparse, and the data is more resultant from an
>>> action vs necessarily being stored as part of the task run/dag run.
>>>
>>> Please let me know on any feedback, concerns, ideas, and more!
>>>
>>> Cheers,
>>> Ace
>>>
>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>

Re: Airflow Feature Design Feedback

Posted by Jarek Potiuk <Ja...@polidea.com>.

+1

On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari <su...@gmail.com>
wrote:

> +1 for the feature. Looking at the schema of the log table, I think it's
> perfect to store such kind of information.
>
> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> This could be pretty valuable for future audits. I’d personally rather
>> avoid adding fields to the DB in general. Could we store it wherever we
>> store normal failure messages?
>>
>> via Newton Mail
>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>
>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>> <ah...@pinterest.com.invalid> wrote:
>>
>> Hi team,
>>
>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get
>> some feedback from the community on how they’d propose designing a feature
>> that has been asked for us multiple times, that we’d like to implement
>> in-house.
>>
>> The ask is the following:
>> - Users want to be able to add an optional reason/description when they
>> click on the Mark Success/Failed for both task level in graph views or dag
>> level in tree views. This is to clearly state why the actions were taken on
>> higher priority flows for when others are stepping in to look in it.
>>
>> For us the feature makes sense to have, and I was wondering if this is
>> something the greater group would like to have upstream as well. And if so
>> then we wanted to know what was preferred.
>>
>> 1. We can add this information in the UI for the confirmation view and an
>> optional textbook to add details similar to how users can add conf
>> variables when triggering a dag option is selected in the UI.
>>
>> Then after this, storing this information is the question. Do we want to
>> store this information in the action logging extra blob and then add a view
>> in the dag view where it has actions just pertaining to this dag and its
>> tasks. (We plan to add this personally anyways for ourselves as going to
>> the big list of actions for all dags and searching is not convenient
>> always).
>>
>> Or do we want to add a new column in the task Instance model and dag run
>> model to store the descriptions and retrieve the information from there.
>>
>> Tradeoffs here include the addition of a schema change, where the new
>> column would generally be sparse, and the data is more resultant from an
>> action vs necessarily being stored as part of the task run/dag run.
>>
>> Please let me know on any feedback, concerns, ideas, and more!
>>
>> Cheers,
>> Ace
>>
>>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: Airflow Feature Design Feedback

Posted by Tao Feng <fe...@gmail.com>.

+1, Lyft uses the extra blob in the log table? to store UI
auditing information.

On Fri, Oct 16, 2020 at 9:42 AM Sumit Maheshwari <su...@gmail.com>
wrote:

> +1 for the feature. Looking at the schema of the log table, I think it's
> perfect to store such kind of information.
>
> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> This could be pretty valuable for future audits. I’d personally rather
>> avoid adding fields to the DB in general. Could we store it wherever we
>> store normal failure messages?
>>
>> via Newton Mail
>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>>
>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
>> <ah...@pinterest.com.invalid> wrote:
>>
>> Hi team,
>>
>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get
>> some feedback from the community on how they’d propose designing a feature
>> that has been asked for us multiple times, that we’d like to implement
>> in-house.
>>
>> The ask is the following:
>> - Users want to be able to add an optional reason/description when they
>> click on the Mark Success/Failed for both task level in graph views or dag
>> level in tree views. This is to clearly state why the actions were taken on
>> higher priority flows for when others are stepping in to look in it.
>>
>> For us the feature makes sense to have, and I was wondering if this is
>> something the greater group would like to have upstream as well. And if so
>> then we wanted to know what was preferred.
>>
>> 1. We can add this information in the UI for the confirmation view and an
>> optional textbook to add details similar to how users can add conf
>> variables when triggering a dag option is selected in the UI.
>>
>> Then after this, storing this information is the question. Do we want to
>> store this information in the action logging extra blob and then add a view
>> in the dag view where it has actions just pertaining to this dag and its
>> tasks. (We plan to add this personally anyways for ourselves as going to
>> the big list of actions for all dags and searching is not convenient
>> always).
>>
>> Or do we want to add a new column in the task Instance model and dag run
>> model to store the descriptions and retrieve the information from there.
>>
>> Tradeoffs here include the addition of a schema change, where the new
>> column would generally be sparse, and the data is more resultant from an
>> action vs necessarily being stored as part of the task run/dag run.
>>
>> Please let me know on any feedback, concerns, ideas, and more!
>>
>> Cheers,
>> Ace
>>
>>

Re: Airflow Feature Design Feedback

Posted by Sumit Maheshwari <su...@gmail.com>.

+1 for the feature. Looking at the schema of the log table, I think it's
perfect to store such kind of information.

On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <da...@gmail.com>
wrote:

> This could be pretty valuable for future audits. I’d personally rather
> avoid adding fields to the DB in general. Could we store it wherever we
> store normal failure messages?
>
> via Newton Mail
> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2>
>
> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey
> <ah...@pinterest.com.invalid> wrote:
>
> Hi team,
>
> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get some
> feedback from the community on how they’d propose designing a feature that
> has been asked for us multiple times, that we’d like to implement in-house.
>
> The ask is the following:
> - Users want to be able to add an optional reason/description when they
> click on the Mark Success/Failed for both task level in graph views or dag
> level in tree views. This is to clearly state why the actions were taken on
> higher priority flows for when others are stepping in to look in it.
>
> For us the feature makes sense to have, and I was wondering if this is
> something the greater group would like to have upstream as well. And if so
> then we wanted to know what was preferred.
>
> 1. We can add this information in the UI for the confirmation view and an
> optional textbook to add details similar to how users can add conf
> variables when triggering a dag option is selected in the UI.
>
> Then after this, storing this information is the question. Do we want to
> store this information in the action logging extra blob and then add a view
> in the dag view where it has actions just pertaining to this dag and its
> tasks. (We plan to add this personally anyways for ourselves as going to
> the big list of actions for all dags and searching is not convenient
> always).
>
> Or do we want to add a new column in the task Instance model and dag run
> model to store the descriptions and retrieve the information from there.
>
> Tradeoffs here include the addition of a schema change, where the new
> column would generally be sparse, and the data is more resultant from an
> action vs necessarily being stored as part of the task run/dag run.
>
> Please let me know on any feedback, concerns, ideas, and more!
>
> Cheers,
> Ace
>
>

Re: Airflow Feature Design Feedback

Posted by Daniel Imberman <da...@gmail.com>.

This could be pretty valuable for future audits. I’d personally rather avoid adding fields to the DB in general. Could we store it wherever we store normal failure messages?

via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2]
On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey <ah...@pinterest.com.invalid> wrote:
Hi team,

For us the feature makes sense to have, and I was wondering if this is something the greater group would like to have upstream as well. And if so then we wanted to know what was preferred.

Or do we want to add a new column in the task Instance model and dag run model to store the descriptions and retrieve the information from there.

Please let me know on any feedback, concerns, ideas, and more!

Cheers,
Ace