You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Colin Son <co...@thinknear.com> on 2016/10/10 19:54:49 UTC
Airflow Web UI Question
Hello,
Whenever a Task Instance fails, retries, and succeeds, I noticed that the
WEB UI doesn't show you that it failed/retried. You would have to look in
the logs of that failed Task Instance and see if that Task Instance failed
or not. How come it doesn't show that the Task Instance's state initially
was failed/retried in the Task Instances page in the Web UI? I noticed that
it marks the state as "Success", even after it failed/retried/and
succeeded. Is there a way to change this?
Thanks
Re: Airflow Web UI Question
Posted by siddharth anand <sa...@apache.org>.
It's in master. It hasn't been picked up in a release yet.
-s
On Mon, Oct 10, 2016 at 3:49 PM, Colin Son <co...@thinknear.com> wrote:
> Hello,
>
> Do you happen to know in what version of Airflow that "task_fail" table was
> introduced?
>
> Thanks
>
> On Mon, Oct 10, 2016 at 3:39 PM, siddharth anand <sa...@apache.org>
> wrote:
>
> > Yes.
> >
> > The task instance table only has one row per unique key : (dag_id,
> task_id,
> > execution_date). To support the "cumulative duration" functionality, we
> had
> > to write "failed task instance rows" to a new table called *task_fail*
> > (a.k.a.
> > TaskFail in models.py). That second table is not currently read to build
> > the task_instances view (in views.py).
> >
> > This should IMHO be a feature of the task_intances view - anyone want to
> > submit a pull request for this feature :
> > https://issues.apache.org/jira/browse/AIRFLOW-562?
> > -s
> >
> > On Mon, Oct 10, 2016 at 1:23 PM, Colin Son <co...@thinknear.com> wrote:
> >
> > > Hello,
> > >
> > > In the article that you sent me, it states this:
> > >
> > > "As mentioned earlier, the current Task Duration chart is misleading.
> For
> > > example, in the chart below, it appears as if each of our 3 tasks is
> > > performing in a nearly consistent manner from run to run. However, what
> > > happens if there are transient failures and retries? Unfortunately, the
> > > chart below only displays the Task Duration for successful attempts and
> > not
> > > the cumulative time taken for that task to eventually succeed!"
> > >
> > > Does this imply that the UI Page under Browse -> Task Instances, will
> > also
> > > only display successful attempts, when the task succeeds on subsequent
> > > retries?
> > >
> > > Thanks
> > >
> > > On Mon, Oct 10, 2016 at 1:06 PM, siddharth anand <sa...@apache.org>
> > > wrote:
> > >
> > > > Colin,
> > > >
> > > > Have a look at the link below:
> > > > https://www.agari.com/a-summer-interns-journey-into-airflow-agari/
> > > >
> > > > We committed some changes to master over the Summer that provide a
> few
> > > new
> > > > UI controls:
> > > > * A new Task Tries view that plots Task Tries as a function of time
> for
> > > > each task in a DAG
> > > > * A Cumulative Duration checkbox to show the total time a task takes,
> > > > including all retries, not just the time taken for the task's
> > eventually
> > > > successful run.
> > > >
> > > > -s
> > > >
> > > > On Mon, Oct 10, 2016 at 12:54 PM, Colin Son <co...@thinknear.com>
> > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Whenever a Task Instance fails, retries, and succeeds, I noticed
> that
> > > the
> > > > > WEB UI doesn't show you that it failed/retried. You would have to
> > look
> > > in
> > > > > the logs of that failed Task Instance and see if that Task Instance
> > > > failed
> > > > > or not. How come it doesn't show that the Task Instance's state
> > > initially
> > > > > was failed/retried in the Task Instances page in the Web UI? I
> > noticed
> > > > that
> > > > > it marks the state as "Success", even after it failed/retried/and
> > > > > succeeded. Is there a way to change this?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>
Re: Airflow Web UI Question
Posted by Colin Son <co...@thinknear.com>.
Hello,
Do you happen to know in what version of Airflow that "task_fail" table was
introduced?
Thanks
On Mon, Oct 10, 2016 at 3:39 PM, siddharth anand <sa...@apache.org> wrote:
> Yes.
>
> The task instance table only has one row per unique key : (dag_id, task_id,
> execution_date). To support the "cumulative duration" functionality, we had
> to write "failed task instance rows" to a new table called *task_fail*
> (a.k.a.
> TaskFail in models.py). That second table is not currently read to build
> the task_instances view (in views.py).
>
> This should IMHO be a feature of the task_intances view - anyone want to
> submit a pull request for this feature :
> https://issues.apache.org/jira/browse/AIRFLOW-562?
> -s
>
> On Mon, Oct 10, 2016 at 1:23 PM, Colin Son <co...@thinknear.com> wrote:
>
> > Hello,
> >
> > In the article that you sent me, it states this:
> >
> > "As mentioned earlier, the current Task Duration chart is misleading. For
> > example, in the chart below, it appears as if each of our 3 tasks is
> > performing in a nearly consistent manner from run to run. However, what
> > happens if there are transient failures and retries? Unfortunately, the
> > chart below only displays the Task Duration for successful attempts and
> not
> > the cumulative time taken for that task to eventually succeed!"
> >
> > Does this imply that the UI Page under Browse -> Task Instances, will
> also
> > only display successful attempts, when the task succeeds on subsequent
> > retries?
> >
> > Thanks
> >
> > On Mon, Oct 10, 2016 at 1:06 PM, siddharth anand <sa...@apache.org>
> > wrote:
> >
> > > Colin,
> > >
> > > Have a look at the link below:
> > > https://www.agari.com/a-summer-interns-journey-into-airflow-agari/
> > >
> > > We committed some changes to master over the Summer that provide a few
> > new
> > > UI controls:
> > > * A new Task Tries view that plots Task Tries as a function of time for
> > > each task in a DAG
> > > * A Cumulative Duration checkbox to show the total time a task takes,
> > > including all retries, not just the time taken for the task's
> eventually
> > > successful run.
> > >
> > > -s
> > >
> > > On Mon, Oct 10, 2016 at 12:54 PM, Colin Son <co...@thinknear.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > Whenever a Task Instance fails, retries, and succeeds, I noticed that
> > the
> > > > WEB UI doesn't show you that it failed/retried. You would have to
> look
> > in
> > > > the logs of that failed Task Instance and see if that Task Instance
> > > failed
> > > > or not. How come it doesn't show that the Task Instance's state
> > initially
> > > > was failed/retried in the Task Instances page in the Web UI? I
> noticed
> > > that
> > > > it marks the state as "Success", even after it failed/retried/and
> > > > succeeded. Is there a way to change this?
> > > >
> > > > Thanks
> > > >
> > >
> >
>
Re: Airflow Web UI Question
Posted by siddharth anand <sa...@apache.org>.
Yes.
The task instance table only has one row per unique key : (dag_id, task_id,
execution_date). To support the "cumulative duration" functionality, we had
to write "failed task instance rows" to a new table called *task_fail* (a.k.a.
TaskFail in models.py). That second table is not currently read to build
the task_instances view (in views.py).
This should IMHO be a feature of the task_intances view - anyone want to
submit a pull request for this feature :
https://issues.apache.org/jira/browse/AIRFLOW-562?
-s
On Mon, Oct 10, 2016 at 1:23 PM, Colin Son <co...@thinknear.com> wrote:
> Hello,
>
> In the article that you sent me, it states this:
>
> "As mentioned earlier, the current Task Duration chart is misleading. For
> example, in the chart below, it appears as if each of our 3 tasks is
> performing in a nearly consistent manner from run to run. However, what
> happens if there are transient failures and retries? Unfortunately, the
> chart below only displays the Task Duration for successful attempts and not
> the cumulative time taken for that task to eventually succeed!"
>
> Does this imply that the UI Page under Browse -> Task Instances, will also
> only display successful attempts, when the task succeeds on subsequent
> retries?
>
> Thanks
>
> On Mon, Oct 10, 2016 at 1:06 PM, siddharth anand <sa...@apache.org>
> wrote:
>
> > Colin,
> >
> > Have a look at the link below:
> > https://www.agari.com/a-summer-interns-journey-into-airflow-agari/
> >
> > We committed some changes to master over the Summer that provide a few
> new
> > UI controls:
> > * A new Task Tries view that plots Task Tries as a function of time for
> > each task in a DAG
> > * A Cumulative Duration checkbox to show the total time a task takes,
> > including all retries, not just the time taken for the task's eventually
> > successful run.
> >
> > -s
> >
> > On Mon, Oct 10, 2016 at 12:54 PM, Colin Son <co...@thinknear.com>
> wrote:
> >
> > > Hello,
> > >
> > > Whenever a Task Instance fails, retries, and succeeds, I noticed that
> the
> > > WEB UI doesn't show you that it failed/retried. You would have to look
> in
> > > the logs of that failed Task Instance and see if that Task Instance
> > failed
> > > or not. How come it doesn't show that the Task Instance's state
> initially
> > > was failed/retried in the Task Instances page in the Web UI? I noticed
> > that
> > > it marks the state as "Success", even after it failed/retried/and
> > > succeeded. Is there a way to change this?
> > >
> > > Thanks
> > >
> >
>
Re: Airflow Web UI Question
Posted by Colin Son <co...@thinknear.com>.
Hello,
In the article that you sent me, it states this:
"As mentioned earlier, the current Task Duration chart is misleading. For
example, in the chart below, it appears as if each of our 3 tasks is
performing in a nearly consistent manner from run to run. However, what
happens if there are transient failures and retries? Unfortunately, the
chart below only displays the Task Duration for successful attempts and not
the cumulative time taken for that task to eventually succeed!"
Does this imply that the UI Page under Browse -> Task Instances, will also
only display successful attempts, when the task succeeds on subsequent
retries?
Thanks
On Mon, Oct 10, 2016 at 1:06 PM, siddharth anand <sa...@apache.org> wrote:
> Colin,
>
> Have a look at the link below:
> https://www.agari.com/a-summer-interns-journey-into-airflow-agari/
>
> We committed some changes to master over the Summer that provide a few new
> UI controls:
> * A new Task Tries view that plots Task Tries as a function of time for
> each task in a DAG
> * A Cumulative Duration checkbox to show the total time a task takes,
> including all retries, not just the time taken for the task's eventually
> successful run.
>
> -s
>
> On Mon, Oct 10, 2016 at 12:54 PM, Colin Son <co...@thinknear.com> wrote:
>
> > Hello,
> >
> > Whenever a Task Instance fails, retries, and succeeds, I noticed that the
> > WEB UI doesn't show you that it failed/retried. You would have to look in
> > the logs of that failed Task Instance and see if that Task Instance
> failed
> > or not. How come it doesn't show that the Task Instance's state initially
> > was failed/retried in the Task Instances page in the Web UI? I noticed
> that
> > it marks the state as "Success", even after it failed/retried/and
> > succeeded. Is there a way to change this?
> >
> > Thanks
> >
>
Re: Airflow Web UI Question
Posted by siddharth anand <sa...@apache.org>.
Colin,
Have a look at the link below:
https://www.agari.com/a-summer-interns-journey-into-airflow-agari/
We committed some changes to master over the Summer that provide a few new
UI controls:
* A new Task Tries view that plots Task Tries as a function of time for
each task in a DAG
* A Cumulative Duration checkbox to show the total time a task takes,
including all retries, not just the time taken for the task's eventually
successful run.
-s
On Mon, Oct 10, 2016 at 12:54 PM, Colin Son <co...@thinknear.com> wrote:
> Hello,
>
> Whenever a Task Instance fails, retries, and succeeds, I noticed that the
> WEB UI doesn't show you that it failed/retried. You would have to look in
> the logs of that failed Task Instance and see if that Task Instance failed
> or not. How come it doesn't show that the Task Instance's state initially
> was failed/retried in the Task Instances page in the Web UI? I noticed that
> it marks the state as "Success", even after it failed/retried/and
> succeeded. Is there a way to change this?
>
> Thanks
>