You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Michael Gong <go...@hotmail.com> on 2018/01/16 18:30:42 UTC
Airflow (1.6) dag's status "Failed" even when a task was running
Hi, Airflow developers,
On Airflow 1.6, we found sometimes a dag's status was set to "Failed" even when some tasks were still running.
The steps to illustrate this behavior is: (suppose a dag contains 2 tasks , A and B.)
1.
let task A and B starts running. --> The dag status is "Running".
2.
Clear task A. --> the task A process will be killed and the task A will set to "Failed".
3.
Task B is still running.
4.
The dag status becomes "Failed", even though the Task B is still running.
I wonder whether this is an Airflow bug or an expected behavior.
Any comments are welcome.
Thanks.
Michael
________________________________
From: Michael Gong <go...@hotmail.com>
Sent: Friday, March 3, 2017 8:56 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: "last_expired" column in the dag table
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Michael,
Could you share the schedule with us? How is it configured. Please provide
the DAG object with the cron definition.
Cheers, Fokko
2018-02-21 5:09 GMT+01:00 Michael Gong <go...@hotmail.com>:
> Hi, airflow dev,
>
>
> We set up several dags running on the airflow server. Most dags run fine,
> but there is one dag which doesn't start at the scheduled time.
>
>
> Then we found in the "dag" table, there is a "last_expired" column. The
> not-started dag has a value, but it's empty for all other dags.
>
>
> Did anyone know what "last_expired" column means ? Could it prevent a dag
> to run ?
>
>
> Any comments are welcomed.
>
>
> Thanks.
>
> Michael
>
>
>
"last_expired" column in the dag table
Posted by Michael Gong <go...@hotmail.com>.
Hi, airflow dev,
We set up several dags running on the airflow server. Most dags run fine, but there is one dag which doesn't start at the scheduled time.
Then we found in the "dag" table, there is a "last_expired" column. The not-started dag has a value, but it's empty for all other dags.
Did anyone know what "last_expired" column means ? Could it prevent a dag to run ?
Any comments are welcomed.
Thanks.
Michael
Re: Airflow (1.6) dag's status "Failed" even when a task was running
Posted by Bolke de Bruin <bd...@gmail.com>.
Airflow 1.6 ??!!
I don’t think any of the maintainers is running that version anymore.
> On 16 Jan 2018, at 19:30, Michael Gong <go...@hotmail.com> wrote:
>
> Hi, Airflow developers,
>
>
> On Airflow 1.6, we found sometimes a dag's status was set to "Failed" even when some tasks were still running.
>
>
> The steps to illustrate this behavior is: (suppose a dag contains 2 tasks , A and B.)
>
>
> 1.
> let task A and B starts running. --> The dag status is "Running".
> 2.
> Clear task A. --> the task A process will be killed and the task A will set to "Failed".
> 3.
> Task B is still running.
> 4.
> The dag status becomes "Failed", even though the Task B is still running.
>
>
> I wonder whether this is an Airflow bug or an expected behavior.
>
>
>
> Any comments are welcome.
>
>
> Thanks.
>
> Michael
>
>
>
>
>
>
>
> ________________________________
> From: Michael Gong <go...@hotmail.com>
> Sent: Friday, March 3, 2017 8:56 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow running different with different user id ?
>
> Hi, Dan,
>
>
> Thanks for the encouraging news.
>
>
> Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
>
>
> I guess it will be passed as an argument for the DAG object , right ?
>
>
> Thanks again.
>
>
> Michael
>
>
>
>
> ________________________________
> From: Dan Davydov <da...@airbnb.com.INVALID>
> Sent: Friday, March 3, 2017 8:44 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow running different with different user id ?
>
> Within a couple of weeks.
>
> On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
>
>> When approximately will it be released?
>>
>> Sent from my PP•KING™ smartphone
>>
>> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
>> wrote:
>> Yes it is starting on 1.8.0 which will be released soon, you can look in
>> the documentation/grep for "run_as".
>>
>> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>>
>>> Hi,
>>>
>>>
>>> Suppose I have 1 airflow instance running 2 different DAGs, is it
>> possible
>>> to specify the 2 DAGs running under 2 different ids ?
>>>
>>>
>>> Any advises are welcomed.
>>>
>>>
>>> Thanks.
>>>
>>> Michael
>>>
>>>
>>>
>>>
>>>
>>