Tasks getting killed as Zombie process ( Mysql DB download specific tasks)

[twinkle <tw...@gmail.com>]

Hi,

I am using airflow v1.7.1.3. Some of the tasks in the pipeline gets killed
as Zombie.

A pattern in that has come out that it happens in those jobs which are
downloading data from MySql DB.

I am doing the following steps in those tasks:

1. getting connection from db, using Hook
2. Execute the query
3. Use csv_write to write the results in csv format.
4. flush the file
5. close the query cursor and then the connection
6. run the gc ( as otherwise we get memory issues)


Sometimes the task is successful and sometimes it is not. This behaviour
require some manual monitoring, which is not desirable.

What can I do to make sure that tasks do not get killed as zombie.

Also, I found that there is a property scheduler_zombie_task_threshold, if
i increase it , then what are the impactful area for this?

Regards,
Twinkle

Re: Tasks getting killed as Zombie process ( Mysql DB download specific tasks)

[Taylor Edmiston <te...@gmail.com>]

The scheduler_zombie_task_threshold var is the number of seconds a job can
live without heartbeat before being killed as a zombie and rescheduled.

Reference -
https://github.com/apache/incubator-airflow/blob/48dab65adc69cd924fd918c6a2934006971fb25d/airflow/config_templates/default_airflow.cfg#L381-L384

*Taylor Edmiston*
TEdmiston.com <https://www.tedmiston.com/> | Blog
<http://blog.tedmiston.com>
Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>


On Mon, Mar 19, 2018 at 12:59 PM, Matthew Housley <matthew.housley@gmail.com
> wrote:

> Hi Twinkle,
> Airflow 1.7 reached end of life roughly a year ago.
> https://cwiki.apache.org/confluence/display/AIRFLOW/
> Airflow+Release+Planning+and+Supported+Release+Lifetime
>
> Could you do some testing to see if you can reproduce this issue with
> Airflow 1.9?
> best,
> Matt
>
> On Fri, Mar 16, 2018 at 4:56 AM twinkle <tw...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am using airflow v1.7.1.3. Some of the tasks in the pipeline gets
> killed
> > as Zombie.
> >
> > A pattern in that has come out that it happens in those jobs which are
> > downloading data from MySql DB.
> >
> > I am doing the following steps in those tasks:
> >
> > 1. getting connection from db, using Hook
> > 2. Execute the query
> > 3. Use csv_write to write the results in csv format.
> > 4. flush the file
> > 5. close the query cursor and then the connection
> > 6. run the gc ( as otherwise we get memory issues)
> >
> >
> > Sometimes the task is successful and sometimes it is not. This behaviour
> > require some manual monitoring, which is not desirable.
> >
> > What can I do to make sure that tasks do not get killed as zombie.
> >
> > Also, I found that there is a property scheduler_zombie_task_threshold,
> if
> > i increase it , then what are the impactful area for this?
> >
> > Regards,
> > Twinkle
> >
>

Re: Tasks getting killed as Zombie process ( Mysql DB download specific tasks)

[Matthew Housley <ma...@gmail.com>]

Hi Twinkle,
Airflow 1.7 reached end of life roughly a year ago.
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime

Could you do some testing to see if you can reproduce this issue with
Airflow 1.9?
best,
Matt

On Fri, Mar 16, 2018 at 4:56 AM twinkle <tw...@gmail.com> wrote:

> Hi,
>
> I am using airflow v1.7.1.3. Some of the tasks in the pipeline gets killed
> as Zombie.
>
> A pattern in that has come out that it happens in those jobs which are
> downloading data from MySql DB.
>
> I am doing the following steps in those tasks:
>
> 1. getting connection from db, using Hook
> 2. Execute the query
> 3. Use csv_write to write the results in csv format.
> 4. flush the file
> 5. close the query cursor and then the connection
> 6. run the gc ( as otherwise we get memory issues)
>
>
> Sometimes the task is successful and sometimes it is not. This behaviour
> require some manual monitoring, which is not desirable.
>
> What can I do to make sure that tasks do not get killed as zombie.
>
> Also, I found that there is a property scheduler_zombie_task_threshold, if
> i increase it , then what are the impactful area for this?
>
> Regards,
> Twinkle
>