Custom Executor issue, killTask, task LOST

[Eren Güven <er...@gmail.com>]

Hello,

*TL;DR*: What is the proper way to kill executor process(tree) from within
executor code?

I have an executor written in python that I use with Marathon to launch my
tasks. I use it to have a control on task status switch STAGING => RUNNING.
I launch a long-running subprocess from the `launchTask`

My problem is that (following the examples such as RENDLER) if I do
`sys.exit(0)` at the end of `killTask`, I get "Failed to call executor's
killTask" on stderr and get TASK_LOST. If I omit the `sys.exit(0)` then a
leftover executor process stays on the slave.

Cheers
Eren

Re: Custom Executor issue, killTask, task LOST

[Eren Güven <er...@gmail.com>]

Thanks for the answer David, I ended up doing exactly that.

On 23 July 2015 at 17:13, David Greenberg <ds...@gmail.com> wrote:

> It's the responsibility of each executor to shut itself down when it
> decides its no longer needed. The idea here is that an executor can stay
> running with zero tasks, so that when the scheduler decides to launch a
> task, the scheduler's already initialized, thus saving startup time.
>
> To avoid the problem you're seeing, have killTask schedule the sys.exit(0)
> to occur in 2 seconds--this will give time for your
> driver.sendStatusUpdate(taskid, TASK_KILLED) to fully send the status, and
> then you can safely exit the executor.
>
> On Thu, Jul 23, 2015 at 8:10 AM Eren Güven <er...@gmail.com> wrote:
>
>> Hello,
>>
>> *TL;DR*: What is the proper way to kill executor process(tree) from
>> within executor code?
>>
>> I have an executor written in python that I use with Marathon to launch
>> my tasks. I use it to have a control on task status switch STAGING =>
>> RUNNING. I launch a long-running subprocess from the `launchTask`
>>
>> My problem is that (following the examples such as RENDLER) if I do
>> `sys.exit(0)` at the end of `killTask`, I get "Failed to call executor's
>> killTask" on stderr and get TASK_LOST. If I omit the `sys.exit(0)` then a
>> leftover executor process stays on the slave.
>>
>> Cheers
>> Eren
>>
>

Re: Custom Executor issue, killTask, task LOST

[David Greenberg <ds...@gmail.com>]

It's the responsibility of each executor to shut itself down when it
decides its no longer needed. The idea here is that an executor can stay
running with zero tasks, so that when the scheduler decides to launch a
task, the scheduler's already initialized, thus saving startup time.

To avoid the problem you're seeing, have killTask schedule the sys.exit(0)
to occur in 2 seconds--this will give time for your
driver.sendStatusUpdate(taskid, TASK_KILLED) to fully send the status, and
then you can safely exit the executor.

On Thu, Jul 23, 2015 at 8:10 AM Eren Güven <er...@gmail.com> wrote:

> Hello,
>
> *TL;DR*: What is the proper way to kill executor process(tree) from
> within executor code?
>
> I have an executor written in python that I use with Marathon to launch my
> tasks. I use it to have a control on task status switch STAGING => RUNNING.
> I launch a long-running subprocess from the `launchTask`
>
> My problem is that (following the examples such as RENDLER) if I do
> `sys.exit(0)` at the end of `killTask`, I get "Failed to call executor's
> killTask" on stderr and get TASK_LOST. If I omit the `sys.exit(0)` then a
> leftover executor process stays on the slave.
>
> Cheers
> Eren
>