[jira] [Created] (MESOS-5295) The task launched by non-checkpointed HTTP command executor will keep running after agent is done

["Qian Zhang (JIRA)" <ji...@apache.org>]

Qian Zhang created MESOS-5295:
---------------------------------

             Summary: The task launched by non-checkpointed HTTP command executor will keep running after agent is done
                 Key: MESOS-5295
                 URL: https://issues.apache.org/jira/browse/MESOS-5295
             Project: Mesos
          Issue Type: Bug
          Components: HTTP API
            Reporter: Qian Zhang
            Assignee: Qian Zhang


When I test HTTP command executor, I found an issue, here is my steps:
1. A framework which has no checkpoint enabled launches a long running task (e.g., sleep 1000).
2. After the task is running, kill the agent.
 
Then I see the HTTP command executor will terminate after 5s ("DEFAULT_EXECUTOR_SHUTDOWN_GRACE_PERIOD"), but the task will always run. This behavior is not consistent with driver based command executor: after agent is killed, that executor will kill the task and then self terminate after 1s (there is a "os::sleep(Seconds(1));" in "reaped()").

The root cause of this difference is, for driver based command executor, when the driver found agent is down, it will call executor->shutdown() (https://github.com/apache/mesos/blob/0.28.1/src/exec/exec.cpp#L487), so the executor will kill the task and then self terminate. But for HTTP command executor, its "disconnected()" will be called (https://github.com/apache/mesos/blob/0.28.1/src/executor/executor.cpp#L388) when agent is down, and currently we do not do anything in its "disconnected()", so the task will keep running and the executor will be killed after 5s (https://github.com/apache/mesos/blob/0.28.1/src/executor/executor.cpp#L623).

The behavior of driver based command executor is correct, we need to make sure HTTP command executor kills the task when agent is down if checkpoint is not enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)