[jira] [Created] (MESOS-2334) Tasks get stuck in TASK_STAGING after a network decode error

["Andreas Raster (JIRA)" <ji...@apache.org>]

Andreas Raster created MESOS-2334:
-------------------------------------

             Summary: Tasks get stuck in TASK_STAGING after a network decode error
                 Key: MESOS-2334
                 URL: https://issues.apache.org/jira/browse/MESOS-2334
             Project: Mesos
          Issue Type: Bug
            Reporter: Andreas Raster


We observed that with a test case that schedules a large amount of small CommandInfo tasks (shell commands that look like this: "sleep `shuf -i 2-3 -n 1`; echo foo >> /share/bar") on a cluster with launchTasks, that sometimes we would get an issue where a single task that has been launched and was set to TASK_STAGING would never receive a TASK_RUNNING message (or any other message at all). So it would then just stay in TASK_STAGING infinitely until we would kill the framework.

We asked in #mesos on freenode about this and got an answer from alexr_:

[15:56:55] <alexr_> henno: thanks for the slave logs
[15:57:09] rakete [~rakete@static.198.2.63.178.clients.your-server.de] has left #mesos
[15:58:47] <alexr_> henno: it looks from the logs, that the slave successfully registers the executor and sends the task
[15:59:07] tillt_ [~Till@212.53.142.20] has joined #mesos
[15:59:30] <alexr_> the executor, for some reason, refuses to start the task, most probably because of the message decoding error

telling us that he suspects the reason is a network decoding error. I am currently not 100% sure what he means by that and I wasn't the guy talking to alexr_ on irc so I cannot post the exact log section that indicates that decoding error. But I'll attach the logs that we supplied to alexr_, so those should contain the relevant information.

The tasks name in question was: 727527fc-a3f3-418d-a44e-ec3bbdd26315

cat /var/log/mesos/mesos-slave.INFO | grep 727527fc-a3f3-418d-a44e-ec3bbdd26315
>> http://paste.ubuntu.com/10160270/

cat /tmp/mesos/slaves/20141217-133241-2867204268-5050-12776-S1/frameworks/20150209-153125-2867204268-5050-2553-0025/executors/29cde3b3-994a-4480-b10e-c49b4fc6c706+0+727527fc-a3f3-418d-a44e-ec3bbdd26315/runs/d73a76e7-aec6-4760-bd48-86b79df89d52/stderr 
>> http://paste.ubuntu.com/10160335/

cat /tmp/mesos/slaves/20141217-133241-2867204268-5050-12776-S1/frameworks/20150209-153125-2867204268-5050-2553-0025/executors/29cde3b3-994a-4480-b10e-c49b4fc6c706+0+727527fc-a3f3-418d-a44e-ec3bbdd26315/runs/d73a76e7-aec6-4760-bd48-86b79df89d52/stdout
>> http://paste.ubuntu.com/10160346/

Now, if some relevant information is still missing, don't hesitate to ask me.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)