You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Elizabeth Lingg (JIRA)" <ji...@apache.org> on 2015/04/10 01:57:12 UTC

[jira] [Created] (MESOS-2605) The slave sometimes does not send active executors during reregistration

Elizabeth Lingg created MESOS-2605:
--------------------------------------

             Summary: The slave sometimes does not send active executors during reregistration
                 Key: MESOS-2605
                 URL: https://issues.apache.org/jira/browse/MESOS-2605
             Project: Mesos
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Elizabeth Lingg


Framework checkpointing is enabled, and the executor successfully reregisters. However, the tasks in that executor are LOST (by abnormal executor termination) because the executor is removed by the mesos master as unknown, task.journalnode.journalnode.NodeExecutor.1428610787834.

See the Slave Logs here for the Task:


Apr 09 19:53:06 ip-10-168-119-78.ec2.internal mesos-slave[25116]: I0409 19:53:06.778790 25126 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: 4eb22075-c319-463d-8f70-94db9caa69c6) for task task.journalnode.journalnode.NodeExecutor.1428609184051 of framework 20150408-002100-4261056010-5050-1047-0008
Apr 09 19:53:06 ip-10-168-119-78.ec2.internal mesos-slave[25116]: I0409 19:53:06.779013 25126 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 4eb22075-c319-463d-8f70-94db9caa69c6) for task task.journalnode.journalnode.NodeExecutor.1428609184051 of framework 20150408-002100-4261056010-5050-1047-0008
Apr 09 19:53:06 ip-10-168-119-78.ec2.internal mesos-slave[25116]: I0409 19:53:06.781788 25123 slave.cpp:2753] Forwarding the update TASK_RUNNING (UUID: 4eb22075-c319-463d-8f70-94db9caa69c6) for task task.journalnode.journalnode.NodeExecutor.1428609184051 of framework 20150408-002100-4261056010-5050-1047-0008 to master@10.142.250.253:5050
Apr 09 19:53:06 ip-10-168-119-78.ec2.internal mesos-slave[25116]: I0409 19:53:06.781889 25123 slave.cpp:2686] Sending acknowledgement for status update TASK_RUNNING (UUID: 4eb22075-c319-463d-8f70-94db9caa69c6) for task task.journalnode.journalnode.NodeExecutor.1428609184051 of framework 20150408-002100-4261056010-5050-1047-0008 to executor(1)@10.168.119.78:47638
Apr 09 19:53:06 ip-10-168-119-78.ec2.internal mesos-slave[25116]: I0409 19:53:06.784503 25124 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 4eb22075-c319-463d-8f70-94db9caa69c6) for task task.journalnode.journalnode.NodeExecutor.1428609184051 of framework 20150408-002100-4261056010-5050-1047-0008
Apr 09 19:53:06 ip-10-168-119-78.ec2.internal mesos-slave[25116]: I0409 19:53:06.784567 25124 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_RUNNING (UUID: 4eb22075-c319-463d-8f70-94db9caa69c6) for task task.journalnode.journalnode.NodeExecutor.1428609184051 of framework 20150408-002100-4261056010-5050-1047-0008

Master Logs:

2958, 2960-3887, 1025-2180, 8082-9041, 9043-9159, 9161-9999, 5052-6999, 7002-7198, 7200-8079, 10001-65535]) on slave 20150407-233647-2059219722-5050-1659-S5 from framework 20150408-002100-4261056010-5050-1047-0008
Apr 09 22:49:45 ip-10-142-250-253.ec2.internal mesos-master[1047]: W0409 22:49:45.153770  1068 master.cpp:4015] Executor executor.journalnode.NodeExecutor.1428610787834 of framework 20150408-002100-4261056010-5050-1047-0008 possibly unknown to the slave 20150407-233647-2059219722-5050-1659-S5 at slave(1)@10.168.119.78:5051 (ec2-54-237-57-237.compute-1.amazonaws.com)
Apr 09 22:49:45 ip-10-142-250-253.ec2.internal mesos-master[1047]: I0409 22:49:45.153801  1068 master.cpp:4714] Removing executor 'executor.journalnode.NodeExecutor.1428610787834' with resources cpus(*):0.1; mem(*):1536 of framework 20150408-002100-4261056010-5050-1047-0008 on slave 20150407-233647-2059219722-5050-1659-S5 at slave(1)@10.168.119.78:5051 (ec2-54-237-57-237.compute-1.amazonaws.com)


Status update TASK_LOST (UUID: 236f8a18-6a6b-48b9-978d-3557460c6bb4) for task task.journalnode.journalnode.NodeExecutor.1428610787834 of framework 20150408-002100-4261056010-5050-1047-0008 from slave 20150407-233647-2059219722-5050-1659-S5 at slave(1)@10.168.119.78:5051 (ec2-54-237-57-237.compute-1.amazonaws.com)

Apr 09 22:49:45 ip-10-142-250-253.ec2.internal mesos-master[1047]: E0409 22:49:45.162519  1072 master.cpp:4568] Ignoring out of order status update for task ct:1428619776000:0:mesos-slave-restart: (TASK_FAILED -> TASK_RUNNING) of framework 20150407-233647-2059219722-5050-1659-0001

Apr 09 22:49:45 ip-10-142-250-253.ec2.internal mesos-master[1047]: I0409 22:49:45.162549  1072 master.cpp:3295] Status update TASK_LOST (UUID: 236f8a18-6a6b-48b9-978d-3557460c6bb4) for task task.journalnode.journalnode.NodeExecutor.1428610787834 of framework 20150408-002100-4261056010-5050-1047-0008 from slave 20150407-233647-2059219722-5050-1659-S5 at slave(1)@10.168.119.78:5051 (ec2-54-237-57-237.compute-1.amazonaws.com)
Apr 09 22:49:45 ip-10-142-250-253.ec2.internal mesos-master[1047]: I0409 22:49:45.162561  1072 master.cpp:3336] Forwarding status update TASK_LOST (UUID: 236f8a18-6a6b-48b9-978d-3557460c6bb4) for task task.journalnode.journalnode.NodeExecutor.1428610787834 of framework 20150408-002100-4261056010-5050-1047-0008



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)