You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Sergei Hanus (Jira)" <ji...@apache.org> on 2020/11/11 06:27:00 UTC
[jira] [Commented] (MESOS-9322) Executor exited accidentally, but
mesos-agent did not report TASK_FAILED event.
[ https://issues.apache.org/jira/browse/MESOS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229770#comment-17229770 ]
Sergei Hanus commented on MESOS-9322:
-------------------------------------
Hi! I'm experiencing the same issue as described in this ticket (but I found it too late and already opened another one - https://issues.apache.org/jira/browse/MESOS-10197.
It has logs attached, and I also tried to check this with the latest 1.10 release - still the same behavior. Agent does not report failed service state, even after I restart the agent. Only cleanup of the corresponding executor in meta folder helps to restore service back to functional.
> Executor exited accidentally, but mesos-agent did not report TASK_FAILED event.
> -------------------------------------------------------------------------------
>
> Key: MESOS-9322
> URL: https://issues.apache.org/jira/browse/MESOS-9322
> Project: Mesos
> Issue Type: Bug
> Components: agent
> Affects Versions: 1.4.1
> Environment: Linux n14-068-081 4.4.0-33.bm.1-amd64 #1 SMP Fri, 01 Sep 2017 18:36:21 +0800 x86_64 GNU/Linux
> OS: debion 8.10
> mesos version: 1.4.1
> Reporter: Shiwei Guo
> Priority: Major
>
> The log about this executor:
> executorid: 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz'
>
> {noformat}
> I0914 10:40:36.448287 2505 slave.cpp:7336] Recovering executor 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz' of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000
> I0914 10:40:36.479209 2511 gc.cpp:58] Scheduling '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8' for gc 3.1546935280563days in the future
> I0914 10:40:36.479287 2511 gc.cpp:58] Scheduling '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8' for gc 3.15469352761481days in the future
> I0914 10:40:36.479310 2511 gc.cpp:58] Scheduling '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85' for gc -1.02171850967407days in the future
> I0914 10:40:36.479337 2511 gc.cpp:58] Scheduling '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85' for gc -1.02171850987259days in the future
> I0914 10:40:36.480459 2514 gc.cpp:169] Deleting /opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85
> I0914 10:40:36.552492 2516 status_update_manager.cpp:211] Recovering executor 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz' of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000
> I0914 10:40:36.553234 2519 containerizer.cpp:665] Recovering container 106c7257-fabb-4d58-8fcb-89b15bb9d404 for executor 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz' of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000
> I0914 10:40:36.591421 2514 gc.cpp:177] Deleted '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85'
> I0914 10:40:36.705005 2534 gc.cpp:169] Deleting /opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85
> I0914 10:40:36.705201 2534 gc.cpp:177] Deleted '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/4b27d1d4-fe67-4475-88bc-14e994acfb85'
> I0914 10:40:36.764660 2514 slave.cpp:6180] Sending reconnect request to executor 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz' of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000 at executor(1)@10.14.68.81:26383
> I0914 10:40:38.765029 2533 slave.cpp:4346] Killing un-reregistered executor 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz' of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000 at executor(1)@10.14.68.81:26383
> I0916 06:09:40.390676 2530 gc.cpp:169] Deleting /opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8
> I0916 06:09:40.432540 2530 gc.cpp:177] Deleted '/opt/tiger/mesos_deploy/mesos_titan/slave/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8'
> I0916 06:09:40.432620 2530 gc.cpp:169] Deleting /opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8
> I0916 06:09:40.437860 2530 gc.cpp:177] Deleted '/opt/tiger/mesos_deploy/mesos_titan/slave/meta/slaves/03def54c-f3f0-4ea5-a886-93fae5e570fa-S3473/frameworks/ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000/executors/gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz/runs/189e4b23-c892-4c87-9069-dfc98ca5edc8'
> W1012 17:14:24.485033 2513 slave.cpp:3100] Ignoring kill task gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;ts:1536560605651 because the executor 'gn:aweme.recommend.cypher_recent.default;ps:aweme.recommend.cypher_recent_default;sg:263;tp:Companion;nm:aweme_cypher_recent;executor:systemd-mesos-executor-0.2.10.tar.gz' of framework ae7c9e78-e0b7-4110-8092-52baf64e4f67-0000 at executor(1)@10.14.68.81:26383 is terminating
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)