You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Jerome Soussens (Jira)" <ji...@apache.org> on 2020/11/02 16:11:00 UTC
[jira] [Comment Edited] (MESOS-10194) Mesos master failure "Check
failed: 'get_(role)' Must be SOME"
[ https://issues.apache.org/jira/browse/MESOS-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224776#comment-17224776 ]
Jerome Soussens edited comment on MESOS-10194 at 11/2/20, 4:10 PM:
-------------------------------------------------------------------
Hi [~asekretenko],
I dont know if it's related but today we had this failure on master with Mesos 1.10.0 :
{code:java}
F1102 11:40:04.209203 6522 hierarchical.cpp:233] Check failed: scalars.at(slaveID) does not contain cpus(allocated: xxxxx):1; mem(allocated: xxxxx):15360
*** Check failure stack trace: ***
e06184040 with resources mem(allocated: stable.main):256 of framework b98761e9-2e84-4971-b678-13b6619b18e1 on agent 4bfe1c0d-aabc-45f4-98fe-3a5480058440-S0 at slave(1)@192.168.250.63:5051 (leta.sophiagenetics.com)
@ 0x7fd65a6fb94d google::LogMessage::SendToLog()
@ 0x7fd65a6f91fb google::LogMessage::Flush()
@ 0x7fd65a6fc3a9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fd659114217 mesos::internal::master::allocator::internal::ScalarResourceTotals::subtract()
@ 0x7fd659118202 mesos::internal::master::allocator::internal::RoleTree::untrackAllocated()
@ 0x7fd659129afa mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::recoverResources()
@ 0x7fd65a637081 process::ProcessBase::consume()
@ 0x7fd65a65ceb7 process::ProcessManager::resume()
@ 0x7fd65a660a76 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
@ 0x7fd65a916d80 execute_native_thread_routine
@ 0x7fd6569cee25 start_thread
@ 0x7fd6561dbbad __clone
mesos-master.service: main process exited, code=killed, status=6/ABRT
{code}
A more complete log : [^mesos_scalars_at_slaveId_crash.log]
was (Author: jerome soussens):
Hi [~asekretenko],
I dont know if it's related but today we had this failure on master with Mesos 1.10.0 :
{code:java}
F1102 11:40:04.209203 6522 hierarchical.cpp:233] Check failed: scalars.at(slaveID) does not contain cpus(allocated: xxxxx):1; mem(allocated: xxxxx):15360
*** Check failure stack trace: ***
e06184040 with resources mem(allocated: stable.main):256 of framework b98761e9-2e84-4971-b678-13b6619b18e1 on agent 4bfe1c0d-aabc-45f4-98fe-3a5480058440-S0 at slave(1)@192.168.250.63:5051 (leta.sophiagenetics.com)
@ 0x7fd65a6fb94d google::LogMessage::SendToLog()
@ 0x7fd65a6f91fb google::LogMessage::Flush()
@ 0x7fd65a6fc3a9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fd659114217 mesos::internal::master::allocator::internal::ScalarResourceTotals::subtract()
@ 0x7fd659118202 mesos::internal::master::allocator::internal::RoleTree::untrackAllocated()
@ 0x7fd659129afa mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::recoverResources()
@ 0x7fd65a637081 process::ProcessBase::consume()
@ 0x7fd65a65ceb7 process::ProcessManager::resume()
@ 0x7fd65a660a76 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
@ 0x7fd65a916d80 execute_native_thread_routine
@ 0x7fd6569cee25 start_thread
@ 0x7fd6561dbbad __clone
mesos-master.service: main process exited, code=killed, status=6/ABRT
{code}
A more complete log : [^mesos_scalars_at_slaveId_crash.log]
> Mesos master failure "Check failed: 'get_(role)' Must be SOME"
> --------------------------------------------------------------
>
> Key: MESOS-10194
> URL: https://issues.apache.org/jira/browse/MESOS-10194
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.10.0, 1.11.0
> Reporter: Jerome Soussens
> Assignee: Andrei Sekretenko
> Priority: Critical
> Attachments: log_mesos_crash_role_13102020.txt, mesos_scalars_at_slaveId_crash.log
>
>
>
> *Impact* : mesos-master crash with log :
> {code:java}
> hierarchical.cpp:460] Check failed: 'get_(role)' Must be SOME
> {code}
> *Possible scenario :*
> A framework, using a specific role, is stopped. More or less at the same time, some remaining task status for this framework comes to the master from the executor. But the roles is no more listed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)