You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Gaurav Garg (JIRA)" <ji...@apache.org> on 2019/05/06 18:21:01 UTC
[jira] [Commented] (MESOS-9767) Add self health monitoring in Mesos
master
[ https://issues.apache.org/jira/browse/MESOS-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834088#comment-16834088 ]
Gaurav Garg commented on MESOS-9767:
------------------------------------
Stack trace of the Mesos master when the hang was detected. Captured using gdb.
Thread 35 (Thread 0x7fa1e7e5b700 (LWP 85875)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0ae768)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf0ae768) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf0ae768) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0ae750) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e7e5b700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 34 (Thread 0x7fa1e765a700 (LWP 85876)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11ff38)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf11ff38) at /usr/include/c++/4.9/functional:1700
---Type <return> to continue, or q <return> to quit---
#7 operator() (this=0x561dcf11ff38) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11ff20) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e765a700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 33 (Thread 0x7fa1e6e59700 (LWP 85877)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11d988)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf11d988) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf11d988) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11d970) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e6e59700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 32 (Thread 0x7fa1e6658700 (LWP 85878)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
---Type <return> to continue, or q <return> to quit---
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf128758)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf128758) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf128758) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf128740) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e6658700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 31 (Thread 0x7fa1e5e57700 (LWP 85879)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11ffc8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf11ffc8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf11ffc8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11ffb0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e5e57700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
---Type <return> to continue, or q <return> to quit---
Thread 30 (Thread 0x7fa1e5656700 (LWP 85880)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf118f18)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf118f18) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf118f18) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf118f00) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e5656700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 29 (Thread 0x7fa1e4e55700 (LWP 85881)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf128ea8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf128ea8) at /usr/include/c++/4.9/functional:1700
---Type <return> to continue, or q <return> to quit---
#7 operator() (this=0x561dcf128ea8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf128e90) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e4e55700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 28 (Thread 0x7fa1e4654700 (LWP 85882)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0145c8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf0145c8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf0145c8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0145b0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e4654700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 27 (Thread 0x7fa1e3e53700 (LWP 85883)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
---Type <return> to continue, or q <return> to quit---
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0026f8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf0026f8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf0026f8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0026e0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e3e53700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 26 (Thread 0x7fa1e3652700 (LWP 85884)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf119668)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf119668) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf119668) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf119650) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e3652700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
---Type <return> to continue, or q <return> to quit---
Thread 25 (Thread 0x7fa1e2e51700 (LWP 85885)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf1190c8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf1190c8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf1190c8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf1190b0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e2e51700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 24 (Thread 0x7fa1e2650700 (LWP 85886)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11d398)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf11d398) at /usr/include/c++/4.9/functional:1700
---Type <return> to continue, or q <return> to quit---
#7 operator() (this=0x561dcf11d398) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11d380) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e2650700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 23 (Thread 0x7fa1e1e4f700 (LWP 85887)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf121f58)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf121f58) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf121f58) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf121f40) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e1e4f700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 22 (Thread 0x7fa1e164e700 (LWP 85888)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
---Type <return> to continue, or q <return> to quit---
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf12d038)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf12d038) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf12d038) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf12d020) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e164e700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 21 (Thread 0x7fa1e0e4d700 (LWP 85889)):
#0 0x00007fa1f05f01c2 in hash_combine_impl (k=52, h=<synthetic pointer>)
at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:264
#1 hash_combine<char> (v=<optimized out>, seed=<synthetic pointer>)
at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:337
#2 hash_range<__gnu_cxx::__normal_iterator<char const*, std::basic_string<char> > > (last=...,
first=52 '4') at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:351
#3 hash_value<char, std::allocator<char> > (v=...)
at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:410
#4 operator() (this=<optimized out>, v=...)
at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:486
#5 boost::hash_combine<std::string> (seed=seed@entry=@0x7fa1e0e4c770: 0, v=...)
at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:337
#6 0x00007fa1f06ad178 in operator() (this=0x7fa1cc02d068, taskId=...)
at /mesos/include/mesos/type_utils.hpp:634
#7 _M_hash_code (this=0x7fa1cc02d068, __k=...) at /usr/include/c++/4.9/bits/hashtable_policy.h:1261
#8 std::_Hashtable<mesos::TaskID, std::pair<mesos::TaskID const, std::_List_iterator<std::pair<mesos---Type <return> to continue, or q <return> to quit---
::TaskID, process::Owned<mesos::Task> > > >, std::allocator<std::pair<mesos::TaskID const, std::_List_iterator<std::pair<mesos::TaskID, process::Owned<mesos::Task> > > > >, std::__detail::_Select1st, std::equal_to<mesos::TaskID>, std::hash<mesos::TaskID>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::count (this=this@entry=0x7fa1cc02d068, __k=...)
at /usr/include/c++/4.9/bits/hashtable.h:1336
#9 0x00007fa1f0663eb2 in count (__x=..., this=0x7fa1cc02d068)
at /usr/include/c++/4.9/bits/unordered_map.h:592
#10 contains (key=..., this=0x7fa1cc02d068) at /mesos/3rdparty/stout/include/stout/hashmap.hpp:88
#11 erase (key=..., this=0x7fa1cc02d050)
at /mesos/3rdparty/stout/include/stout/boundedhashmap.hpp:92
#12 mesos::internal::master::Master::__reregisterSlave(process::UPID const&, mesos::internal::ReregisterSlaveMessage&&, process::Future<bool> const&) (this=0x561dcf047380, pid=...,
reregisterSlaveMessage=<unknown type in /usr/local/lib/libmesos-1.6.0.so, CU 0x30075d6, DIE 0x38a83be>, future=...) at /mesos/src/master/master.cpp:7369
#13 0x00007fa1f14d54e1 in operator() (args#0=0x561dcf048620, this=<optimized out>)
at /mesos/3rdparty/libprocess/../stout/include/stout/lambda.hpp:443
#14 process::ProcessBase::consume(process::DispatchEvent&&) (this=<optimized out>,
event=<optimized out>) at /mesos/3rdparty/libprocess/src/process.cpp:3577
#15 0x00007fa1f14e89b2 in serve (
event=<unknown type in /usr/local/lib/libmesos-1.6.0.so, CU 0x14b8d81b, DIE 0x14e9f25d>,
this=0x561dcf048620) at /mesos/3rdparty/libprocess/include/process/process.hpp:87
#16 process::ProcessManager::resume (this=<optimized out>, process=0x561dcf048620)
at /mesos/3rdparty/libprocess/src/process.cpp:3002
#17 0x00007fa1f14ee226 in operator() (__closure=0x561dcf119158)
at /mesos/3rdparty/libprocess/src/process.cpp:2511
#18 _M_invoke<> (this=0x561dcf119158) at /usr/include/c++/4.9/functional:1700
#19 operator() (this=0x561dcf119158) at /usr/include/c++/4.9/functional:1688
#20 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf119140) at /usr/include/c++/4.9/thread:115
---Type <return> to continue, or q <return> to quit---
#21 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#22 0x00007fa1ee520064 in start_thread (arg=0x7fa1e0e4d700) at pthread_create.c:309
#23 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 20 (Thread 0x7fa1e064c700 (LWP 85890)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf123148)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf123148) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf123148) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf123130) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e064c700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 19 (Thread 0x7fa1dfe4b700 (LWP 85891)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
---Type <return> to continue, or q <return> to quit---
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe7e28)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcefe7e28) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcefe7e28) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe7e10) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dfe4b700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 18 (Thread 0x7fa1df64a700 (LWP 85892)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe76e8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcefe76e8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcefe76e8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe76d0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1df64a700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 17 (Thread 0x7fa1dee49700 (LWP 85893)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
---Type <return> to continue, or q <return> to quit---
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe81c8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcefe81c8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcefe81c8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe81b0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dee49700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 16 (Thread 0x7fa1de648700 (LWP 85894)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe7a88)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcefe7a88) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcefe7a88) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe7a70) at /usr/include/c++/4.9/thread:115
---Type <return> to continue, or q <return> to quit---
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1de648700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 15 (Thread 0x7fa1dde47700 (LWP 85895)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf061c58)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf061c58) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf061c58) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf061c40) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dde47700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 14 (Thread 0x7fa1dd646700 (LWP 85896)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
---Type <return> to continue, or q <return> to quit---
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf012fb8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf012fb8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf012fb8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf012fa0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dd646700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 13 (Thread 0x7fa1dce45700 (LWP 85897)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf058ce8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf058ce8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf058ce8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf058cd0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dce45700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 12 (Thread 0x7fa1dc644700 (LWP 85898)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
---Type <return> to continue, or q <return> to quit---
#1 0x00007fa1f14d6e82 in wait (this=<optimized out>)
at /mesos/3rdparty/libprocess/src/semaphore.hpp:115
#2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154
#3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73
#4 process::ProcessManager::dequeue (this=0x561dcf063970)
at /mesos/3rdparty/libprocess/src/process.cpp:3305
#5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0569d8)
at /mesos/3rdparty/libprocess/src/process.cpp:2505
#6 _M_invoke<> (this=0x561dcf0569d8) at /usr/include/c++/4.9/functional:1700
#7 operator() (this=0x561dcf0569d8) at /usr/include/c++/4.9/functional:1688
#8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0569c0) at /usr/include/c++/4.9/thread:115
#9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dc644700) at pthread_create.c:309
#11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 11 (Thread 0x7fa1dbe43700 (LWP 85899)):
#0 0x00007fa1ee255c03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fa1ecba7b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2 0x00007fa1ecb92d5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3 0x00007fa1f155fbed in process::EventLoop::run ()
at /mesos/3rdparty/libprocess/src/libevent.cpp:98
#4 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007fa1ee520064 in start_thread (arg=0x7fa1dbe43700) at pthread_create.c:309
#6 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 10 (Thread 0x7fa1db2da700 (LWP 85900)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007fa1f15c4a03 in BGThread (this=0x561dcf0608e0) at util/env_posix.cc:573
---Type <return> to continue, or q <return> to quit---
#2 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x561dcf0608e0)
at util/env_posix.cc:520
#3 0x00007fa1ee520064 in start_thread (arg=0x7fa1db2da700) at pthread_create.c:309
#4 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 9 (Thread 0x7fa1d8ad5700 (LWP 97884)):
#0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fa1f17070cb in do_io (v=0x7fa15c033390) at src/mt_adaptor.c:386
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d8ad5700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 8 (Thread 0x7fa17affd700 (LWP 97885)):
#0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fa1f17070cb in do_io (v=0x7fa17c004050) at src/mt_adaptor.c:386
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa17affd700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 7 (Thread 0x7fa17b7fe700 (LWP 97886)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007fa1f1707274 in do_completion (v=0x7fa15c033390) at src/mt_adaptor.c:462
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa17b7fe700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 6 (Thread 0x7fa1d9ad7700 (LWP 97887)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007fa1f1707274 in do_completion (v=0x7fa17c004050) at src/mt_adaptor.c:462
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d9ad7700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
---Type <return> to continue, or q <return> to quit---
Thread 5 (Thread 0x7fa1d92d6700 (LWP 97888)):
#0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fa1f17070cb in do_io (v=0x7fa1b000e0d0) at src/mt_adaptor.c:386
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d92d6700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 4 (Thread 0x7fa17bfff700 (LWP 97889)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007fa1f1707274 in do_completion (v=0x7fa1b000e0d0) at src/mt_adaptor.c:462
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa17bfff700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 3 (Thread 0x7fa1daad9700 (LWP 97890)):
#0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fa1f17070cb in do_io (v=0x7fa1c800c370) at src/mt_adaptor.c:386
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa1daad9700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 2 (Thread 0x7fa1da2d8700 (LWP 97891)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007fa1f1707274 in do_completion (v=0x7fa1c800c370) at src/mt_adaptor.c:462
#2 0x00007fa1ee520064 in start_thread (arg=0x7fa1da2d8700) at pthread_create.c:309
#3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 1 (Thread 0x7fa1f2718800 (LWP 85829)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
---Type <return> to continue, or q <return> to quit---
#1 0x00007fa1ee7e7c9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007fa1f14e8f80 in synchronized_wait<std::condition_variable, std::mutex> (
mutex=0x561dcf0489d8, cv=0x561dcf048a00)
at /mesos/3rdparty/libprocess/../stout/include/stout/synchronized.hpp:201
#3 wait (this=0x561dcf0489d0) at /mesos/3rdparty/libprocess/src/gate.hpp:50
#4 process::ProcessManager::wait (this=<optimized out>, pid=...)
at /mesos/3rdparty/libprocess/src/process.cpp:3227
#5 0x00007fa1f14ebf17 in process::wait (pid=..., duration=...)
at /mesos/3rdparty/libprocess/src/process.cpp:3968
#6 0x0000561dcddc05c9 in main (argc=<optimized out>, argv=<optimized out>)
at /mesos/src/master/main.cpp:546
> Add self health monitoring in Mesos master
> ------------------------------------------
>
> Key: MESOS-9767
> URL: https://issues.apache.org/jira/browse/MESOS-9767
> Project: Mesos
> Issue Type: Task
> Components: master
> Affects Versions: 1.6.0
> Reporter: Gaurav Garg
> Priority: Major
> Fix For: 1.7.2
>
>
> We have seen issue where Mesos master got stuck and was not responding to HTTP endpoints like "/metrics/snapshot". This results in calls by the frameworks and metrics collector to the master to hang. Currently we emit 'master alive' metric using prometheus. If master hangs, this metrics is not published and we detect the hangs using alerts on top of this metrics. By the time someone would have got the alert and restarted the master process, 15-30mins would have passed by. This results in SLA violation by Mesos cluster users.
> It will be nice to implement a self health check monitoring to detect if the Mesos master is hung/stuck. This will help us to quickly crash the master process so that one of the other member of the quorum can acquire ZK leadership lock.
> We can use the "/master/health" endpoint for health checks.
> Health checks can be initiated in [src/master/main.cpp|[https://github.com/apache/mesos/blob/master/src/master/main.cpp]] just after the child master process is [spawned.|[https://github.com/apache/mesos/blob/master/src/master/main.cpp#L543]]
> We can leverage the [HealthChecker|[https://github.com/apache/mesos/blob/master/src/checks/health_checker.hpp]] for this one. One downside is that HealthChecker currently takes TaskId as an input which is not valid for master health check.
> We can add following flags to control the self heath checking:
> # self_monitoring_enabled: Whether self monitoring is enabled.
> # self_monitoring_consecutive_failures: After this many number of health failures, master is crashed.
> # self_monitoring_interval_secs: Interval at which health checks are performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)