You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/06/29 16:11:59 UTC
[GitHub] [incubator-pegasus] acelyc111 opened a new issue, #1024: CI workflows often failed for some flaky tests
acelyc111 opened a new issue, #1024:
URL: https://github.com/apache/incubator-pegasus/issues/1024
1. ASAN failed test
```
[ RUN ] bulk_load_failover_test.app_info_inconsistency
=================================================================
==9760==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000c838b0 at pc 0x55d6c185e647 bp 0x1478bd5831b0 sp 0x1478bd5831a0
READ of size 8 at 0x602000c838b0 thread T25 (test_meta.THREA)
#0 0x55d6c185e646 in std::__detail::_Node_iterator_base<int, false>::_M_incr() /usr/include/c++/7/bits/hashtable_policy.h:314
#1 0x55d6c185e646 in std::__detail::_Node_iterator<int, true, false>::operator++() /usr/include/c++/7/bits/hashtable_policy.h:369
#2 0x55d6c185e646 in dsn::replication::bulk_load_service::try_to_continue_bulk_load() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:1704
#3 0x55d6c185ea96 in dsn::replication::bulk_load_service::initialize_bulk_load_service() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:64
#4 0x55d6c1389191 in dsn::replication::bulk_load_failover_test_app_info_inconsistency_Test::TestBody() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/test/meta_bulk_load_service_test.cpp:1315
#5 0x55d6c242a851 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x202a851)
#6 0x55d6c2424e86 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x2024e86)
#7 0x55d6c2409095 in testing::Test::Run() (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x2009095)
#8 0x55d6c24099bd in testing::TestInfo::Run() (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x20099bd)
#9 0x55d6c240a035 in testing::TestCase::Run() (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x200a035)
#10 0x55d6c2410ed9 in testing::internal::UnitTestImpl::RunAllTests() (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x2010ed9)
#11 0x55d6c242b968 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x202b968)
#12 0x55d6c2425c54 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x2025c54)
#13 0x55d6c240fabf in testing::UnitTest::Run() (/__w/incubator-pegasus/incubator-pegasus/rdsn/builder/src/meta/test/dsn.meta.test+0x200fabf)
#14 0x55d6c12a4537 in RUN_ALL_TESTS() /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/gtest/gtest.h:2233
#15 0x55d6c12a4537 in dsn::replication::meta_service_test_app::start(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/test/main.cpp:86
#16 0x55d6c208af14 in dsn::service_node::start_app() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_engine.cpp:74
#17 0x55d6c20b6fe3 in dsn::service_control_task::exec() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/tool_api.cpp:60
#18 0x55d6c21a824c in dsn::task::exec_internal() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task.cpp:176
#19 0x55d6c220aad5 in dsn::task_worker::loop() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_worker.cpp:224
#20 0x55d6c220b9ac in dsn::task_worker::run_internal() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_worker.cpp:204
#21 0x55d6c220bf43 in void std::__invoke_impl<void, void (dsn::task_worker::*&)(), dsn::task_worker*&>(std::__invoke_memfun_deref, void (dsn::task_worker::*&)(), dsn::task_worker*&) /usr/include/c++/7/bits/invoke.h:73
#22 0x55d6c220bf43 in std::__invoke_result<void (dsn::task_worker::*&)(), dsn::task_worker*&>::type std::__invoke<void (dsn::task_worker::*&)(), dsn::task_worker*&>(void (dsn::task_worker::*&)(), dsn::task_worker*&) /usr/include/c++/7/bits/invoke.h:95
#23 0x55d6c220bf43 in void std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/7/functional:467
#24 0x55d6c220bf43 in void std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>::operator()<, void>() /usr/include/c++/7/functional:551
#25 0x55d6c220bf43 in void std::__invoke_impl<void, std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>(std::__invoke_other, std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>&&) /usr/include/c++/7/bits/invoke.h:60
#26 0x55d6c220bf43 in std::__invoke_result<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>::type std::__invoke<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>(std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>&&) /usr/include/c++/7/bits/invoke.h:95
#27 0x55d6c220bf43 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/7/thread:234
#28 0x55d6c220bf43 in std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > >::operator()() /usr/include/c++/7/thread:243
#29 0x55d6c220bf43 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > > >::_M_run() /usr/include/c++/7/thread:186
#30 0x1478c6f9d6de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
#31 0x1478c7d556da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#32 0x1478c665a61e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12161e)
0x602000c838b0 is located 0 bytes inside of 16-byte region [0x602000c838b0,0x602000c838c0)
freed by thread T33 (test_meta.THREA) here:
#0 0x1478c95c32c0 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe12c0)
#1 0x55d6c1875481 in __gnu_cxx::new_allocator<std::__detail::_Hash_node<int, false> >::deallocate(std::__detail::_Hash_node<int, false>*, unsigned long) /usr/include/c++/7/ext/new_allocator.h:125
#2 0x55d6c1875481 in std::allocator_traits<std::allocator<std::__detail::_Hash_node<int, false> > >::deallocate(std::allocator<std::__detail::_Hash_node<int, false> >&, std::__detail::_Hash_node<int, false>*, unsigned long) /usr/include/c++/7/bits/alloc_traits.h:462
#3 0x55d6c1875481 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<int, false> > >::_M_deallocate_node(std::__detail::_Hash_node<int, false>*) /usr/include/c++/7/bits/hashtable_policy.h:2086
#4 0x55d6c1875481 in std::_Hashtable<int, int, std::allocator<int>, std::__detail::_Identity, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_erase(unsigned long, std::__detail::_Hash_node_base*, std::__detail::_Hash_node<int, false>*) /usr/include/c++/7/bits/hashtable.h:1890
#5 0x55d6c1875481 in std::_Hashtable<int, int, std::allocator<int>, std::__detail::_Identity, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_erase(std::integral_constant<bool, true>, int const&) /usr/include/c++/7/bits/hashtable.h:1916
#6 0x55d6c1875481 in std::_Hashtable<int, int, std::allocator<int>, std::__detail::_Identity, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::erase(int const&) /usr/include/c++/7/bits/hashtable.h:759
#7 0x55d6c1875481 in std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> >::erase(int const&) /usr/include/c++/7/bits/unordered_set.h:544
#8 0x55d6c1875481 in dsn::replication::bulk_load_service::reset_local_bulk_load_states_unlocked(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:1404
#9 0x55d6c1886b60 in dsn::replication::bulk_load_service::reset_local_bulk_load_states(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:1424
#10 0x55d6c1889f9a in operator() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:1372
#11 0x55d6c1889f9a in _M_invoke /usr/include/c++/7/bits/std_function.h:316
#12 0x55d6c1b14fea in std::function<void (dsn::error_code)>::operator()(dsn::error_code) const /usr/include/c++/7/bits/std_function.h:706
#13 0x55d6c1b14fea in decltype (((declval<std::function<void (dsn::error_code)>&>)())((declval<dsn::error_code>)())) dsn::absl::base_internal::Callable::Invoke<std::function<void (dsn::error_code)>&, dsn::error_code>(std::function<void (dsn::error_code)>&, dsn::error_code&&) /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/absl/base/internal/invoke.h:180
#14 0x55d6c1b14fea in decltype (dsn::absl::base_internal::Invoker<std::function<void (dsn::error_code)>&, dsn::error_code>::type::Invoke((declval<std::function<void (dsn::error_code)>&>)(), (declval<dsn::error_code>)())) dsn::absl::base_internal::Invoke<std::function<void (dsn::error_code)>&, dsn::error_code>(std::function<void (dsn::error_code)>&, dsn::error_code&&) /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/absl/base/internal/invoke.h:212
#15 0x55d6c1b14fea in decltype (Invoke((forward<std::function<void (dsn::error_code)>&>)({parm#1}), (get<0ul>)((forward<std::tuple<dsn::error_code> >)({parm#2})))) dsn::absl::utility_internal::apply_helper<std::function<void (dsn::error_code)>&, std::tuple<dsn::error_code>, 0ul>(std::function<void (dsn::error_code)>&, std::tuple<dsn::error_code>&&, dsn::absl::integer_sequence<unsigned long, 0ul>) /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/absl/utility/utility.h:154
#16 0x55d6c1b14fea in _ZN3dsn4absl5applyIRSt8functionIFvNS_10error_codeEEESt5tupleIJS3_EEEEDTcl12apply_helpercl7forwardIT_Efp_Ecl7forwardIT0_Efp0_EtlNS0_16utility_internal3GenImXsrSt10tuple_sizeINSt16remove_referenceISA_E4typeEE5valueEE4typeEEEEOS9_OSA_ /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/absl/utility/utility.h:198
#17 0x55d6c1b14fea in dsn::future_task<dsn::error_code>::exec() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/tool-api/task.h:392
#18 0x55d6c21a824c in dsn::task::exec_internal() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task.cpp:176
#19 0x55d6c220aad5 in dsn::task_worker::loop() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_worker.cpp:224
#20 0x55d6c220b9ac in dsn::task_worker::run_internal() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_worker.cpp:204
#21 0x55d6c220bf43 in void std::__invoke_impl<void, void (dsn::task_worker::*&)(), dsn::task_worker*&>(std::__invoke_memfun_deref, void (dsn::task_worker::*&)(), dsn::task_worker*&) /usr/include/c++/7/bits/invoke.h:73
#22 0x55d6c220bf43 in std::__invoke_result<void (dsn::task_worker::*&)(), dsn::task_worker*&>::type std::__invoke<void (dsn::task_worker::*&)(), dsn::task_worker*&>(void (dsn::task_worker::*&)(), dsn::task_worker*&) /usr/include/c++/7/bits/invoke.h:95
#23 0x55d6c220bf43 in void std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/7/functional:467
#24 0x55d6c220bf43 in void std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>::operator()<, void>() /usr/include/c++/7/functional:551
#25 0x55d6c220bf43 in void std::__invoke_impl<void, std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>(std::__invoke_other, std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>&&) /usr/include/c++/7/bits/invoke.h:60
#26 0x55d6c220bf43 in std::__invoke_result<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>::type std::__invoke<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>(std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>&&) /usr/include/c++/7/bits/invoke.h:95
#27 0x55d6c220bf43 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/7/thread:234
#28 0x55d6c220bf43 in std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > >::operator()() /usr/include/c++/7/thread:243
#29 0x55d6c220bf43 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > > >::_M_run() /usr/include/c++/7/thread:186
#30 0x1478c6f9d6de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
previously allocated by thread T33 (test_meta.THREA) here:
#0 0x1478c95c2448 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0448)
#1 0x55d6c1840b51 in __gnu_cxx::new_allocator<std::__detail::_Hash_node<int, false> >::allocate(unsigned long, void const*) /usr/include/c++/7/ext/new_allocator.h:111
#2 0x55d6c1840b51 in std::allocator_traits<std::allocator<std::__detail::_Hash_node<int, false> > >::allocate(std::allocator<std::__detail::_Hash_node<int, false> >&, unsigned long) /usr/include/c++/7/bits/alloc_traits.h:436
#3 0x55d6c1840b51 in std::__detail::_Hash_node<int, false>* std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<int, false> > >::_M_allocate_node<int const&>(int const&) /usr/include/c++/7/bits/hashtable_policy.h:2060
#4 0x55d6c1840b51 in std::__detail::_Hash_node<int, false>* std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<int, false> > >::operator()<int const&>(int const&) const /usr/include/c++/7/bits/hashtable_policy.h:182
#5 0x55d6c1840b51 in std::pair<std::__detail::_Node_iterator<int, true, false>, bool> std::_Hashtable<int, int, std::allocator<int>, std::__detail::_Identity, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert<int const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<int, false> > > >(int const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<int, false> > > const&, std::integral_constant<bool, true>) /usr/include/c++/7/bits/hashtable.h:1821
#6 0x55d6c1840b51 in std::__detail::_Insert_base<int, int, std::allocator<int>, std::__detail::_Identity, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::insert(int const&) /usr/include/c++/7/bits/hashtable_policy.h:843
#7 0x55d6c1840b51 in std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> >::insert(int const&) /usr/include/c++/7/bits/unordered_set.h:420
#8 0x55d6c1840b51 in operator() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:1660
#9 0x55d6c1af5a9e in std::function<void (dsn::error_code, dsn::blob const&)>::operator()(dsn::error_code, dsn::blob const&) const /usr/include/c++/7/bits/std_function.h:706
#10 0x55d6c1af5a9e in operator() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_state_service_simple.cpp:459
#11 0x55d6c1af5a9e in _M_invoke /usr/include/c++/7/bits/std_function.h:316
#12 0x55d6c21a824c in dsn::task::exec_internal() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task.cpp:176
#13 0x55d6c220aad5 in dsn::task_worker::loop() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_worker.cpp:224
#14 0x55d6c220b9ac in dsn::task_worker::run_internal() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_worker.cpp:204
#15 0x55d6c220bf43 in void std::__invoke_impl<void, void (dsn::task_worker::*&)(), dsn::task_worker*&>(std::__invoke_memfun_deref, void (dsn::task_worker::*&)(), dsn::task_worker*&) /usr/include/c++/7/bits/invoke.h:73
#16 0x55d6c220bf43 in std::__invoke_result<void (dsn::task_worker::*&)(), dsn::task_worker*&>::type std::__invoke<void (dsn::task_worker::*&)(), dsn::task_worker*&>(void (dsn::task_worker::*&)(), dsn::task_worker*&) /usr/include/c++/7/bits/invoke.h:95
#17 0x55d6c220bf43 in void std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/7/functional:467
#18 0x55d6c220bf43 in void std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>::operator()<, void>() /usr/include/c++/7/functional:551
#19 0x55d6c220bf43 in void std::__invoke_impl<void, std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>(std::__invoke_other, std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>&&) /usr/include/c++/7/bits/invoke.h:60
#20 0x55d6c220bf43 in std::__invoke_result<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>::type std::__invoke<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>>(std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()>&&) /usr/include/c++/7/bits/invoke.h:95
#21 0x55d6c220bf43 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/7/thread:234
#22 0x55d6c220bf43 in std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > >::operator()() /usr/include/c++/7/thread:243
#23 0x55d6c220bf43 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (dsn::task_worker::*(dsn::task_worker*))()> > > >::_M_run() /usr/include/c++/7/thread:186
#24 0x1478c6f9d6de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
Thread T25 (test_meta.THREA) created by T0 here:
#0 0x1478c9519d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
#1 0x1478c6f9d994 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd994)
#2 0x55d6c21c2f1b in dsn::task_worker_pool::start() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_engine.cpp:89
#3 0x55d6c21c6ae0 in dsn::task_engine::start() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_engine.cpp:239
#4 0x55d6c2093665 in dsn::service_node::start() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_engine.cpp:122
#5 0x55d6c2094d1e in dsn::service_engine::start_node(dsn::service_app_spec&) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_engine.cpp:247
#6 0x55d6c207ddb3 in run /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_api_c.cpp:511
#7 0x55d6c2082b63 in dsn_run_config(char const*, bool) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_api_c.cpp:184
#8 0x55d6c0ddf169 in main /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/test/main.cpp:99
#9 0x1478c655ac86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21c86)
Thread T33 (test_meta.THREA) created by T0 here:
#0 0x1478c9519d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
#1 0x1478c6f9d994 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd994)
#2 0x55d6c21c2f1b in dsn::task_worker_pool::start() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_engine.cpp:89
#3 0x55d6c21c6ae0 in dsn::task_engine::start() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/task/task_engine.cpp:239
#4 0x55d6c2093665 in dsn::service_node::start() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_engine.cpp:122
#5 0x55d6c2094d1e in dsn::service_engine::start_node(dsn::service_app_spec&) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_engine.cpp:247
#6 0x55d6c207ddb3 in run /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_api_c.cpp:511
#7 0x55d6c2082b63 in dsn_run_config(char const*, bool) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/runtime/service_api_c.cpp:184
#8 0x55d6c0ddf169 in main /__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/test/main.cpp:99
#9 0x1478c655ac86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21c86)
SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/7/bits/hashtable_policy.h:314 in std::__detail::_Node_iterator_base<int, false>::_M_incr()
Shadow bytes around the buggy address:
0x0c04801886c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04801886d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04801886e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04801886f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480188700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0480188710: fa fa fa fa fa fa[fd]fd fa fa fa fa fa fa fa fa
0x0c0480188720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480188730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480188740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480188750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480188760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==9760==ABORTING
ERROR: run dsn.meta.test failed, return_code = 1
Error: Process completed with exit code 1.
```
2. LSAN failed
```
[ RUN ] mutation_log_test.replay_single_file_1
E2022-06-29 12:20:50.265 (1656505250265889832 9838) replica.default0.0000265500010001: replica.cpp:569:init_disk_tag(): [1.1@] get disk tag of ./test-log failed: ERR_OBJECT_NOT_FOUND, init it to empty
E2022-06-29 12:20:50.266 (1656505250266339030 9838) replica.default0.0000265500010001: log_file.cpp:218:read_next_log_block(): invalid data header magic: 0x0
E2022-06-29 12:20:50.266 (1656505250266349930 9838) replica.default0.0000265500010001: log_file.cpp:102:open_read(): read first log entry of file ./test-log/log.1.0 failed, err = ERR_INVALID_DATA. Rename the file to ./test-log/log.1.0.removed
/__w/incubator-pegasus/incubator-pegasus/rdsn/src/replica/test/mutation_log_test.cpp:372: Failure
Expected: ec
Which is: ERR_INVALID_DATA
To be equal to: ERR_OK
ERR_INVALID_DATA
[ FAILED ] mutation_log_test.replay_single_file_1 (1 ms)
```
3. USAN failed
```
[ RUN ] bulk_load_failover_test.app_info_inconsistency
/usr/include/c++/7/bits/hashtable_policy.h:360:35: runtime error: member call on misaligned address 0x55415641e5894855 for type 'struct _Hash_node_value_base', which requires 8 byte alignment
0x55415641e5894855: note: pointer points here
<memory cannot be printed>
/usr/include/c++/7/bits/hashtable_policy.h:260:26: runtime error: member call on misaligned address 0x55415641e5894855 for type 'struct _Hash_node_value_base', which requires 8 byte alignment
0x55415641e5894855: note: pointer points here
<memory cannot be printed>
/usr/include/c++/7/bits/hashtable_policy.h:252:34: runtime error: member call on misaligned address 0x55415641e589485d for type 'struct __aligned_buffer', which requires 4 byte alignment
0x55415641e589485d: note: pointer points here
<memory cannot be printed>
/usr/include/c++/7/ext/aligned_buffer.h:110:41: runtime error: member call on misaligned address 0x55415641e589485d for type 'struct __aligned_buffer', which requires 4 byte alignment
0x55415641e589485d: note: pointer points here
<memory cannot be printed>
/usr/include/c++/7/bits/hashtable_policy.h:260:26: runtime error: reference binding to misaligned address 0x55415641e589485d for type 'int', which requires 4 byte alignment
0x55415641e589485d: note: pointer points here
<memory cannot be printed>
/usr/include/c++/7/bits/hashtable_policy.h:360:35: runtime error: reference binding to misaligned address 0x55415641e589485d for type 'const int', which requires 4 byte alignment
0x55415641e589485d: note: pointer points here
<memory cannot be printed>
/__w/incubator-pegasus/incubator-pegasus/rdsn/src/meta/meta_bulk_load_service.cpp:1704:30: runtime error: load of misaligned address 0x55415641e589485d for type 'const int', which requires 4 byte alignment
0x55415641e589485d: note: pointer points here
<memory cannot be printed>
E2022-06-29 12:55:14.398 (1656507314398709061 9783) test_meta.THREAD_POOL_DEFAULT0.0000261e00010001: meta_bulk_load_service.cpp:1798:validate_app(): app(bulk_load_failover_table) has different app_id or partition_count, bulk load app_id = 2, partition_count = 4, current app_id = 2, partition_count = 8
E2022-06-29 12:55:14.398 (1656507314398843462 9783) test_meta.THREAD_POOL_DEFAULT0.0000261e00010001: meta_bulk_load_service.cpp:1726:try_to_continue_app_bulk_load(): app(name=,app_id=0) is not existed or not available
E2022-06-29 12:55:14.398 (1656507314398902863 9783) test_meta.THREAD_POOL_DEFAULT0.0000261e00010001: meta_bulk_load_service.cpp:1726:try_to_continue_app_bulk_load(): app(name=,app_id=0) is not existed or not available
got signal id: 11
Segmentation fault (core dumped)
ERROR: run dsn.meta.test failed, return_code = 139
Error: Process completed with exit code 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 closed issue #1024: CI workflows often failed for some flaky tests
Posted by GitBox <gi...@apache.org>.
acelyc111 closed issue #1024: CI workflows often failed for some flaky tests
URL: https://github.com/apache/incubator-pegasus/issues/1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org