You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/06/30 09:54:25 UTC
[GitHub] [incubator-pegasus] empiredan opened a new issue, #1026: Fix heap-use-after-free error while percentile timer is still running after percentile has already been destructed
empiredan opened a new issue, #1026:
URL: https://github.com/apache/incubator-pegasus/issues/1026
`heap-use-after-free` error was found while percentile timer is executed (for complete error info, please see https://github.com/apache/incubator-pegasus/runs/7123305604?check_suite_focus=true#step:8:1770):
```
==5035==ERROR: AddressSanitizer: heap-use-after-free on address 0x6100000033d8 at pc 0x5631bbff27f1 bp 0x152a4f5e7900 sp 0x152a4f5e78f0
READ of size 8 at 0x6100000033d8 thread T131
#0 0x5631bbff27f0 in void dsn::stl_nth_element_finder<long, std::less<long> >::operator()<__gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > > >(__gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >) /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/nth_element.h:94
#1 0x5631bbff27f0 in dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::find_nth_elements() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/metrics.h:681
#2 0x5631bbfb5bae in void std::__invoke_impl<void, void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*&)(), dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*&>(std::__invoke_memfun_deref, void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*&)(), dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*&) /usr/include/c++/7/bits/invoke.h:73
#3 0x5631bbfb5bae in std::__invoke_result<void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*&)(), dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*&>::type std::__invoke<void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*&)(), dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*&>(void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*&)(), dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*&) /usr/include/c++/7/bits/invoke.h:95
#4 0x5631bbfb5bae in void std::_Bind<void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*(dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/7/functional:467
#5 0x5631bbfb5bae in void std::_Bind<void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*(dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*))()>::operator()<, void>() /usr/include/c++/7/functional:551
#6 0x5631bbfb5bae in std::_Function_handler<void (), std::_Bind<void (dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::*(dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>*))()> >::_M_invoke(std::_Any_data const&) /usr/include/c++/7/bits/std_function.h:316
#7 0x152a55900883 in std::function<void ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706
#8 0x152a55900883 in dsn::percentile_timer::on_timer(boost::system::error_code const&) /__w/incubator-pegasus/incubator-pegasus/rdsn/src/utils/metrics.cpp:145
#9 0x152a55924430 in void std::__invoke_impl<void, void (dsn::percentile_timer::*&)(boost::system::error_code const&), dsn::percentile_timer*&, boost::system::error_code const&>(std::__invoke_memfun_deref, void (dsn::percentile_timer::*&)(boost::system::error_code const&), dsn::percentile_timer*&, boost::system::error_code const&) /usr/include/c++/7/bits/invoke.h:73
#10 0x152a55924430 in std::__invoke_result<void (dsn::percentile_timer::*&)(boost::system::error_code const&), dsn::percentile_timer*&, boost::system::error_code const&>::type std::__invoke<void (dsn::percentile_timer::*&)(boost::system::error_code const&), dsn::percentile_timer*&, boost::system::error_code const&>(void (dsn::percentile_timer::*&)(boost::system::error_code const&), dsn::percentile_timer*&, boost::system::error_code const&) /usr/include/c++/7/bits/invoke.h:95
#11 0x152a55924430 in void std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>::__call<void, boost::system::error_code const&, 0ul, 1ul>(std::tuple<boost::system::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) /usr/include/c++/7/functional:467
#12 0x152a55924430 in void std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>::operator()<boost::system::error_code const&, void>(boost::system::error_code const&) /usr/include/c++/7/functional:551
#13 0x152a55924430 in boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code>::operator()() /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/bind_handler.hpp:65
#14 0x152a55924430 in void boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code> >(boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code>&, ...) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/handler_invoke_hook.hpp:69
#15 0x152a55924430 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code>, std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)> >(boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code>&, std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>&) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#16 0x152a55924430 in void boost::asio::detail::handler_work<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::asio::system_executor>::complete<boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code> >(boost::asio::detail::binder1<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>, boost::system::error_code>&, std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)>&) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/handler_work.hpp:82
#17 0x152a55924430 in boost::asio::detail::wait_handler<std::_Bind<void (dsn::percentile_timer::*(dsn::percentile_timer*, std::_Placeholder<1>))(boost::system::error_code const&)> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/wait_handler.hpp:72
#18 0x152a559543d7 in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/scheduler_operation.hpp:40
#19 0x152a559543d7 in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/scheduler.ipp:401
#20 0x152a559543d7 in boost::asio::detail::scheduler::run(boost::system::error_code&) /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/scheduler.ipp:154
#21 0x152a559543d7 in boost::asio::io_context::run() /__w/incubator-pegasus/incubator-pegasus/rdsn/thirdparty/output/include/boost/asio/impl/io_context.ipp:62
#22 0x152a559543d7 in operator() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/utils/shared_io_service.cpp:46
#23 0x152a559543d7 in __invoke_impl<void, dsn::tools::shared_io_service::shared_io_service()::<lambda()> > /usr/include/c++/7/bits/invoke.h:60
#24 0x152a559543d7 in __invoke<dsn::tools::shared_io_service::shared_io_service()::<lambda()> > /usr/include/c++/7/bits/invoke.h:95
#25 0x152a559543d7 in _M_invoke<0> /usr/include/c++/7/thread:234
#26 0x152a559543d7 in operator() /usr/include/c++/7/thread:243
#27 0x152a559543d7 in _M_run /usr/include/c++/7/thread:186
#28 0x152a54d0b6de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
#29 0x152a554cf6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#30 0x152a5476661e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12161e)
0x6100000033d8 is located 152 bytes inside of 192-byte region [0x610000003340,0x610000003400)
freed by thread T0 here:
#0 0x152a55dbc9c8 in operator delete(void*, unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe19c8)
#1 0x5631bbfc1b6e in dsn::percentile<long, dsn::stl_nth_element_finder<long, std::less<long> >, void>::~percentile() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/metrics.h:656
#2 0x5631bbf9c889 in dsn::ref_counter::release_ref() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/autoref_ptr.h:84
#3 0x5631bbf9c889 in dsn::ref_ptr<dsn::metric>::~ref_ptr() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/autoref_ptr.h:139
#4 0x5631bbf9c889 in std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >::~pair() /usr/include/c++/7/bits/stl_pair.h:208
#5 0x5631bbf9c889 in void __gnu_cxx::new_allocator<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > >::destroy<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > >(std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >*) /usr/include/c++/7/ext/new_allocator.h:140
#6 0x5631bbf9c889 in void std::allocator_traits<std::allocator<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > > >::destroy<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > >(std::allocator<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > >&, std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >*) /usr/include/c++/7/bits/alloc_traits.h:487
#7 0x5631bbf9c889 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >, false> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >, false>*) /usr/include/c++/7/bits/hashtable_policy.h:2084
#8 0x5631bbfbd353 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >, false>*) /usr/include/c++/7/bits/hashtable_policy.h:2097
#9 0x5631bbfbd353 in std::_Hashtable<dsn::metric_prototype const*, std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >, std::allocator<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > >, std::__detail::_Select1st, std::equal_to<dsn::metric_prototype const*>, std::hash<dsn::metric_prototype const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::clear() /usr/include/c++/7/bits/hashtable.h:2032
#10 0x152a558fbe6b in std::_Hashtable<dsn::metric_prototype const*, std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> >, std::allocator<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > >, std::__detail::_Select1st, std::equal_to<dsn::metric_prototype const*>, std::hash<dsn::metric_prototype const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::~_Hashtable() /usr/include/c++/7/bits/hashtable.h:1358
#11 0x152a558fbe6b in std::unordered_map<dsn::metric_prototype const*, dsn::ref_ptr<dsn::metric>, std::hash<dsn::metric_prototype const*>, std::equal_to<dsn::metric_prototype const*>, std::allocator<std::pair<dsn::metric_prototype const* const, dsn::ref_ptr<dsn::metric> > > >::~unordered_map() /usr/include/c++/7/bits/unordered_map.h:101
#12 0x152a558fbe6b in dsn::metric_entity::~metric_entity() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/utils/metrics.cpp:32
#13 0x152a558fbfe0 in dsn::metric_entity::~metric_entity() /__w/incubator-pegasus/incubator-pegasus/rdsn/src/utils/metrics.cpp:32
#14 0x5631bbfb9321 in dsn::ref_counter::release_ref() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/autoref_ptr.h:84
#15 0x5631bbfb9321 in dsn::ref_ptr<dsn::metric_entity>::~ref_ptr() /__w/incubator-pegasus/incubator-pegasus/rdsn/include/dsn/utility/autoref_ptr.h:139
#16 0x5631bbfb9321 in std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >::~pair() /usr/include/c++/7/bits/stl_pair.h:208
#17 0x5631bbfb9321 in void __gnu_cxx::new_allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >*) /usr/include/c++/7/ext/new_allocator.h:140
#18 0x5631bbfb9321 in void std::allocator_traits<std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > >(std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >*) /usr/include/c++/7/bits/alloc_traits.h:487
#19 0x5631bbfb9321 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >, true>*) /usr/include/c++/7/bits/hashtable_policy.h:2084
#20 0x5631bbfb9321 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >, true> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >, true>*) /usr/include/c++/7/bits/hashtable_policy.h:2097
#21 0x5631bbfb9321 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() /usr/include/c++/7/bits/hashtable.h:2032
#22 0x5631bbfb9321 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::ref_ptr<dsn::metric_entity> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable() /usr/include/c++/7/bits/hashtable.h:1358
#23 0x152a54688030 (/lib/x86_64-linux-gnu/libc.so.6+0x43030)
```
From the stack, we can draw a conclusion that percentile timer is still running after percentile has already been destructed. Once percentile is decided to be destructed, the timer should be cancelled firstly to ensure that there is not any member of percentile that is still be operated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] empiredan closed issue #1026: Fix heap-use-after-free error while percentile timer is still running after percentile has already been destructed
Posted by GitBox <gi...@apache.org>.
empiredan closed issue #1026: Fix heap-use-after-free error while percentile timer is still running after percentile has already been destructed
URL: https://github.com/apache/incubator-pegasus/issues/1026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org