You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2020/12/09 05:33:23 UTC
[kudu-CR] [client] make metacache reset safe
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16839
Change subject: [client] make metacache reset safe
......................................................................
[client] make metacache reset safe
I noticed that the newly added TxnManagerTest.BeginManyTransactions
test scenario started failing with ASAN heap-use-after-free warnings.
After looking a that, it turned out that the original code was assuming
the cache wouldn't be ever reset before calling the MetaCache's
destructor. However, changelist 232474a51 introduced a new method
MetaCache::ClearCache() and since then the method is being called upon
altering a table if the partitioning scheme has been updated.
This patch resolves the issue by introducing so-called tablet server
registry that's never reset indeed, where entries in the tablet server
cache are just references to the entries in the registry (they are
raw pointers, actually).
The newly added test scenario was reliably producing AddressSanitizer's
heap-use-after-free warnings every time I ran it using ASAN build.
Below is a snapshot of the relevant traces captured when running the
new test scenario without the changes in the client metacache.
AddressSanitizer: heap-use-after-free on address 0x608000129e20 at pc 0x00000078bd54 bp 0x7fa731d0b240 sp 0x7fa731d0b238
READ of size 4 at 0x608000129e20 thread T149 (rpc reactor-146)
#0 0x78bd53 in base::subtle::NoBarrier_Load(int const volatile*) src/kudu/gutil/atomicops-internals-x86.h:200:10
#1 0x7fa79520e227 in base::SpinLock::SpinLoop(long, int*) src/kudu/gutil/spinlock.cc:86:10
#2 0x7fa79520e38b in base::SpinLock::SlowLock() src/kudu/gutil/spinlock.cc:104:25
#3 0x7fa7a099aab0 in std::unique_lock<kudu::simple_spinlock>::lock() ../../../include/c++/8/bits/std_mutex.h:267:17
#4 0x7fa7a0991e3e in std::unique_lock<kudu::simple_spinlock>::unique_lock(kudu::simple_spinlock&) ../../../include/c++/8/bits/std_mutex.h:197:2
#5 0x7fa7a0abfda1 in kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, std::function<void (kudu::Status const&)> const&) src/kudu/client/meta_cache.cc:145:39
#6 0x7fa7a0ac60f5 in kudu::client::internal::MetaCacheServerPicker::PickLeader(std::function<void (kudu::Status const&, kudu::client::internal::RemoteTabletServer*)> const&, kudu::MonoTime const&) src/kudu/client/meta_cache.cc:524:11
#7 0x7fa7a09b2dcf in kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer, kudu::tserver::WriteRequestPB, kudu::tserver::WriteResponsePB>::SendRpc() src/kudu/rpc/retriable_rpc.h:163:19
#8 0x7fa7a09ac6cc in kudu::client::internal::Batcher::FlushBuffer(kudu::client::internal::RemoteTablet*, std::vector<kudu::client::internal::InFlightOp*, std::allocator<kudu::client::internal::InFlightOp*> > const&) src/kudu/client/batcher.cc:911:8
#9 0x7fa7a09a9e38 in kudu::client::internal::Batcher::FlushBuffersIfReady() src/kudu/client/batcher.cc:884:5
#10 0x7fa7a09abd2d in kudu::client::internal::Batcher::TabletLookupFinished(kudu::client::internal::InFlightOp*, kudu::Status const&) src/kudu/client/batcher.cc:851:3
#11 0x7fa7a0acec66 in kudu::client::internal::LookupRpc::SendRpcCb(kudu::Status const&) src/kudu/client/meta_cache.cc:923:3
#12 0x7fa7a0ab8800 in kudu::client::internal::AsyncLeaderMasterRpc<kudu::master::GetTableLocationsRequestPB, kudu::master::GetTableLocationsResponsePB>::SendRpc()::'lambda'()::operator()() const src/kudu/client/master_proxy_rpc.cc:130:26
0x608000129e20 is located 0 bytes inside of 96-byte region [0x608000129e20,0x608000129e80)
freed by thread T163 here:
#0 0x65c650 in operator delete(void*) thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:160:399
#1 0x7fa7a0aec859 in void STLDeleteContainerPairSecondPointers<std::__detail::_Node_iterator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, false, true> >(std::__detail::_Node_iterator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, false, true>, std::__detail::_Node_iterator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, false, true>) src/kudu/gutil/stl_util.h:199:5
#2 0x7fa7a0ad96d1 in void STLDeleteValues<std::unordered_map<std::string, kudu::client::internal::RemoteTabletServer*, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*> > > >(std::unordered_map<std::string, kudu::client::internal::RemoteTabletServer*, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*> > >*) src/kudu/gutil/stl_util.h:400:3
#3 0x7fa7a0ad4188 in kudu::client::internal::MetaCache::ClearCache() src/kudu/client/meta_cache.cc:1257:3
previously allocated by thread T149 (rpc reactor-146) here:
#0 0x65bc98 in operator new(unsigned long) thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:99:386
#1 0x7fa7a0ac7bcb in kudu::client::internal::MetaCache::UpdateTabletServerUnlocked(kudu::master::TSInfoPB const&) src/kudu/client/meta_cache.cc:596:48
#2 0x7fa7a0ad0802 in kudu::client::internal::MetaCache::ProcessGetTableLocationsResponse(kudu::client::KuduTable const*, std::string const&, bool, kudu::master::GetTableLocationsResponsePB const&, kudu::client::internal::MetaCacheEntry*, int) src/kudu/client/meta_cache.cc:1030:7
#3 0x7fa7a0acf9c0 in kudu::client::internal::MetaCache::ProcessLookupResponse(kudu::client::internal::LookupRpc const&, kudu::client::internal::MetaCacheEntry*, int) src/kudu/client/meta_cache.cc:941:10
#4 0x7fa7a0ace64e in kudu::client::internal::LookupRpc::SendRpcCb(kudu::Status const&) src/kudu/client/meta_cache.cc:911:31
#5 0x7fa7a0ab8800 in kudu::client::internal::AsyncLeaderMasterRpc<kudu::master::GetTableLocationsRequestPB, kudu::master::GetTableLocationsResponsePB>::SendRpc()::'lambda'()::operator()() const src/kudu/client/master_proxy_rpc.cc:130:26
#6 0x7fa79b2c1620 in kudu::rpc::OutboundCall::CallCallback() src/kudu/rpc/outbound_call.cc:274:5
#7 0x7fa79b2c1af0 in kudu::rpc::OutboundCall::SetResponse(std::unique_ptr<kudu::rpc::CallResponse, std::default_delete<kudu::rpc::CallResponse> >) src/kudu/rpc/outbound_call.cc:306:5
#8 0x7fa79b26ef5e in kudu::rpc::Connection::HandleCallResponse(std::unique_ptr<kudu::rpc::InboundTransfer, std::default_delete<kudu::rpc::InboundTransfer> >) src/kudu/rpc/connection.cc:735:14
#9 0x7fa79b26e0d6 in kudu::rpc::Connection::ReadHandler(ev::io&, int) src/kudu/rpc/connection.cc:673:7
SUMMARY: AddressSanitizer: heap-use-after-free src/kudu/gutil/atomicops-internals-x86.h:200:10 in base::subtle::NoBarrier_Load(int const volatile*)
Shadow bytes around the buggy address:
0x0c108001d370: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d380: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d390: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d3a0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d3b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c108001d3c0: fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d3d0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d3e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d3f0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d400: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d410: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
---
M src/kudu/client/client-test.cc
M src/kudu/client/client.h
M src/kudu/client/meta_cache.cc
M src/kudu/client/meta_cache.h
4 files changed, 65 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/39/16839/1
--
To view, visit http://gerrit.cloudera.org:8080/16839
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Gerrit-Change-Number: 16839
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] [client] make metacache reset safe
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16839 )
Change subject: [client] make metacache reset safe
......................................................................
Patch Set 1:
(1 comment)
> Thanks for finding and fixing this!
Thank you for fast review!
http://gerrit.cloudera.org:8080/#/c/16839/1//COMMIT_MSG
Commit Message:
PS1:
I also verified that the TxnManagerTest.BeginManyTransactions scenario no longer fails with similar warning once this fix applied.
Before:
http://dist-test.cloudera.org//job?job_id=aserbin.1607497251.54221
After:
http://dist-test.cloudera.org//job?job_id=aserbin.1607497958.62002
The relevant part of the ASAN reports for TxnManagerTest.BeginManyTransactions before the fix was:
AddressSanitizer: heap-use-after-free on address 0x606000569fd8 at pc 0x0000004d6d8a bp 0x7f83c62f27d0 sp 0x7f83c62f1f80
READ of size 1342177400 at 0x606000569fd8 thread T168 (rpc reactor-136)
#0 0x4d6d89 in __asan_memcpy thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3273
#1 0x52f96d in std::char_traits<char>::copy(char*, char const*, unsigned long) ../../../include/c++/8/bits/char_traits.h:352:33
#2 0x52f6eb in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) ../../../include/c++/8/bits/basic_string.tcc:1081:2
#3 0x52f50d in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) ../../../include/c++/8/bits/basic_string.tcc:611:35
#4 0x7f8429d95347 in kudu::client::internal::RemoteTabletServer::ToString() const src/kudu/client/meta_cache.cc:225:16
#5 0x7f842ace83db in kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer, kudu::tserver::CoordinateTransactionRequestPB, kudu::tserver::CoordinateTransactionResponsePB>::SendRpcCb(kudu::Status const&) src/kudu/rpc/retriable_rpc.h:323:85
#6 0x7f842acf2400 in kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer, kudu::tserver::CoordinateTransactionRequestPB, kudu::tserver::CoordinateTransactionResponsePB>::ReplicaFoundCb(kudu::Status const&, kudu::client::internal::RemoteTabletServer*)::'lambda'()::operator()() const src/kudu/rpc/retriable_rpc.h:307:32
#7 0x7f8421245a30 in kudu::rpc::OutboundCall::CallCallback() src/kudu/rpc/outbound_call.cc:274:5
#8 0x7f8421245f00 in kudu::rpc::OutboundCall::SetResponse(std::unique_ptr<kudu::rpc::CallResponse, std::default_delete<kudu::rpc::CallResponse> >) src/kudu/rpc/outbound_call.cc:306:5
#9 0x7f84211f336e in kudu::rpc::Connection::HandleCallResponse(std::unique_ptr<kudu::rpc::InboundTransfer, std::default_delete<kudu::rpc::InboundTransfer> >) src/kudu/rpc/connection.cc:735:14
0x606000569ff9 is located 0 bytes to the right of 57-byte region [0x606000569fc0,0x606000569ff9)
freed by thread T70 (rpc worker-1265) here:
#0 0x517650 in operator delete(void*) thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:160:399
#1 0x531aee in std::string::_Rep::_M_destroy(std::allocator<char> const&) ../../../include/c++/8/bits/basic_string.tcc:899:29
#2 0x529dfe in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() ../../../include/c++/8/bits/basic_string.h:3640:19
#3 0x7f8429db5bd3 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, true>*) ../../../include/c++/8/bits/hashtable_policy.h:2100:7
#4 0x7f8429db5ae5 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, true> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, true>*) ../../../include/c++/8/bits/hashtable_policy.h:2113:4
#5 0x7f8429db5961 in std::_Hashtable<std::string, std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, std::allocator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() ../../../include/c++/8/bits/hashtable.h:2050:13
#6 0x7f8429da8188 in kudu::client::internal::MetaCache::ClearCache() src/kudu/client/meta_cache.cc:1257:3
#7 0x7f8429cb408d in kudu::client::KuduTableAlterer::Alter() src/kudu/client/client.cc:1487:41
#8 0x7f842ad3679e in kudu::transactions::TxnSystemClient::AddTxnStatusTableRangeWithClient(long, long, kudu::client::KuduClient*) src/kudu/transactions/txn_system_client.cc:119:9
#9 0x7f842f0b87e0 in kudu::transactions::TxnSystemClient::AddTxnStatusTableRange(long, long) src/kudu/transactions/txn_system_client.h:75:12
#10 0x7f842f0b628a in kudu::transactions::TxnManager::BeginTransaction(std::string const&, kudu::MonoTime const&, long*, unsigned int*) src/kudu/master/txn_manager.cc:187:48
#11 0x7f842f0b999d in kudu::transactions::TxnManagerServiceImpl::BeginTransaction(kudu::transactions::BeginTransactionRequestPB const*, kudu::transactions::BeginTransactionResponsePB*, kudu::rpc::RpcContext*) src/kudu/master/txn_manager_service.cc:75:42
previously allocated by thread T166 (rpc reactor-136) here:
#0 0x516c98 in operator new(unsigned long) thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:99:386
#1 0x52f7a8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) ../../../include/c++/8/bits/basic_string.tcc:1057:49
#2 0x5328d2 in std::string::_M_mutate(unsigned long, unsigned long, unsigned long) ../../../include/c++/8/bits/basic_string.tcc:929:16
#3 0x53281e in std::string::_M_replace_safe(unsigned long, unsigned long, char const*, unsigned long) ../../../include/c++/8/bits/basic_string.tcc:1134:7
#4 0x7f841c621af8 in google::protobuf::internal::EpsCopyInputStream::ReadString(char const*, int, std::string*) thirdparty/src/protobuf-3.14.0/src/google/protobuf/parse_context.h:156:26
#5 0x7f841c621af8 in google::protobuf::internal::InlineGreedyStringParser(std::string*, char const*, google::protobuf::internal::ParseContext*) thirdparty/src/protobuf-3.14.0/src/google/protobuf/parse_context.cc:441:25
#6 0x7f84292a9a7a in kudu::master::TSInfoPB::_InternalParse(char const*, google::protobuf::internal::ParseContext*) build/coordinator-retry.asan/src/kudu/master/master.pb.cc:10257:17
#7 0x7f842937253e in char const* google::protobuf::internal::ParseContext::ParseMessage<kudu::master::TSInfoPB>(kudu::master::TSInfoPB*, char const*) thirdparty/installed/uninstrumented/include/google/protobuf/parse_context.h:647:14
#8 0x7f84292e0693 in kudu::master::GetTableLocationsResponsePB::_InternalParse(char const*, google::protobuf::internal::ParseContext*) build/coordinator-retry.asan/src/kudu/master/master.pb.cc:15172:24
SUMMARY: AddressSanitizer: heap-use-after-free thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3273 in __asan_memcpy
Shadow bytes around the buggy address:
0x0c0c800a53a0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
0x0c0c800a53b0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
0x0c0c800a53c0: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c0c800a53d0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
0x0c0c800a53e0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
=>0x0c0c800a53f0: fd fd fd fd fa fa fa fa fd fd fd[fd]fd fd fd fd
0x0c0c800a5400: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
0x0c0c800a5410: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
0x0c0c800a5420: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0c800a5430: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
0x0c0c800a5440: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
--
To view, visit http://gerrit.cloudera.org:8080/16839
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Gerrit-Change-Number: 16839
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 09 Dec 2020 07:23:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [client] make metacache reset safe
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16839 )
Change subject: [client] make metacache reset safe
......................................................................
[client] make metacache reset safe
I noticed that the newly added TxnManagerTest.BeginManyTransactions
test scenario started failing with ASAN heap-use-after-free warnings.
After looking a that, it turned out that the original code was assuming
the cache wouldn't be ever reset before calling the MetaCache's
destructor. However, changelist 232474a51 introduced a new method
MetaCache::ClearCache() and since then the method is being called upon
altering a table if the partitioning scheme has been updated.
This patch resolves the issue by introducing so-called tablet server
registry that's never reset indeed, where entries in the tablet server
cache are just references to the entries in the registry (they are
raw pointers, actually).
The newly added test scenario was reliably producing AddressSanitizer's
heap-use-after-free warnings every time I ran it using ASAN build.
Below is a snapshot of the relevant traces captured when running the
new test scenario without the changes in the client metacache.
AddressSanitizer: heap-use-after-free on address 0x608000129e20 at pc 0x00000078bd54 bp 0x7fa731d0b240 sp 0x7fa731d0b238
READ of size 4 at 0x608000129e20 thread T149 (rpc reactor-146)
#0 0x78bd53 in base::subtle::NoBarrier_Load(int const volatile*) src/kudu/gutil/atomicops-internals-x86.h:200:10
#1 0x7fa79520e227 in base::SpinLock::SpinLoop(long, int*) src/kudu/gutil/spinlock.cc:86:10
#2 0x7fa79520e38b in base::SpinLock::SlowLock() src/kudu/gutil/spinlock.cc:104:25
#3 0x7fa7a099aab0 in std::unique_lock<kudu::simple_spinlock>::lock() ../../../include/c++/8/bits/std_mutex.h:267:17
#4 0x7fa7a0991e3e in std::unique_lock<kudu::simple_spinlock>::unique_lock(kudu::simple_spinlock&) ../../../include/c++/8/bits/std_mutex.h:197:2
#5 0x7fa7a0abfda1 in kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, std::function<void (kudu::Status const&)> const&) src/kudu/client/meta_cache.cc:145:39
#6 0x7fa7a0ac60f5 in kudu::client::internal::MetaCacheServerPicker::PickLeader(std::function<void (kudu::Status const&, kudu::client::internal::RemoteTabletServer*)> const&, kudu::MonoTime const&) src/kudu/client/meta_cache.cc:524:11
#7 0x7fa7a09b2dcf in kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer, kudu::tserver::WriteRequestPB, kudu::tserver::WriteResponsePB>::SendRpc() src/kudu/rpc/retriable_rpc.h:163:19
#8 0x7fa7a09ac6cc in kudu::client::internal::Batcher::FlushBuffer(kudu::client::internal::RemoteTablet*, std::vector<kudu::client::internal::InFlightOp*, std::allocator<kudu::client::internal::InFlightOp*> > const&) src/kudu/client/batcher.cc:911:8
#9 0x7fa7a09a9e38 in kudu::client::internal::Batcher::FlushBuffersIfReady() src/kudu/client/batcher.cc:884:5
#10 0x7fa7a09abd2d in kudu::client::internal::Batcher::TabletLookupFinished(kudu::client::internal::InFlightOp*, kudu::Status const&) src/kudu/client/batcher.cc:851:3
#11 0x7fa7a0acec66 in kudu::client::internal::LookupRpc::SendRpcCb(kudu::Status const&) src/kudu/client/meta_cache.cc:923:3
#12 0x7fa7a0ab8800 in kudu::client::internal::AsyncLeaderMasterRpc<kudu::master::GetTableLocationsRequestPB, kudu::master::GetTableLocationsResponsePB>::SendRpc()::'lambda'()::operator()() const src/kudu/client/master_proxy_rpc.cc:130:26
0x608000129e20 is located 0 bytes inside of 96-byte region [0x608000129e20,0x608000129e80)
freed by thread T163 here:
#0 0x65c650 in operator delete(void*) thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:160:399
#1 0x7fa7a0aec859 in void STLDeleteContainerPairSecondPointers<std::__detail::_Node_iterator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, false, true> >(std::__detail::_Node_iterator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, false, true>, std::__detail::_Node_iterator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*>, false, true>) src/kudu/gutil/stl_util.h:199:5
#2 0x7fa7a0ad96d1 in void STLDeleteValues<std::unordered_map<std::string, kudu::client::internal::RemoteTabletServer*, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*> > > >(std::unordered_map<std::string, kudu::client::internal::RemoteTabletServer*, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, kudu::client::internal::RemoteTabletServer*> > >*) src/kudu/gutil/stl_util.h:400:3
#3 0x7fa7a0ad4188 in kudu::client::internal::MetaCache::ClearCache() src/kudu/client/meta_cache.cc:1257:3
previously allocated by thread T149 (rpc reactor-146) here:
#0 0x65bc98 in operator new(unsigned long) thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:99:386
#1 0x7fa7a0ac7bcb in kudu::client::internal::MetaCache::UpdateTabletServerUnlocked(kudu::master::TSInfoPB const&) src/kudu/client/meta_cache.cc:596:48
#2 0x7fa7a0ad0802 in kudu::client::internal::MetaCache::ProcessGetTableLocationsResponse(kudu::client::KuduTable const*, std::string const&, bool, kudu::master::GetTableLocationsResponsePB const&, kudu::client::internal::MetaCacheEntry*, int) src/kudu/client/meta_cache.cc:1030:7
#3 0x7fa7a0acf9c0 in kudu::client::internal::MetaCache::ProcessLookupResponse(kudu::client::internal::LookupRpc const&, kudu::client::internal::MetaCacheEntry*, int) src/kudu/client/meta_cache.cc:941:10
#4 0x7fa7a0ace64e in kudu::client::internal::LookupRpc::SendRpcCb(kudu::Status const&) src/kudu/client/meta_cache.cc:911:31
#5 0x7fa7a0ab8800 in kudu::client::internal::AsyncLeaderMasterRpc<kudu::master::GetTableLocationsRequestPB, kudu::master::GetTableLocationsResponsePB>::SendRpc()::'lambda'()::operator()() const src/kudu/client/master_proxy_rpc.cc:130:26
#6 0x7fa79b2c1620 in kudu::rpc::OutboundCall::CallCallback() src/kudu/rpc/outbound_call.cc:274:5
#7 0x7fa79b2c1af0 in kudu::rpc::OutboundCall::SetResponse(std::unique_ptr<kudu::rpc::CallResponse, std::default_delete<kudu::rpc::CallResponse> >) src/kudu/rpc/outbound_call.cc:306:5
#8 0x7fa79b26ef5e in kudu::rpc::Connection::HandleCallResponse(std::unique_ptr<kudu::rpc::InboundTransfer, std::default_delete<kudu::rpc::InboundTransfer> >) src/kudu/rpc/connection.cc:735:14
#9 0x7fa79b26e0d6 in kudu::rpc::Connection::ReadHandler(ev::io&, int) src/kudu/rpc/connection.cc:673:7
SUMMARY: AddressSanitizer: heap-use-after-free src/kudu/gutil/atomicops-internals-x86.h:200:10 in base::subtle::NoBarrier_Load(int const volatile*)
Shadow bytes around the buggy address:
0x0c108001d370: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d380: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d390: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d3a0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d3b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c108001d3c0: fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d3d0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d3e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c108001d3f0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d400: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c108001d410: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Reviewed-on: http://gerrit.cloudera.org:8080/16839
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/client/client-test.cc
M src/kudu/client/client.h
M src/kudu/client/meta_cache.cc
M src/kudu/client/meta_cache.h
4 files changed, 65 insertions(+), 15 deletions(-)
Approvals:
Alexey Serbin: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/16839
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Gerrit-Change-Number: 16839
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [client] make metacache reset safe
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16839 )
Change subject: [client] make metacache reset safe
......................................................................
Patch Set 1: Code-Review+2
Thanks for finding and fixing this!
--
To view, visit http://gerrit.cloudera.org:8080/16839
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Gerrit-Change-Number: 16839
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 09 Dec 2020 06:50:22 +0000
Gerrit-HasComments: No
[kudu-CR] [client] make metacache reset safe
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16839 )
Change subject: [client] make metacache reset safe
......................................................................
Patch Set 1: Verified+1
unrelated test failure in RaftConsensusITest.TestLargeBatches
--
To view, visit http://gerrit.cloudera.org:8080/16839
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Gerrit-Change-Number: 16839
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 09 Dec 2020 06:47:08 +0000
Gerrit-HasComments: No
[kudu-CR] [client] make metacache reset safe
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [client] make metacache reset safe
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/16839
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I03ec9318526fbfc2da9b068eb3bbd9cd996efbca
Gerrit-Change-Number: 16839
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)