You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2017/11/03 22:03:01 UTC
[jira] [Resolved] (IMPALA-6137) ASAN heap-use-after-free in
HdfsTextScanner::CheckForSplitDelimiter()
[ https://issues.apache.org/jira/browse/IMPALA-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-6137.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.11.0
IMPALA-6137: fix text scanner split delim mem mgmt
The bug was that the buffer pointed to by byte_buffer_ptr_ could be freed
by ReleaseCompletedResources() before CheckForSplitDelimiter() was called.
The simple fix is to copy out the single byte that is needed each time
the buffer is filled.
Testing:
Ran exhaustive query tests under ASAN with --disable_mem_pools=true.
Before the change test_text_split_delimiters reliably caused an ASAN
failure when run with --disable_mem_pools=true. We should get this
coverage automatically once the I/O mgr switches to using the buffer
pool, which uses ASAN poisoning on freed buffers.
Change-Id: Iddbb5cf6acc8f0b0e0b4c205c334f21e03d06f1c
Reviewed-on: http://gerrit.cloudera.org:8080/8438
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
> ASAN heap-use-after-free in HdfsTextScanner::CheckForSplitDelimiter()
> ---------------------------------------------------------------------
>
> Key: IMPALA-6137
> URL: https://issues.apache.org/jira/browse/IMPALA-6137
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.11.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: correctness, crash, resource-management
> Fix For: Impala 2.11.0
>
>
> While working on the scanner memory management I found some latent issues with memory lifetime in the text scanners.
> Here's a problem ASAN uncovered on my private branch.
> {code}
> ==8817==ERROR: AddressSanitizer: heap-use-after-free on address 0x6310008d4803 at pc 0x000001ba3453 bp 0x7faa70821a90 sp 0x7faa70821a88
> READ of size 1 at 0x6310008d4803 thread T12271
> #0 0x1ba3452 in impala::HdfsTextScanner::CheckForSplitDelimiter(bool*) /tmp/be/src/exec/hdfs-text-scanner.cc:705:10
> #1 0x1ba14a9 in impala::HdfsTextScanner::FinishScanRange(impala::RowBatch*) /tmp/be/src/exec/hdfs-text-scanner.cc:243:39
> #2 0x1ba6cee in impala::HdfsTextScanner::GetNextInternal(impala::RowBatch*) /tmp/be/src/exec/hdfs-text-scanner.cc:451:41
> #3 0x1b5353e in impala::HdfsScanner::ProcessSplit() /tmp/be/src/exec/hdfs-scanner.cc:120:21
> #4 0x1b19345 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::DiskIoMgr::ScanRange*) /tmp/be/s
> rc/exec/hdfs-scan-node.cc:532:21
> #5 0x1b18609 in impala::HdfsScanNode::ScannerThread() /tmp/be/src/exec/hdfs-scan-node.cc:441:16
> #6 0x160bc82 in boost::function0<void>::operator()() const /tmp/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #7 0x1a323e7 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /tmp/be/src/util/thread.cc:352:3
> #8 0x1a3d175 in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >
> ::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::strin
> g const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /tmp/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457:9
> #9 0x1a3cff1 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /tmp/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #10 0x23db4d9 in thread_proxy (/home/tarmstrong/Impala/incubator-impala/be/build/debug/service/impalad+0x23db4d9)
> #11 0x7fad4fe1c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
> #12 0x7fad4f93c3dc in clone /build/glibc-bfm8X4/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> 0x6310008d4803 is located 3 bytes inside of 65536-byte region [0x6310008d4800,0x6310008e4800)
> freed by thread T12271 here:
> #0 0x12fc600 in __interceptor_free /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:47
> #1 0x16bcdbe in impala::DiskIoRequestContext::FreeBuffer(impala::DiskIoMgr::BufferDescriptor*) /tmp/be/src/runtime/disk-io-mgr-reader-context.cc:48:3
> #2 0x16a73fc in impala::DiskIoMgr::ReturnBuffer(std::unique_ptr<impala::DiskIoMgr::BufferDescriptor, std::default_delete<impala::DiskIoMgr::BufferDescriptor> >) /tmp/be/src/runtime/disk-io-mgr.cc:463:15
> #3 0x1d8fc18 in impala::ScannerContext::Stream::ReleaseCompletedResources(bool) /tmp/be/src/exec/scanner-context.cc:109:44
> #4 0x1d8fa1d in impala::ScannerContext::ReleaseCompletedResources(bool) /tmp/be/src/exec/scanner-context.cc:63:18
> #5 0x1b54838 in impala::HdfsScanner::CommitRows(int, impala::RowBatch*) /tmp/be/src/exec/hdfs-scanner.cc:195:15
> #6 0x1ba5622 in impala::HdfsTextScanner::ProcessRange(impala::RowBatch*, int*) /tmp/be/src/exec/hdfs-text-scanner.cc:403:41
> #7 0x1ba6a82 in impala::HdfsTextScanner::GetNextInternal(impala::RowBatch*) /tmp/be/src/exec/hdfs-text-scanner.cc:443:41
> #8 0x1b5353e in impala::HdfsScanner::ProcessSplit() /tmp/be/src/exec/hdfs-scanner.cc:120:21
> #9 0x1b19345 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::DiskIoMgr::ScanRange*) /tmp/be/src/exec/hdfs-scan-node.cc:532:21
> #10 0x1b18609 in impala::HdfsScanNode::ScannerThread() /tmp/be/src/exec/hdfs-scan-node.cc:441:16
> #11 0x160bc82 in boost::function0<void>::operator()() const /tmp/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #12 0x1a323e7 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /tmp/be/src/util/thread.cc:352:3
> #13 0x1a3d175 in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /tmp/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457:9
> {code}
> Here's a different problem uncovered on master:
> {code}
> ./buildall.sh -asan -skiptests -noclean -ninja -notests && start-impala-cluster.py --impalad_args=--disable_mem_pools=true && impala-py.test -n4 --verbose tests/query_test/test_scanners.py tests/query_test/test_aggregation.py --workload_exploration_strategy=functional-query:exhaustive -k text
> ==11633==ERROR: AddressSanitizer: heap-use-after-free on address 0x6310010b882e at pc 0x0000012e8065 bp 0x7f3e8c708150 sp 0x7f3e8c707900
> READ of size 1 at 0x6310010b882e thread T899
> #0 0x12e8064 in __asan_memcpy /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:413
> #1 0x1709a72 in impala::Tuple::DeepCopyVarlenData(impala::TupleDescriptor const&, char**, int*, bool) /tmp/be/src/runtime/tuple.cc:143:5
> #2 0x16eedb5 in impala::RowBatch::SerializeInternal(long, impala::FixedSizeHashTable<impala::Tuple*, int>*, impala::TRowBatch*) /tmp/be/src/runtime/row-batch.cc:281:14
> #3 0x16ed1c8 in impala::RowBatch::Serialize(impala::TRowBatch*, bool) /tmp/be/src/runtime/row-batch.cc:188:5
> #4 0x16ecf7b in impala::RowBatch::Serialize(impala::TRowBatch*) /tmp/be/src/runtime/row-batch.cc:161:10
> #5 0x22d3ea9 in impala::DataStreamSender::SerializeBatch(impala::RowBatch*, impala::TRowBatch*, int) /tmp/be/src/runtime/data-stream-sender.cc:518:46
> #6 0x22d67d6 in impala::DataStreamSender::Send(impala::RuntimeState*, impala::RowBatch*) /tmp/be/src/runtime/data-stream-sender.cc:429:41
> #7 0x17352ae in impala::FragmentInstanceState::ExecInternal() /tmp/be/src/runtime/fragment-instance-state.cc:275:48
> #8 0x17324ac in impala::FragmentInstanceState::Exec() /tmp/be/src/runtime/fragment-instance-state.cc:89:14
> #9 0x16dc212 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /tmp/be/src/runtime/query-state.cc:380:24
> #10 0x160e6e2 in boost::function0<void>::operator()() const /tmp/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #11 0x1a390c7 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /tmp/be/src/util/thread.cc:352:3
> #12 0x1a43e55 in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /tmp/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457:9
> #13 0x1a43cd1 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /tmp/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #14 0x23e24a9 in thread_proxy (/home/tarmstrong/Impala/incubator-impala/be/build/debug/service/impalad+0x23e24a9)
> #15 0x7f414dab86b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
> #16 0x7f414d5d83dc in clone /build/glibc-bfm8X4/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> 0x6310010b882e is located 46 bytes inside of 65536-byte region [0x6310010b8800,0x6310010c8800)
> freed by thread T911 here:
> #0 0x1336960 in operator delete[](void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:114
> #1 0x16acae0 in impala::DiskIoMgr::FreeBufferMemory(impala::DiskIoMgr::BufferDescriptor*) /tmp/be/src/runtime/disk-io-mgr.cc:811:7
> #2 0x16ac545 in impala::DiskIoMgr::ReturnBuffer(std::unique_ptr<impala::DiskIoMgr::BufferDescriptor, std::default_delete<impala::DiskIoMgr::BufferDescriptor> >) /tmp/be/src/runtime/disk-io-mgr.cc:688:7
> #3 0x1d96df4 in impala::ScannerContext::Stream::ReleaseCompletedResources(impala::RowBatch*, bool) /tmp/be/src/exec/scanner-context.cc:117:46
> #4 0x1d96b50 in impala::ScannerContext::ReleaseCompletedResources(impala::RowBatch*, bool) /tmp/be/src/exec/scanner-context.cc:63:18
> #5 0x1ba71fe in impala::HdfsTextScanner::Close(impala::RowBatch*) /tmp/be/src/exec/hdfs-text-scanner.cc:169:15
> #6 0x7f3ed973c141 in impala::HdfsLzoTextScanner::Close(impala::RowBatch*) (/home/tarmstrong/Impala/Impala-lzo/build/libimpalalzo.so+0x1a141)
> #7 0x1b5c0bc in impala::HdfsScanner::Close() /tmp/be/src/exec/hdfs-scanner.cc:129:3
> #8 0x1b21c94 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::DiskIoMgr::ScanRange*) /tmp/be/src/exec/hdfs-scan-node.cc:551:12
> #9 0x1b20c95 in impala::HdfsScanNode::ScannerThread() /tmp/be/src/exec/hdfs-scan-node.cc:442:16
> #10 0x160e6e2 in boost::function0<void>::operator()() const /tmp/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #11 0x1a390c7 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /tmp/be/src/util/thread.cc:352:3
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)