[jira] [Resolved] (IMPALA-7704) ASAN tests failing in HdfsParquetTableWriter

["Joe McDonnell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell resolved IMPALA-7704.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.1.0

The ASAN tests now run successfully. The IMPALA-7644 fix will be reapplied using that JIRA.

> ASAN tests failing in HdfsParquetTableWriter
> --------------------------------------------
>
>                 Key: IMPALA-7704
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7704
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 3.1.0
>            Reporter: Joe McDonnell
>            Assignee: Zoltán Borók-Nagy
>            Priority: Blocker
>              Labels: broken-build
>             Fix For: Impala 3.1.0
>
>
> ASAN tests have been failing for the last few runs. Here is the output:
> {noformat}
> ==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0 at pc 0x0000016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
> READ of size 32681 at 0x7ef9312e0ec0 thread T82364
> #0 0x16a0c97 in __interceptor_memcpy /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
> #1 0x7f02e43c2dca in jni_SetByteArrayRegion (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
> #2 0x4ae29b7 in hdfsWrite /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
> #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
> #4 0x25ac037 in impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
> #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
> #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
> #7 0x251e424 in impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*, impala::OutputPartition*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
> #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
> #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
> #16 0x39a11e9 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9)
> #17 0x7f02e19f4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
> #18 0x7f02e150b34c in __clone (/lib64/libc.so.6+0xf834c)
> 0x7ef9312e0ec0 is located 161472 bytes inside of 524288-byte region [0x7ef9312b9800,0x7ef931339800)
> allocated by thread T82364 here:
> #0 0x16fcd88 in malloc /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:67
> #1 0x1f2a5ec in impala::MemPool::FindChunk(long, bool) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.cc:140:45
> #2 0x1f44c37 in unsigned char* impala::MemPool::Allocate<false>(long, int) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.h:284:9
> #3 0x1f44c37 in impala::MemPool::Allocate(long) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.h:107
> #4 0x25ad32c in impala::HdfsParquetTableWriter::BaseColumnWriter::FinalizeCurrentPage() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:797:61
> #5 0x25bbdea in impala::HdfsParquetTableWriter::BaseColumnWriter::AppendRow(impala::TupleRow*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:590:5
> #6 0x25b36cc in impala::HdfsParquetTableWriter::AppendRows(impala::RowBatch*, std::vector<int, std::allocator<int> > const&, bool*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1124:7
> #7 0x251ddec in impala::HdfsTableSink::WriteRowsToPartition(impala::RuntimeState*, impala::RowBatch*, std::pair<std::unique_ptr<impala::OutputPartition, std::default_delete<impala::OutputPartition> >, std::vector<int, std::allocator<int> > >*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:256:5
> #8 0x2522f01 in impala::HdfsTableSink::Send(impala::RuntimeState*, impala::RowBatch*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:590:5
> #9 0x1fc07ad in impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:316:5
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
> #16 0x39a11e9 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9)
> Thread T82364 created by T82363 here:
> #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
> #2 0x45e0360d (<unknown module>)
> Thread T82363 created by T207 here:
> #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
> #2 0x45e0360d (<unknown module>)
> Thread T207 created by T199 here:
> #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
> #2 0x45e0360d (<unknown module>)
> Thread T199 created by T198 here:
> #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
> #2 0x45e0360d (<unknown module>)
> Thread T198 created by T0 here:
> #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
> #2 0x45e0360d (<unknown module>)
> SUMMARY: AddressSanitizer: use-after-poison /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738 in __interceptor_memcpy
> Shadow bytes around the buggy address:
> 0x0fdfa6254180: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa6254190: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa62541a0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa62541b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa62541c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> =>0x0fdfa62541d0: f7 f7 f7 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa62541e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa62541f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa6254200: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa6254210: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> 0x0fdfa6254220: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07 
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==117268==ABORTING{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org