[jira] [Resolved] (IMPALA-3555) Possible hang in ASAN with huge memcpy

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-3555.
-----------------------------------
    Resolution: Cannot Reproduce

> Possible hang in ASAN with huge memcpy
> --------------------------------------
>
>                 Key: IMPALA-3555
>                 URL: https://issues.apache.org/jira/browse/IMPALA-3555
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.5.0
>            Reporter: Matthew Jacobs
>            Priority: Minor
>              Labels: asan, hang, resource-management
>         Attachments: asan-hang.impalad.INFO.zip, asan-hang-query-profile.out, asan-hang.stacks-full.out, asan-hang.stacks.out
>
>
> Running the ASAN exhaustive build with mem-pools disabled seems to be hanging while running the large strings tests. It looks like there may be an issue with ASAN reallocating very large buffers.
> The query (abridged here, see attachment) seems to be hung:
> {code}
> select length(group_concat(l_comment, "!")) from (
> select l_comment from tpch_parquet.lineitem union all
> select l_comment from tpch_parquet.lineitem union all
> ... -- 48 more copies...
> select l_comment from tpch_parquet.lineitem) a
> {code}
> It looks like the hang is in the following stack:
> {code}
> Thread 6 (Thread 0x7fd323df7700 (LWP 13170)):
> #0  0x0000003ad5689c20 in memcpy () from /lib64/libc.so.6
> #1  0x0000000000e85628 in __asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:548
> #2  0x0000000000f25102 in realloc () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:79
> #3  0x0000000001b3ed7b in impala_udf::FunctionContext::Reallocate(unsigned char*, int) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/udf/udf.cc:310
> #4  0x000000000187e50f in impala::AggregateFunctions::StringConcatUpdate(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::StringVal*) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exprs/aggregate-functions-ir.cc:623
> #5  0x0000000001874fb5 in impala::AggFnEvaluator::Update(impala_udf::FunctionContext*, impala::TupleRow*, impala::Tuple*, void*) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exprs/agg-fn-evaluator.cc:365
> #6  0x00000000017af3cc in impala::PartitionedAggregationNode::UpdateTuple(impala_udf::FunctionContext**, impala::Tuple*, impala::TupleRow*, bool) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:1066
> #7  0x00000000017bf0c0 in impala::PartitionedAggregationNode::ProcessBatchNoGrouping(impala::RowBatch*, impala::HashTableCtx const*) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/partitioned-aggregation-node-ir.cc:30
> #8  0x00000000017a4428 in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:337
> #9  0x0000000001b1f8d7 in impala::PlanFragmentExecutor::OpenInternal() () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:355
> #10 0x0000000001b1eace in impala::PlanFragmentExecutor::Open() () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:327
> #11 0x00000000014bf0b3 in impala::FragmentMgr::FragmentExecState::Exec() () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/fragment-exec-state.cc:54
> #12 0x00000000014b18c6 in impala::FragmentMgr::FragmentThread(impala::TUniqueId) () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/fragment-mgr.cc:86
> #13 0x00000000014b6fbe in boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>::operator()(impala::FragmentMgr*, impala::TUniqueId) const () at /tmp/impala-deps/boost-1.57.0/include/boost/bind/mem_fn_template.hpp:165
> {code}
> All stacks (both frames-only and including locals) are attached, though the instance that was hung is no longer available. When I detached the debugger the query in question died (unreachable impalad) and the tests continued. I canceled the run though so that I could kick off another.
> As of now I don't have any reason to believe this affects non-ASAN builds.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)