You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2019/01/25 17:43:00 UTC
[jira] [Commented] (IMPALA-8118) Build failure: Connection failure
in multiple tests
[ https://issues.apache.org/jira/browse/IMPALA-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752498#comment-16752498 ]
Tim Armstrong commented on IMPALA-8118:
---------------------------------------
From ee_tests/impalad.ERROR
{noformat}
E0125 05:25:21.153029 65917 krpc-data-stream-sender.cc:344] 974ec7e179ce3698:6f26a17d00000005] channel send to 127.0.0.1:27002 failed: (fragment_instance_id=974ec7e179ce3698:6f26a17d0000000b): Mem
ory limit exceeded: Failed to allocate row batch
EXCHANGE_NODE (id=11) could not allocate 32.00 KB without exceeding limit.
Error occurred on backend impala-ec2-centos74-r4-4xlarge-ondemand-1162.vpc.cloudera.com:22002
Memory left in process limit: 11.08 GB
Memory left in query limit: 1.83 KB
Query(974ec7e179ce3698:6f26a17d00000000): Limit=130.00 MB Reservation=80.88 MB ReservationLimit=98.00 MB OtherMemory=49.12 MB Total=130.00 MB Peak=130.01 MB
Unclaimed reservations: Reservation=26.88 MB OtherMemory=0 Total=26.88 MB Peak=70.88 MB
Fragment 974ec7e179ce3698:6f26a17d0000000e: Reservation=0 OtherMemory=1.67 MB Total=1.67 MB Peak=1.67 MB
SORT_NODE (id=9): Total=0 Peak=0
AGGREGATION_NODE (id=16): Reservation=0 OtherMemory=61.12 KB Total=61.12 KB Peak=61.12 KB
GroupingAggregator 0: Total=61.12 KB Peak=61.12 KB
Exprs: Total=61.12 KB Peak=61.12 KB
EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=0
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=17): Total=1.97 KB Peak=1.97 KB
CodeGen: Total=1.60 MB Peak=1.60 MB
Fragment 974ec7e179ce3698:6f26a17d0000000b: Reservation=3.00 MB OtherMemory=40.26 MB Total=43.26 MB Peak=43.26 MB
Runtime Filter Bank: Reservation=3.00 MB ReservationLimit=3.00 MB OtherMemory=0 Total=3.00 MB Peak=3.00 MB
AGGREGATION_NODE (id=8): Reservation=0 OtherMemory=61.12 KB Total=61.12 KB Peak=61.12 KB
GroupingAggregator 0: Total=61.12 KB Peak=61.12 KB
Exprs: Total=61.12 KB Peak=61.12 KB
HASH_JOIN_NODE (id=7): Total=58.25 KB Peak=58.25 KB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB
HASH_JOIN_NODE (id=6): Total=50.25 KB Peak=50.25 KB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB
HASH_JOIN_NODE (id=5): Total=42.25 KB Peak=42.25 KB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=5): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=5) Exprs: Total=13.12 KB Peak=13.12 KB
EXCHANGE_NODE (id=10): Reservation=10.20 MB OtherMemory=19.29 KB Total=10.21 MB Peak=10.21 MB
KrpcDeferredRpcs: Total=19.29 KB Peak=19.29 KB
EXCHANGE_NODE (id=11): Reservation=13.14 MB OtherMemory=0 Total=13.14 MB Peak=13.14 MB
KrpcDeferredRpcs: Total=0 Peak=0
EXCHANGE_NODE (id=12): Reservation=10.30 MB OtherMemory=0 Total=10.30 MB Peak=10.30 MB
KrpcDeferredRpcs: Total=0 Peak=0
AGGREGATION_NODE (id=14): Reservation=0 OtherMemory=17.12 KB Total=17.12 KB Peak=17.12 KB
GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
Exprs: Total=17.12 KB Peak=17.12 KB
:
EXCHANGE_NODE (id=13): Reservation=360.00 KB OtherMemory=0 Total=360.00 KB Peak=360.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=15): Total=5.91 KB Peak=5.91 KB
CodeGen: Total=6.01 MB Peak=6.01 MB
Fragment 974ec7e179ce3698:6f26a17d00000002: Reservation=10.00 MB OtherMemory=1.80 MB Total=11.80 MB Peak=11.82 MB
Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
HDFS_SCAN_NODE (id=2): Reservation=8.00 MB OtherMemory=1.68 MB Total=9.68 MB Peak=9.70 MB
Queued Batches: Total=1.25 MB Peak=1.27 MB
KrpcDataStreamSender (dst_id=10): Total=90.57 KB Peak=106.57 KB
CodeGen: Total=3.28 KB Peak=403.50 KB
Fragment 974ec7e179ce3698:6f26a17d00000004: Reservation=28.00 MB OtherMemory=3.58 MB Total=31.58 MB Peak=31.62 MB
Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
HDFS_SCAN_NODE (id=1): Reservation=26.00 MB OtherMemory=3.48 MB Total=29.48 MB Peak=29.52 MB
Queued Batches: Total=2.24 MB Peak=2.29 MB
KrpcDataStreamSender (dst_id=11): Total=54.36 KB Peak=94.36 KB
CodeGen: Total=3.28 KB Peak=412.50 KB
Fragment 974ec7e179ce3698:6f26a17d00000008: Reservation=13.00 MB OtherMemory=1.81 MB Total=14.81 MB Peak=14.86 MB
AGGREGATION_NODE (id=4): Reservation=5.00 MB OtherMemory=42.12 KB Total=5.04 MB Peak=5.04 MB
GroupingAggregator 0: Reservation=5.00 MB OtherMemory=17.12 KB Total=5.02 MB Peak=5.02 MB
Exprs: Total=17.12 KB Peak=17.12 KB
HDFS_SCAN_NODE (id=3): Reservation=8.00 MB OtherMemory=1.66 MB Total=9.66 MB Peak=9.70 MB
Queued Batches: Total=1.23 MB Peak=1.27 MB
KrpcDataStreamSender (dst_id=13): Total=83.35 KB Peak=99.35 KB
CodeGen: Total=6.09 KB Peak=650.00 KB
=================================================================
==5504==ERROR: AddressSanitizer: use-after-poison on address 0x631001cd518c at pc 0x000002012a0c bp 0x7fe2fda4c640 sp 0x7fe2fda4c638
READ of size 1 at 0x631001cd518c thread T113733
#0 0x2012a0b in impala::Tuple::IsNull(impala::NullIndicatorOffset const&) const /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/tuple.h:241:13
#1 0x28088c1 in impala::AggFnEvaluator::SerializeOrFinalize(impala::Tuple*, impala::SlotDescriptor const&, impala::Tuple*, void*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exprs/agg-fn-evaluator.cc:393:29
#2 0x27740b8 in impala::AggFnEvaluator::Finalize(std::vector<impala::AggFnEvaluator*, std::allocator<impala::AggFnEvaluator*> > const&, impala::Tuple*, impala::Tuple*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exprs/agg-fn-evaluator.h:307:15
#3 0x27aa286 in impala::GroupingAggregator::CleanupHashTbl(std::vector<impala::AggFnEvaluator*, std::allocator<impala::AggFnEvaluator*> > const&, impala::HashTable::Iterator) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:351:7
#4 0x27aa7a2 in impala::GroupingAggregator::ClosePartitions() /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:930:5
#5 0x27aaae4 in impala::GroupingAggregator::Close(impala::RuntimeState*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:383:3
#6 0x275fce7 in impala::AggregationNode::Close(impala::RuntimeState*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/aggregation-node.cc:139:32
#7 0x2067e49 in impala::FragmentInstanceState::Close() /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:368:42
#8 0x206317a in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:99:3
#9 0x207d432 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/query-state.cc:584:24
#10 0x1d77076 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-cdh6.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
#11 0x24b8326 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/util/thread.cc:359:3
#12 0x24c3818 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-cdh6.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
#13 0x24c366b in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-cdh6.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
#14 0x3c7fbd9 in thread_proxy (/data0/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/build/debug/service/impalad+0x3c7fbd9)
#15 0x7fecdb4a4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
#16 0x7fecdafbb34c in __clone (/lib64/libc.so.6+0xf834c)
0x631001cd518c is located 396 bytes inside of 65536-byte region [0x631001cd5000,0x631001ce5000)
allocated by thread T113733 here:
#0 0x177f2e5 in posix_memalign /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:148
#1 0x250ccb1 in impala::SystemAllocator::AllocateViaMalloc(long, unsigned char**) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/system-allocator.cc:123:12
#2 0x250c339 in impala::SystemAllocator::Allocate(long, impala::BufferPool::BufferHandle*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/system-allocator.cc:67:5
#3 0x250f759 in impala::BufferPool::BufferAllocator::AllocateInternal(impala::BufferPool::Client*, long, impala::BufferPool::BufferHandle*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-allocator.cc:362:38
#4 0x250eaeb in impala::BufferPool::BufferAllocator::Allocate(impala::BufferPool::ClientHandle*, long, impala::BufferPool::BufferHandle*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-allocator.cc:269:3
#5 0x24f2399 in impala::BufferPool::AllocateBuffer(impala::BufferPool::ClientHandle*, long, impala::BufferPool::BufferHandle*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:232:31
#6 0x24f1f2e in impala::BufferPool::CreatePage(impala::BufferPool::ClientHandle*, long, impala::BufferPool::PageHandle*, impala::BufferPool::BufferHandle const**) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/bufferpool/buffer-pool.cc:145:3
#7 0x2ce8255 in impala::BufferedTupleStream::NewWritePage(long) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:371:3
#8 0x2ce9016 in impala::BufferedTupleStream::AdvanceWritePage(long, bool*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:443:3
#9 0x2ced09b in impala::BufferedTupleStream::AddRowCustomBeginSlow(long, impala::Status*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:905:13
#10 0x27adc5e in impala::BufferedTupleStream::AddRowCustomBegin(long, impala::Status*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/runtime/buffered-tuple-stream.inline.h:38:12
#11 0x27adc5e in impala::GroupingAggregator::ConstructIntermediateTuple(std::vector<impala::AggFnEvaluator*, std::allocator<impala::AggFnEvaluator*> > const&, impala::BufferedTupleStream*, impala::Status*) /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/be/src/exec/grouping-aggregator.cc:497
#12 0x7fe416b19f11 (<unknown module>)
{noformat}
> Build failure: Connection failure in multiple tests
> ---------------------------------------------------
>
> Key: IMPALA-8118
> URL: https://issues.apache.org/jira/browse/IMPALA-8118
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Paul Rogers
> Assignee: Lenisha Gandhi
> Priority: Blocker
> Fix For: Impala 3.1.0
>
>
> Build of latest master, with ASAN, failed with the following error, which to my newbie eyes looks like a connection failure:
> {noformat}
> 05:42:04 =================================== FAILURES ===================================
> 05:42:04 TestQueriesTextTables.test_data_source_tables[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 05:42:04 [gw5] linux2 -- Python 2.7.5 /data/jenkins/workspace/impala-cdh6.x-core-asan/repos/Impala/bin/../infra/python/env/bin/python
> 05:42:04 query_test/test_queries.py:174: in test_data_source_tables
> 05:42:04 self.run_test_case('QueryTest/data-source-tables', vector)
> 05:42:04 common/impala_test_suite.py:472: in run_test_case
> 05:42:04 result = self.__execute_query(target_impalad_client, query, user=user)
> ...
> 05:42:04 handle = self.execute_query_async(query_string, user=user)
> 05:42:04 beeswax/impala_beeswax.py:351: in execute_query_async
> 05:42:04 handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> 05:42:04 beeswax/impala_beeswax.py:512: in __do_rpc
> 05:42:04 raise ImpalaBeeswaxException(self.__build_error_message(e), e)
> 05:42:04 E ImpalaBeeswaxException: ImpalaBeeswaxException:
> 05:42:04 E INNER EXCEPTION: <class 'thrift.transport.TTransport.TTransportException'>
> 05:42:04 E MESSAGE: TSocket read 0 bytes
> 05:42:04 ----------------------------- Captured stderr call -----------------------------
> ...
> 05:42:04 -- executing against localhost:21000
> 05:42:04 select *
> 05:42:04 from alltypes_datasource
> 05:42:04 where float_col != 0 and
> 05:42:04 int_col >= 1990 limit 5;
> {noformat}
> A similar error appears for multiple other tests. Then:
> {noformat}
> 05:42:04 TTransportException: Could not connect to localhost:21050
> 05:42:04 !!!!!!!!!!!!!!!!!!! Interrupted: stopping after 10 failures !!!!!!!!!!!!!!!!!!!!
> {noformat}
> I wonder if these are just symptoms of a failure in the BE code due to ASAN being enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org