You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Michael Ho (JIRA)" <ji...@apache.org> on 2019/02/26 23:04:00 UTC
[jira] [Resolved] (IMPALA-8239) Check failed:
deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ +
num_pending_enqueue_) > 0
[ https://issues.apache.org/jira/browse/IMPALA-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Ho resolved IMPALA-8239.
--------------------------------
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0
> ----------------------------------------------------------------------------------------------------
>
> Key: IMPALA-8239
> URL: https://issues.apache.org/jira/browse/IMPALA-8239
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.2.0
> Reporter: Tim Armstrong
> Assignee: Michael Ho
> Priority: Blocker
> Labels: crash
> Fix For: Impala 3.2.0
>
> Attachments: IMPALA-8239-logs.tar.gz
>
>
> {noformat}
> F0221 12:36:27.886497 157443 krpc-data-stream-recvr.cc:234] 3843237c92f4c6b5:b5aa170b00000092] Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0
> *** Check failure stack trace: ***
> @ 0x47eb92c google::LogMessage::Fail()
> @ 0x47ed1d1 google::LogMessage::SendToLog()
> @ 0x47eb306 google::LogMessage::Flush()
> @ 0x47ee8cd google::LogMessageFatal::~LogMessageFatal()
> @ 0x1ed07a0 impala::KrpcDataStreamRecvr::SenderQueue::GetBatch()
> @ 0x1ed65e7 impala::KrpcDataStreamRecvr::GetBatch()
> @ 0x22c5d67 impala::ExchangeNode::FillInputRowBatch()
> @ 0x22c5953 impala::ExchangeNode::Open()
> @ 0x240079a impala::BlockingJoinNode::ProcessBuildInputAndOpenProbe()
> @ 0x238720f impala::PartitionedHashJoinNode::Open()
> @ 0x23e6d76 impala::AggregationNode::Open()
> @ 0x1f52b2f impala::FragmentInstanceState::Open()
> @ 0x1f4f7f3 impala::FragmentInstanceState::Exec()
> @ 0x1f62dd9 impala::QueryState::ExecFInstance()
> @ 0x1f610bb _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x1f6421a _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1d766b3 boost::function0<>::operator()()
> @ 0x2223e4e impala::Thread::SuperviseThread()
> @ 0x222c1d2 boost::_bi::list5<>::operator()<>()
> @ 0x222c0f6 boost::_bi::bind_t<>::operator()()
> @ 0x222c0b9 boost::detail::thread_data<>::run()
> @ 0x3715469 thread_proxy
> @ 0x7f262e1efe24 start_thread
> @ 0x7f262df1d34c __clone
> {noformat}
> Immediately before it in the log a different thread printed:
> {noformat}
> E0221 12:36:27.557099 157280 krpc-data-stream-sender.cc:344] 4145938c29f31877:6ed16e5e0000022d] channel send to 10.17.211.20:27000 failed: (fragment_instance_id=4145938c29f31877:6ed16e5e0000024c): Memory limit exceeded: Failed to allocate row batch
> EXCHANGE_NODE (id=189) could not allocate 8.00 KB without exceeding limit.
> Error occurred on backend vc0510.halxg.cloudera.com:22000
> Memory left in process limit: 12.11 GB
> Memory left in query limit: 541.65 MB
> Query(4145938c29f31877:6ed16e5e00000000): Limit=1.39 GB Reservation=704.88 MB ReservationLimit=1.11 GB OtherMemory=174.47 MB Total=879.35 MB Peak=926.55 MB
> Unclaimed reservations: Reservation=430.88 MB OtherMemory=0 Total=430.88 MB Peak=740.31 MB
> <fragment instance>: Reservation=0 OtherMemory=0 Total=0 Peak=1.59 MB
> SORT_NODE (id=118): Total=0 Peak=4.00 KB
> AGGREGATION_NODE (id=203): Reservation=0 OtherMemory=0 Total=0 Peak=81.12 KB
> GroupingAggregator 0: Total=0 Peak=81.12 KB
> EXCHANGE_NODE (id=202): Reservation=0 OtherMemory=0 Total=0 Peak=0
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=204): Total=0 Peak=1.75 KB
> CodeGen: Total=0 Peak=1.50 MB
> <fragment instance>: Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> AGGREGATION_NODE (id=117): Reservation=0 OtherMemory=0 Total=0 Peak=81.12 KB
> GroupingAggregator 0: Total=0 Peak=81.12 KB
> UNION_NODE (id=0): Total=0 Peak=0
> AGGREGATION_NODE (id=144): Reservation=0 OtherMemory=0 Total=0 Peak=34.12 KB
> GroupingAggregator 0: Total=0 Peak=34.12 KB
> {noformat}
> Logs for the crashing thread:
> {noformat}
> I0221 12:36:23.677255 157443 query-state.cc:624] 3843237c92f4c6b5:b5aa170b00000092] Executing instance. instance_id=3843237c92f4c6b5:b5aa170b00000092 fragment_idx=10 per_fragment_instance_idx=7 coord_state_idx=10
> #in-flight=525
> {noformat}
> The query was:
> {noformat}
> I0221 12:36:18.419795 140378 impala-server.cc:1063] 3843237c92f4c6b5:b5aa170b00000000] Registered query query_id=3843237c92f4c6b5:b5aa170b00000000 session_id=f24bd763b83e0b5e:77f6ea4b07165d9b
> I0221 12:36:18.420265 140378 Frontend.java:1245] 3843237c92f4c6b5:b5aa170b00000000] Analyzing query: /* Mem: 218 MB. Coordinator: vc0515.halxg.cloudera.com. */
> -- RESULT MISMATCH FROM ORIGINAL in DECIMAL value
> -- FIXED. CAST RESULT QUOTIENT TO DECIMAL(15, 4), TAKE ACTUAL RESULT AS EXPECTED
> select promotions,total,cast(promotions*100.00/total as decimal(15,4))
> from
> (select sum(ss_ext_sales_price) promotions
> from store_sales
> ,store
> ,promotion
> ,date_dim
> ,customer
> ,customer_address
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_store_sk = s_store_sk
> and ss_promo_sk = p_promo_sk
> and ss_customer_sk= c_customer_sk
> and ca_address_sk = c_current_addr_sk
> and ss_item_sk = i_item_sk
> and ca_gmt_offset = -5
> and i_category = 'Jewelry'
> and (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y')
> and s_gmt_offset = -5
> and d_year = 1998
> and d_moy = 1) promotional_sales,
> (select sum(ss_ext_sales_price) total
> from store_sales
> ,store
> ,date_dim
> ,customer
> ,customer_address
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_store_sk = s_store_sk
> and ss_customer_sk= c_customer_sk
> and ca_address_sk = c_current_addr_sk
> and ss_item_sk = i_item_sk
> and ca_gmt_offset = -5
> and i_category = 'Jewelry'
> and s_gmt_offset = -5
> and d_year = 1998
> and d_moy = 1) all_sales
> order by promotions, total
> {noformat}
> Logs of the crashing Impala and coordinator are in [^IMPALA-8239-logs.tar.gz]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)