You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Zoram Thanga (JIRA)" <ji...@apache.org> on 2018/05/11 05:25:00 UTC
[jira] [Resolved] (IMPALA-6996) PartitionedAggregationNode::Close()
should not dump stack trace
[ https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zoram Thanga resolved IMPALA-6996.
----------------------------------
Resolution: Duplicate
> PartitionedAggregationNode::Close() should not dump stack trace
> ---------------------------------------------------------------
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.12.0
> Reporter: Zoram Thanga
> Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode. Upon processing the failure, the fragment instance that hit the MEM_LIMIT tries to allocate more memory, and repeatedly dumps the following error message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded: FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend <redacted>.com:22000 by fragment b429287bdabe184:9e7b5e960000002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e9600000000): memory limit exceeded. Limit=6.00 GB Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e960000002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e960000000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661 impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152 impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
> -----------------------------------------------------------------------------------------------------------------------
> 03:AGGREGATE 32 0.000ns 0.000ns 0 1.06B 2.75 GB 165.23 GB FINALIZE
> 02:EXCHANGE 32 1s246ms 7s429ms 412.85M 1.06B 0 0 HASH(<redacted>)
> 01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B 3.09 GB 165.23 GB STREAMING
> 00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB 616.00 MB <redacted>
> Errors: Memory limit exceeded: Error occurred on backend <redacted>.com:22000 by fragment b429287bdabe184:9e7b5e960000002c
> Memory left in process limit: 386.76 GB
> Memory left in query limit: -19080.00 B
> {noformat}
> And the query string:
> {noformat}
> create TABLE TT><redacted>
> STORED AS PARQUET
> TBLPROPERTIES('parquet.compression'='SNAPPY')
> T>AS
> T>SELECT T> <redacted>,
> <redacted>,
> <redacted>,
> MAX(<redacted>) AS <redacted>,
> MAX(<redacted>) AS <redacted>,
> MAX(<redacted>) AS <redacted>,
> MAX(<redacted>) AS <redacted>
> T>FROM TT> <redacted>
> GROUP BY <redacted>,
> <redacted>,
> <redacted>
> Coordinator: <redacted>.com:22000
> Query Options (non default): MEM_LIMIT=6442450944,REQUEST_POOL=<redacted>,SYNC_DDL=1,MT_DOP=0
> DDL Type: CREATE_TABLE_AS_SELECT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)