You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Yida Wu (Jira)" <ji...@apache.org> on 2025/08/15 19:49:00 UTC
[jira] [Created] (IMPALA-14316) Large batch insertion can lead to high memory usage and backpressure
Yida Wu created IMPALA-14316:
--------------------------------
Summary: Large batch insertion can lead to high memory usage and backpressure
Key: IMPALA-14316
URL: https://issues.apache.org/jira/browse/IMPALA-14316
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.5.0
Reporter: Yida Wu
Impalad could experience rapid and significant memory growth when INSERT statements with a large number of VALUES clauses (e.g., 10000). The memory is not released after the query completes, and end up with backpressure errors while causing impalad to become unresponsive to new queries. This behavior is not observed with smaller batch sizes (e.g., 100).
{code:java}
Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances request on impala.ControlService from 127.0.0.1:58626 dropped due to backpressure. The service queue contains 0 items out of a maximum of 2147483647; memory consumption is 0.
{code}
pprof shows that the massive allocation is caused by TUnionNode and TExpr objects, it is possible that the expression vectors in TUnionNode not being released promptly and needs further investigation into the cause.
{code:java}
10000 batch pprof growth:
Total: 5847.7 MB
4989.0 85.3% 85.3% 5103.0 87.3% std::vector::operator= (inline)
527.1 9.0% 94.3% 527.1 9.0% JVM_FindSignal
114.0 1.9% 96.3% 5103.0 87.3% std::vector::vector (inline)
0.0 0.0% 100.0% 5504.7 94.1% boost::detail::function::void_function_obj_invoker0::invoke
0.0 0.0% 100.0% 5504.7 94.1% boost::detail::function::void_function_obj_invoker0::invoke (inline)
0.0 0.0% 100.0% 5512.7 94.3% boost::detail::thread_data::run
0.0 0.0% 100.0% 5512.7 94.3% boost::detail::thread_data::run (inline)
...
0.0 0.0% 100.0% 478.7 8.2% impala::DeserializeThriftMsg
...
0.0 0.0% 100.0% 4849.0 82.9% impala::QueryExecMgr::ExecuteQueryHelper
0.0 0.0% 100.0% 4849.0 82.9% impala::QueryState::StartFInstances
....
0.0 0.0% 100.0% 4989.0 85.3% impala::TExpr::TExpr
0.0 0.0% 100.0% 94.0 1.6% impala::TExpr::read
0.0 0.0% 100.0% 90.0 1.5% impala::TExpr::read (inline)
0.0 0.0% 100.0% 287.0 4.9% impala::TExprNode::TExprNode
0.0 0.0% 100.0% 4.0 0.1% impala::TExprNode::read
0.0 0.0% 100.0% 447.0 7.6% impala::TPlan::operator=
0.0 0.0% 100.0% 95.0 1.6% impala::TPlan::read
0.0 0.0% 100.0% 88.0 1.5% impala::TPlanExecInfo::read
0.0 0.0% 100.0% 447.0 7.6% impala::TPlanFragment::TPlanFragment
0.0 0.0% 100.0% 95.0 1.6% impala::TPlanFragment::read
0.0 0.0% 100.0% 5103.0 87.3% impala::TPlanNode::TPlanNode
0.0 0.0% 100.0% 95.0 1.6% impala::TPlanNode::read
0.0 0.0% 100.0% 88.0 1.5% impala::TQueryExecRequest::read
0.0 0.0% 100.0% 5103.0 87.3% impala::TUnionNode::operator=
0.0 0.0% 100.0% 95.0 1.6% impala::TUnionNode::read
0.0 0.0% 100.0% 1.0 0.0% impala::TUnionNode::read (inline)
...
100 batch pprof growth:
Total: 402.0 MB
227.5 56.6% 56.6% 227.5 56.6% JVM_FindSignal
...
9.0 2.2% 91.5% 10.0 2.5% std::vector::operator= (inline)
2.0 0.5% 97.0% 2.0 0.5% std::vector::_M_default_append (inline)
1.0 0.2% 97.3% 2.0 0.5% Java_java_lang_ClassLoader_defineClass1
...
0.0 0.0% 100.0% 23.0 5.7% impala::ClientRequestState::FinishExecQueryOrDmlRequest
0.0 0.0% 100.0% 2.0 0.5% impala::ControlService::ExecQueryFInstances
...
0.0 0.0% 100.0% 23.0 5.7% impala::Coordinator::Exec
0.0 0.0% 100.0% 22.0 5.5% impala::Coordinator::FinishBackendStartup
0.0 0.0% 100.0% 1.0 0.2% impala::Coordinator::StartBackendExec
0.0 0.0% 100.0% 2.0 0.5% impala::DeserializeThriftMsg
...
0.0 0.0% 100.0% 9.0 2.2% impala::TExpr::TExpr
0.0 0.0% 100.0% 2.0 0.5% impala::TExpr::read
0.0 0.0% 100.0% 2.0 0.5% impala::TExpr::read (inline)
0.0 0.0% 100.0% 1.0 0.2% impala::TExprNode::TExprNode
0.0 0.0% 100.0% 2.0 0.5% impala::TPlan::operator=
0.0 0.0% 100.0% 2.0 0.5% impala::TPlan::read
0.0 0.0% 100.0% 2.0 0.5% impala::TPlanFragment::TPlanFragment
0.0 0.0% 100.0% 2.0 0.5% impala::TPlanFragment::read
0.0 0.0% 100.0% 10.0 2.5% impala::TPlanNode::TPlanNode
0.0 0.0% 100.0% 2.0 0.5% impala::TPlanNode::read
0.0 0.0% 100.0% 10.0 2.5% impala::TUnionNode::operator=
0.0 0.0% 100.0% 2.0 0.5% impala::TUnionNode::read
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org