You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Lars Volker (JIRA)" <ji...@apache.org> on 2019/04/04 18:53:00 UTC
[jira] [Updated] (IMPALA-8388) Misleading error message when
rejecting incoming RPCs b/c of memory pressure
[ https://issues.apache.org/jira/browse/IMPALA-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Volker updated IMPALA-8388:
--------------------------------
Description:
When running out of memory we reject incoming RPCs (expected). However, our error message assumes that the queue is full and prints it as INT_MAX (the queue size):
{code}
void ImpalaServicePool::RejectTooBusy(kudu::rpc::InboundCall* c) {
string err_msg =
Substitute("$0 request on $1 from $2 dropped due to backpressure. "
"The service queue contains $3 items out of a maximum of $4; "
"memory consumption is $5.",
c->remote_method().method_name(),
service_->service_name(),
c->remote_address().ToString(),
service_queue_.estimated_queue_length(),
service_queue_.max_size(), // <-- HERE
PrettyPrinter::Print(service_mem_tracker_->consumption(), TUnit::BYTES));
{code}
The error currently looks like this:
{noformat}
I0404 11:35:43.276937 54321 impala-service-pool.cc:126] EndDataStream request on impala.DataStreamService from 1.2.3.4:56789 dropped due to backpressure. The service queue is full; it has 2147483647 items. Contents of service queue:
{noformat}
was:
When running out of memory we reject incoming RPCs (expected). However, our error message assumes that the queue is full and prints it as INT_MAX (the queue size):
{code}
void ImpalaServicePool::RejectTooBusy(kudu::rpc::InboundCall* c) {
string err_msg =
Substitute("$0 request on $1 from $2 dropped due to backpressure. "
"The service queue contains $3 items out of a maximum of $4; "
"memory consumption is $5.",
c->remote_method().method_name(),
service_->service_name(),
c->remote_address().ToString(),
service_queue_.estimated_queue_length(),
service_queue_.max_size(), // <-- HERE
PrettyPrinter::Print(service_mem_tracker_->consumption(), TUnit::BYTES));
{code}
> Misleading error message when rejecting incoming RPCs b/c of memory pressure
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8388
> URL: https://issues.apache.org/jira/browse/IMPALA-8388
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.12.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
> Reporter: Lars Volker
> Priority: Major
> Labels: observability, supportability
>
> When running out of memory we reject incoming RPCs (expected). However, our error message assumes that the queue is full and prints it as INT_MAX (the queue size):
> {code}
> void ImpalaServicePool::RejectTooBusy(kudu::rpc::InboundCall* c) {
> string err_msg =
> Substitute("$0 request on $1 from $2 dropped due to backpressure. "
> "The service queue contains $3 items out of a maximum of $4; "
> "memory consumption is $5.",
> c->remote_method().method_name(),
> service_->service_name(),
> c->remote_address().ToString(),
> service_queue_.estimated_queue_length(),
> service_queue_.max_size(), // <-- HERE
> PrettyPrinter::Print(service_mem_tracker_->consumption(), TUnit::BYTES));
> {code}
> The error currently looks like this:
> {noformat}
> I0404 11:35:43.276937 54321 impala-service-pool.cc:126] EndDataStream request on impala.DataStreamService from 1.2.3.4:56789 dropped due to backpressure. The service queue is full; it has 2147483647 items. Contents of service queue:
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org