You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Fifteen (Jira)" <ji...@apache.org> on 2020/11/19 09:39:00 UTC
[jira] [Created] (IMPALA-10342) Alleviating congestion caused by
row-level warnings
Fifteen created IMPALA-10342:
--------------------------------
Summary: Alleviating congestion caused by row-level warnings
Key: IMPALA-10342
URL: https://issues.apache.org/jira/browse/IMPALA-10342
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Fifteen
Attachments: image-2020-11-19-17-30-22-918.png
By default, when encounting error, both `get_json_object()` and `DecimalOperators::IntToDecimalVal` will raise warning. During to their stateless nature, functions keep throwing messages. Hence the warning flood will easily overwhelm cluster's processing capacity.
To be specific, we have observed these bottlenecks:
*Exchange Receiver*: the default value for `rpc_max_message_size` is 50MB. The flooding warning messages carried by ReportExecStatusPB will exceed that limit, causing report without profile. Even though the report message size is less than those limit, the bandwidth consumption is non-trivial.
*Storage:* like https://issues.apache.org/jira/browse/IMPALA-5256 , warning messages produces huge log files since `stdout/stderr` won't be redirected when glog is rolling log files.
*Coordinator*: runtime profiles will be seriialized to thrift and stored in Coordinator's memory. The warning flood will make `Untracked Memory` rising rapidly. I have made a mem sample and found most memory were used by RuntimeProfile and Strings.
!image-2020-11-19-17-30-22-918.png!
Solution:
# We have a straightforward solution by changing `AddWarning()` to `no-op`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org