You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Fifteen (Jira)" <ji...@apache.org> on 2020/11/20 01:34:00 UTC
[jira] [Comment Edited] (IMPALA-9865) Utility to pretty-print
thrift profiles at various levels
[ https://issues.apache.org/jira/browse/IMPALA-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235818#comment-17235818 ]
Fifteen edited comment on IMPALA-9865 at 11/20/20, 1:33 AM:
------------------------------------------------------------
We have just encountered a similar situation, but I think our approach is highly imperfect:
# With MT_DOP set to 64;
# A single profile can be very large. For the record, 1.4GB profile downloaded from ClouderaManager
# With growing number of queries, ServiceMonitor halt constantly. CPU usage can reach 90%, we believe it's busy with unserializing huge profiles.
# We found that the longest part is 'Detailed Fragments', which is less interesting than `Averaged Fragment` part.
# So we modified on `Coordinator.cc`, remove 'Detailed Fragment' and kept 'Averaged Fragment'
# Now CM seems to work well with 'Slimed Profiles', each profile's size is now XX KBs
!image-2020-11-20-09-25-08-082.png!
was (Author: fifteencai):
We have just encountered a similar situation, and we want to share this with the community:
# With MT_DOP set to 64;
# A single profile can be very large. For the record, 1.4GB profile downloaded from ClouderaManager
# With growing number of queries, ServiceMonitor halt constantly. CPU usage can reach 90%, we believe it's busy with unserializing huge profiles.
# We found that the longest part is 'Detailed Fragments', which is less interesting than `Averaged Fragment` part.
# So we modified on `Coordinator.cc`, remove 'Detailed Fragment' and kept 'Averaged Fragment'
# Now CM seems to work well with 'Slimed Profiles', each profile's size is now XX KBs
!image-2020-11-20-09-25-08-082.png!
> Utility to pretty-print thrift profiles at various levels
> ---------------------------------------------------------
>
> Key: IMPALA-9865
> URL: https://issues.apache.org/jira/browse/IMPALA-9865
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Attachments: image-2020-11-20-09-25-08-082.png
>
>
> The prototyping work in IMPALA-9382 revealed some hard trade-offs between having a full-fidelity text profile and readability.
> We want to have a text profile with less information by default so that it is more readable, and rely on the thrift profile for more detailed debugging. This would be easier if we provided a utility that can pretty-print a thrift profile at different levels of detail.
> This JIRA is to reduce the default level of pretty-printing for aggregated profiles, but provide a utility that can dump both the basic and full versions. My thought is to start off with the same 4 levels as explain, but maybe only implement 2 levels to start off with - basic and extended.
> The utility should be able to handle the same cases as bin/parse-thrift-profile.py (profile log and single profile in a file) and maybe print only a specified query from a profile log. We can use the DeserializeFromArchiveString() method that was removed in IMPALA-9381, then pretty-print the deserialised profile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org