[jira] [Commented] (IMPALA-9382) Prototype denser runtime profile implementation

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265777#comment-17265777 ] 

ASF subversion and git services commented on IMPALA-9382:
---------------------------------------------------------

Commit 5fddbb569a68bd438d9f4847481c8db1fce5b67c in impala's branch refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5fddbb5 ]

IMPALA-9865: part 2/2: add verbosity to profile tool

Adds a --profile_verbosity option for impala-profile-tool with
the following levels:
* 0: minimal
* 1: legacy - matches old output, this is the default still
* 2: default - basic descriptive stats, used for V2 profile.
* 3: extended
* 4: full

This will help with transition to the V2 profile because we
can have a nice, high-level, readable text profile by default
with the option to produce more detailed profiles and alternate
views of the profile from the thrift profile.

Use the profile version in impala-profile-tool to dump the
more verbose output for the V2 profile while preserving the
same output for the legacy profile.

Reduce verbosity of v2 profile output - only include mean/min/max
by default. I intend to refine the output at the different
verbosity levels for the v2 profiles further as part of IMPALA-9382,
it is still fairly noisy.

Fix output with/without gen_experimental_profile - there
was a small difference in that the summary stats were not
output in the averaged profile.

Testing:
* Add an end-to-end test that generates output for a small
  profile log and compares against expected files.
* Tweak other profile tests to reflect changes to output.

Change-Id: I82618a813e29af7996dfaed78873b2a73bc0231d
Reviewed-on: http://gerrit.cloudera.org:8080/16881
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Prototype denser runtime profile implementation
> -----------------------------------------------
>
>                 Key: IMPALA-9382
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9382
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Major
>         Attachments: profile_504b379400cba9f2_2d2cf00700000000, tpcds_q10_profile_v1.txt, tpcds_q10_profile_v2.txt, tpcds_q10_profile_v2.txt
>
>
> RuntimeProfile trees can potentially stress the memory allocator and use up a lot more memory and cache than is really necessary:
> * std::map is used throughout, and allocates a node per map entry. We do depend on the counters being displayed in-order, but we would probably be better of storing the counters in a vector and lazily sorting when needed (since the set of counters is generally static after Prepare()).
> * We store the same counter names redundantly all over the place. We'd probably be best off using a pool of constant counter names (we could just require registering them upfront).
> There may be a small gain from switching thrift to using unordered_map, e.g. for the info strings that appear with some frequency in profiles.
> However, I think we need to restructure the thrift representation and in-memory representation to get significant gains.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org