[jira] [Commented] (IMPALA-7550) Create documentation for all profile fields

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996903#comment-16996903 ] 

ASF subversion and git services commented on IMPALA-7550:
---------------------------------------------------------

Commit f2f348c0f93208a0f34c33b6a4dc82f4d9d4b290 in impala's branch refs/heads/master from Jiawei Wang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f2f348c ]

IMPALA-7550: Add documentation to profile counters

This change changes the way developers define profile counters by
generating counters from a counter registry. All the profile counters
will be register there first and then used in the same way as before.
By doing so, we will be able to manage profile counters in a way that
we can define profile counters documentation. For example:

Declaration:
PROFILE_DEFINE_COUNTER(NumBackends, STABLE_HIGH, TUnit::UNIT,
    "Number of backends running this query.");

Initialization:
COUNTER_SET(PROFILE_NumBackends.Instantiate(query_profile_), num_backends);

This shall be how we define a NumBackends counter. It follows with
its significance, type, description in the declaration part.

Users now will be able to view profile counters documentation under
query_profile page, there is a Profile Documentation button which
leads to /profile_docs.

More details:
This commit did the following refactors on profile counters.
1. Add a singleton registry for runtime profile counters prototypes,
similiar to what Kudu does for metrics. This allows us to generate
profile documentation for all counters from the code. We add
/profile_docs and a correspoding UI for the documentation of profile
counters.

2. Profile counters are also annotated with their significance to users.
* STABLE_HIGH - High level and stable counters, always useful for measuring
query performance and status. Counters that everyone is interested. should
rarely change and if it does we will make some effort to notify users.

* STABLE_LOW - Low level and stable counters. Interesting counters to monitor
 and analyze by machine. It will probably be interesting under some
 circumstances for users.

* Unstable - Unstable but useful. Useful to understand query performance,
but subject to change, particularly if the implementation changes.
E.g. MaterializeTupleTimer

* Debug -  Debugging counters. Generally not useful to users of Impala,
 the main use case is low-level debugging. Can be hidden to reduce noise
 for most consumers of profiles.

3. We have around 250 counters. This commit did the replacement in
scan-node and hdfs-scan-node-base and coordinator.

Concers:
The downside is that we will have duplicate comments of query profiles
both in the header file and the .cc file.
Additionally a (arguably good) limitation is that profile counter names
need to be unique.

Change-Id: Idc03faddb27754001290bb6d899840e2cbe7ccb7
Reviewed-on: http://gerrit.cloudera.org:8080/14776
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Create documentation for all profile fields
> -------------------------------------------
>
>                 Key: IMPALA-7550
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7550
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Docs
>    Affects Versions: Impala 3.1.0
>            Reporter: Lars Volker
>            Assignee: Jiawei Wang
>            Priority: Critical
>              Labels: supportability
>
> It would be great to have some documentation for all profile fields. Currently users often have to look at the source code to figure out the exact meaning of a field. Instead we should investigate ways to generate documentation on the fields, e.g. by cleaning and automatically extracting relevant comments from the source code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org