[jira] [Resolved] (IMPALA-6870) SummaryStatsCounter should be included in averaged profile

["Tim Armstrong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-6870.
-----------------------------------
    Fix Version/s: Impala 4.0
       Resolution: Fixed

commit 9429bd779de986d3e61858bef7e258bd73a2cacd
Author: Tim Armstrong <ta...@cloudera.com>
Date:   Sun May 17 16:37:46 2020 -0700

    IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
    
    This reworks the status reporting so that serialized
    AggregatedRuntimeProfile objects are sent from executors
    to coordinators. These profiles are substantially denser
    and faster to process for higher mt_dop values. The aggregation
    is also done in a single step, merging the aggregated thrift
    profile from the executor directly into the final aggregated
    profile, instead of converting it to an unaggregated profile
    first.
    
    The changes required were:
    * A new Update() method for AggregatedRuntimeProfile that
      updates the profile from a serialised AggregateRuntimeProfile
      for a subset of the instances. The code is generalized from the
      existing InitFromThrift() code path.
    * Per-fragment reports included in the status report protobuf
      when --gen_experimental_profile=true.
    * Logic on the coordinator that either consumes serialized
      AggregatedRuntimeProfile per fragment, when
      --gen_experimental_profile=true, or consumes a serialized
      RuntimeProfile per finstance otherwise.
    
    This also adds support for event sequences and time series
    in the aggregated profile, so the amount of information
    in the aggregated profile is now on par with the basic profile.
    
    We also finish off support for JSON profile. The JSON profile is
    more stripped down because we do not need to round-trip profiles
    via JSON and it is a much less dense profile representation.
    
    Part 3 will clean up and improve the display of the profile.
    
    Testing:
    * Add sanity tests for aggregated runtime profile.
    * Add unit tests to exercise aggregation of the various counter types
    * Ran core tests.
    
    Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca
    Reviewed-on: http://gerrit.cloudera.org:8080/16057
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>


> SummaryStatsCounter should be included in averaged profile
> ----------------------------------------------------------
>
>                 Key: IMPALA-6870
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6870
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.12.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Major
>              Labels: observability
>             Fix For: Impala 4.0
>
>
> Summary stats like FooterProcessingTime don't show up in the averaged fragment.  We should be able to merge these counters to produce overall statistics:



--
This message was sent by Atlassian Jira
(v8.3.4#803005)