[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.

["Vineet Garg (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461760#comment-16461760 ] 

Vineet Garg commented on TEZ-3911:
----------------------------------

[~ewohlstadter] That was my mistake initializing it to 0. I have fixed that in the latest patch. Thanks for pointing it out.
Latest patch(3) adds config flag as well as apis to retrieve min/max. [~gopalv] suggested to not use config flag and enable new aggregations by default. [~gopalv] Can you expand on your second comment about adding extra abstract class to handle this. It sounds like my approach might not be correct since. Please take a look at the latest patch I uploaded.


> Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
> -------------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3911
>                 URL: https://issues.apache.org/jira/browse/TEZ-3911
>             Project: Apache Tez
>          Issue Type: New Feature
>            Reporter: Eric Wohlstadter
>            Assignee: Vineet Garg
>            Priority: Critical
>             Fix For: 0.9.next
>
>         Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, TEZ-3911.003.patch
>
>
> Consumers of HistoryLoggingService reported counters are currently required to compute any task-level aggregations other than "sum". This is inefficient as Tez is already "scanning" over this data. Computing incremental aggregates shouldn't require additional scans by ATS consumers. 
> Provide an option for Task counter aggregations other than "sum". Computation of these extra counters can be turned on/off.
> The option will generate "synthetic" counters at final aggregation time for reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. 
> Only incremental aggregations will be supported (min/max/avg). Aggregation computation will be folded into the existing "aggregation loop" beginning at VertexImpl.incrTaskCounters.
> Extra aggregations will only be supported during final counter aggregation.
> Aggregations will only include the "bestAttempt" for each task.
> A design doc will be provided.
> Because final task aggregation holds a lock, a performance report will be provided. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)