[jira] [Commented] (YARN-4821) Have a separate NM timeline publishing-interval

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230813#comment-15230813 ] 

Vinod Kumar Vavilapalli commented on YARN-4821:
-----------------------------------------------

bq. This proposal is simply to use a different publishing interval just for the timeline publishing
+1. We should completely decouple these two. If the publishing-interval is configured to be not a multiple of the monitoring-interval, the publisher could only look at the last N values from the monitor before the last cycle.

Can you also please have a read at YARN-3332 and see if you can organize code in a bit of independent way?

A related data point for deciding the interval itself - the Hadoop Metrics plugin pulls metrics from all of our daemons and pushes them out periodically - with a default value of 10 sec IIRC. This is the periodicity for most of the production clusters. Assuming adding container-metrics data to this still keep the total outgoing data to the same or immediate order of magnitude (say 250 metrics per NM + (50 containers * 50 metrics)), we should be okay with the same frequency. Anything more frequent will need careful benchmarking.

> Have a separate NM timeline publishing-interval
> -----------------------------------------------
>
>                 Key: YARN-4821
>                 URL: https://issues.apache.org/jira/browse/YARN-4821
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
>
> Currently the interval with which NM publishes container CPU and memory metrics is tied to {{yarn.nodemanager.resource-monitor.interval-ms}} whose default is 3 seconds. This is too aggressive.
> There should be a separate configuration that controls how often {{NMTimelinePublisher}} publishes container metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)