[jira] [Commented] (YARN-5638) Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery

["Li Lu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494688#comment-15494688 ] 

Li Lu commented on YARN-5638:
-----------------------------

Updated the description of this JIRA. What we need here is not a new type of "collector id", but to store timestamp data in the RMs and NMs for the collectors. This can address the problem when we rebuild collector status for a new active rm, as discussed in YARN-3359: 

bq. when one application has two different attempts running (due to some network problems, for example) and the RM is trying to rebuild collector status, the RM needs to know which collector is for the latest app attempt and which one is for the stale attempt.

We do not necessarily need to associate collectors to application attempts. Actually, according to timeline server v2 design, we should only associate app collectors to applications. However, when maintaining collector data in RMs and NMs, we can store the timestamp of each collector. In this way, when the RM needs to rebuild collector status, it can gather all known collector data from NMs, use the timestamp to decide the most recent state of the collectors, and then rebuild all states. 

> Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery
> -----------------------------------------------------------------------------------------------------
>
>                 Key: YARN-5638
>                 URL: https://issues.apache.org/jira/browse/YARN-5638
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>
> As discussed in YARN-3359, we need to further identify timeline collectors' creation order to rebuild collector discovery data in the RM. This JIRA proposes to use <rm_timestamp, logical_version_number> to order collectors for each application in the RM. This timestamp can then be used when a standby RM becomes active and rebuild collector discovery data. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org