[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data

["Zhijie Shen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhijie Shen updated YARN-953:
-----------------------------

    Attachment: YARN-953.5.patch

I updated the a new patch, which coordinates the change of writer interface. The basic design is not changed: we still have a RMApplicationHistoryWriter, which handles writing requests from RM on different threads asynchronously. There're some other major changes:

1. Instead of handling all the writing events in one separate thread, I define a dispatcher vector to improve concurrency, and also ensure the events of one application is scheduled in the same thread. Therefore, the writing events of different applications will be processed concurrently, while events of the same application will be processed in the order where they are scheduled (It's important to ensure the events scheduled before applicationFinished to be processed first).

2. Make sure applicationFinished is called after all applicationAttemptsFinished, especially in the killing case, where RMApp moves to the final state before RMAppAttempt.

3. Improve the test cases.

4. Fix the break of other tests in RM project.

There's something to be handled separately:

1. We need to make RMContainer have more information to fill ContainerHistoryData. It's going to be done in YARN-974

2. Like RMStateStore, RMApplicationHistoryWriter needs to flush all the pending events given RM stops. We can make use of the update in YARN-1121 later.

> [YARN-321] Change ResourceManager to use HistoryStorage to log history data
> ---------------------------------------------------------------------------
>
>                 Key: YARN-953
>                 URL: https://issues.apache.org/jira/browse/YARN-953
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Zhijie Shen
>         Attachments: YARN-953-5.patch, YARN-953.1.patch, YARN-953.2.patch, YARN-953.3.patch, YARN-953.4.patch, YARN-953.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)