[jira] [Updated] (YARN-3039) [Aggregator wireup] Implement ATS app-appgregator service discovery

["Junping Du (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Junping Du updated YARN-3039:
-----------------------------
    Attachment: YARN-3039-v2-incomplete.patch

Update a patch (haven't finished yet) to reflect some of discussions above.
Including:
    + maintain the app aggregator info in RMApp, with event model (done)
    + aggregator update in NM-RM heartbeat (done)
    + aggregator update in AM-RM allocation request/response (done)
    + Persistent aggregator update in RMStateStore (fix previous patch)
    + a new API to ResourceTrackerService to register app aggregator to RM (done)
    + adding a new protocol between aggregator and NM
      - new proto file (and proto structure for request and response)  -- done.
      - interfaces: (protocol, request, response)
         - AggregatorNodemanagerProtocol (done)
         - AggregatorNodemanagerProtocolPBClientImpl (TODO)
         - NMAggregatorService (TODO, server impl)
         - AggregatorNodemanagerProtocolPB (done)
         - AggregatorNodemanagerProtocolPBServiceImpl (done)
         - ReportNewAggregatorsInfoRequest/Response (and PBs) (done)
         - ReportNewAggregatorsInfoRequestPBImpl (done)
         - ReportNewAggregatorsInfoResponse (done)
         - ReportNewAggregatorsInfoResponsePBImpl (done)
         - AppAggregatorsMap (done)
           AppAggregatorsMapPBImpl (done)

Not included yet:
    + NM hosting new protocol
    + Aggregator call new protocol client
    + aggregator info get recovered during NM restart
    + make TimelineClient Observer pattern to observe the change of aggregator address.

Will update the proposal afterwards.

> [Aggregator wireup] Implement ATS app-appgregator service discovery
> -------------------------------------------------------------------
>
>                 Key: YARN-3039
>                 URL: https://issues.apache.org/jira/browse/YARN-3039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Junping Du
>         Attachments: Service Binding for applicationaggregator of ATS (draft).pdf, YARN-3039-no-test.patch, YARN-3039-v2-incomplete.patch
>
>
> Per design in YARN-2928, implement ATS writer service discovery. This is essential for off-node clients to send writes to the right ATS writer. This should also handle the case of AM failures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)