[jira] [Updated] (SAMZA-58) Use YARN's *Async API

["Zhijie Shen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhijie Shen updated SAMZA-58:
-----------------------------

    Attachment: SAMZA-58.1.patch

I've created a patch, which use AMRMClientAsync to replace AMRMClient. In this case YarnAppMaster is no longer required, but I keep the loop, because I want to block SamzaAppMaster termination until all the listeners are done.

Two related issues that I've found, but haven't fixed in this patch:
1. The heartbeat interval was set to 1s by hard code. I kept it, but ideally it's good to be configurable. Thoughts?
2. Samza will always report progress = 0 to YARN. Though it doesn't do harm to the application execution, but users are going to be confused when monitoring Samza jobs' progress

NMClient can be replaced by NMClientAsync as well. However, we need to think of the logic when startContainer becomes nonblocking. Therefore, I prefer to solve it separately. Thoughts?

> Use YARN's *Async API
> ---------------------
>
>                 Key: SAMZA-58
>                 URL: https://issues.apache.org/jira/browse/SAMZA-58
>             Project: Samza
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 0.6.0
>            Reporter: Chris Riccomini
>            Assignee: Zhijie Shen
>         Attachments: SAMZA-58.1.patch
>
>
> YARN 2.2.0 has a nice Async API for clients and AMs. This API didn't exist when we did the initial YARN integration for Samza. We should upgrade Samza to use these new APIs.
> The API is loosely based off Samza's own AM code, so we can probably strip out a lot of it (YarnAppMaster, mainly), and switch everything over to the call-back based API.
> For details, see:
> https://issues.apache.org/jira/browse/YARN-417
> This new API is used in DistributedShell now, so we can use that for testing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)