You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2014/04/03 00:25:16 UTC
[jira] [Commented] (SAMZA-58) Use YARN's AMRMClientAsync client
library
[ https://issues.apache.org/jira/browse/SAMZA-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958259#comment-13958259 ]
Chris Riccomini commented on SAMZA-58:
--------------------------------------
bq. 1. The heartbeat interval was set to 1s by hard code. I kept it, but ideally it's good to be configurable. Thoughts?
Totally agree. I think we can add a setting in YarnConfig to tweak this.
bq. 2. Samza will always report progress = 0 to YARN. Though it doesn't do harm to the application execution, but users are going to be confused when monitoring Samza jobs' progress
Yeah, I wasn't sure what to do there. YARN doesn't really handle infinitely running jobs well (or at least it didn't at the time I wrote this). Do you have any idea what the recommended approach is now? I've heard some people just put random numbers in there every time. Others I've heard slide it up and down like an animation.
> Use YARN's AMRMClientAsync client library
> -----------------------------------------
>
> Key: SAMZA-58
> URL: https://issues.apache.org/jira/browse/SAMZA-58
> Project: Samza
> Issue Type: Bug
> Components: yarn
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Zhijie Shen
> Attachments: SAMZA-58.1.patch
>
>
> YARN 2.2.0 has a nice Async API for clients and AMs. This API didn't exist when we did the initial YARN integration for Samza. We should upgrade Samza to use these new APIs.
> The API is loosely based off Samza's own AM code, so we can probably strip out a lot of it (YarnAppMaster, mainly), and switch everything over to the call-back based API.
> For details, see:
> https://issues.apache.org/jira/browse/YARN-417
> This new API is used in DistributedShell now, so we can use that for testing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)