You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Gustavo Anatoly (JIRA)" <ji...@apache.org> on 2015/05/27 20:32:18 UTC
[jira] [Commented] (SAMZA-680) Invert the JobCoordinator and AM
logic
[ https://issues.apache.org/jira/browse/SAMZA-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561448#comment-14561448 ]
Gustavo Anatoly commented on SAMZA-680:
---------------------------------------
Hi, [~naveenatceg]
I was taking a look on the implementation and how to proceed it. But I have some doubts:
1. JobCoordinator would extend AMRMClientAsync.CallbackHandler?
2. If, yes in the previous question. How to proceed with classes related to samza-yarn, once
we have that import YarnAppMasterListener to pass it to implemented methods required by AMRMClientAsync.CallbackHandler?
Thanks.
> Invert the JobCoordinator and AM logic
> --------------------------------------
>
> Key: SAMZA-680
> URL: https://issues.apache.org/jira/browse/SAMZA-680
> Project: Samza
> Issue Type: Improvement
> Reporter: Naveen Somasundaram
>
> Currently, the YARN AM pretty much dictates how the JobCoordinator works. This creates lot of inflexibility on how we can control failures or even integrate with new system (Mesos).
> For e.g., https://issues.apache.org/jira/browse/SAMZA-465?focusedCommentId=14522043&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14522043
> It would nice to invert the logic to JobCoordinator, and JobCoordinator has a global view of container failures, config changes etc. This simplifies lot of implementation specifics (for e.g., dynamic scaling becomes easier).
> A another nice to have, would be make this logic pluggable.
> {noformat}
> e.g.,
> jobcoordinator.core=com.linkedin.YarnJobCoordinator
> or
> jobcoordinator.core=com.linkedin.MesosJobCoordinator
> or
> jobcoordinator.core=com.linkedin.SomeCustomJobCoordinator
> {noformat}
> It would be nice for users who operate it as a service, to inject their own logic (e.g., Users extending, say, YanJobCoordinator to their own custom class and do some special logging/actions before container failure)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)