[jira] [Commented] (SAMZA-680) Invert the JobCoordinator and AM logic

["Jagadish (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202311#comment-15202311 ] 

Jagadish commented on SAMZA-680:
--------------------------------

Review board up at :

 https://reviews.apache.org/r/44920/


> Invert the JobCoordinator and AM logic
> --------------------------------------
>
>                 Key: SAMZA-680
>                 URL: https://issues.apache.org/jira/browse/SAMZA-680
>             Project: Samza
>          Issue Type: Sub-task
>            Reporter: Naveen Somasundaram
>            Assignee: Jagadish
>
> Currently, the YARN AM pretty much dictates how the JobCoordinator works. This creates lot of inflexibility on how we can control failures or even integrate with new system (Mesos). 
> For e.g., https://issues.apache.org/jira/browse/SAMZA-465?focusedCommentId=14522043&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14522043
> It would nice to invert the logic to JobCoordinator, and JobCoordinator has a global view of container failures, config changes etc. This simplifies lot of implementation specifics (for e.g., dynamic scaling becomes easier). 
> A another nice to have, would be make this logic pluggable. 
> {noformat}
> e.g.,
> job.clustermanager = org.apache.samza.YarnClusterManager
> job.clustermanager = org.apache.samza.MesosClusterManager
>  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)