[jira] [Commented] (GIRAPH-574) Move Giraph Master node functionality to AppMaster or launch directly from AppMaster in YARN profile

["Eli Reisman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GIRAPH-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604468#comment-13604468 ] 

Eli Reisman commented on GIRAPH-574:
------------------------------------

Think aggregators, master compute, other things? also...

                
> Move Giraph Master node functionality to AppMaster or launch directly from AppMaster in YARN profile
> ----------------------------------------------------------------------------------------------------
>
>                 Key: GIRAPH-574
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-574
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Eli Reisman
>            Priority: Minor
>
> As folks read the Giraph on YARN code it is inevitable it will occur to someone "Well, if the job fails when the ApplicationMaster fails, could we move some or all of our Master task functions there and just call it master?"
> Yes. In two ways.
> One, we launch a dedicated master process marked as such with setup responsibilities, and we assess from the app master how the launch went. We keep launching "masters" until one takes. Then, we launch the workers.
> Another is to simply run MasterThread and associated stuff from the App Master directly, and when we know its up and running properly, only then does app master launch the workers.
> The YARN app master can be rebooted is designed to be a place for fault-tolerant "master node" stuff to happen. However, I think a larger purpose is to act as a meta-master for launching a DAG of jobs within the run of a single app master lifecycle. Or the app master cant act as any of these things, or something else I haven't thought of. The architecture is fairly malleable.
> This is not a requirement for us, and maybe not a good idea at all. This is just a placeholder JIRA to discuss and collect ideas since as I said above someone is going to bring it up ;)
> Thank you for reading.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira