You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Hyunsik Choi (JIRA)" <ji...@apache.org> on 2010/04/13 04:25:49 UTC
[jira] Created: (HAMA-251) Add job manager to BSPMaster
Add job manager to BSPMaster
----------------------------
Key: HAMA-251
URL: https://issues.apache.org/jira/browse/HAMA-251
Project: Hama
Issue Type: Improvement
Affects Versions: 0.2.0
Reporter: Hyunsik Choi
Assignee: Hyunsik Choi
Fix For: 0.2.0
In regard to BSPMaster, I plan to design as follows:
* BSPMaster is responsible to manage jobs submitted by clients.
* BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
* BSPMaster is responsible to control job progress and BSPPeers actions.
It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyunsik Choi updated HAMA-251:
------------------------------
Component/s: bsp
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867355#action_12867355 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
Ok, I'll move BSPExample to test package. As you mentioned, this patch is getting bigger. I'm going to convert this subtask to an issue in order to divide it into several small subtasks.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856281#action_12856281 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
I made a wiki page ( http://wiki.apache.org/hama/BSPMaster ). The discussion will be documented into this wiki page.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyunsik Choi resolved HAMA-251.
-------------------------------
Resolution: Fixed
The attached patch was merged with HAMA-244's patch and was committed. Now, job manager achieves the below two features among three features that was planned. Only two features earlier are committed because they are very essential.
* BSPMaster is responsible to manage jobs submitted by clients.
* BSPMaster is responsible to control job progress and BSPPeers actions.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856343#action_12856343 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
Ok, I'll attach an UML diagram that represents BSPMaster and its related things.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-251:
--------------------------------
Parent: HAMA-244
Issue Type: Sub-task (was: Improvement)
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyunsik Choi updated HAMA-251:
------------------------------
Parent: (was: HAMA-244)
Issue Type: New Feature (was: Sub-task)
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865430#action_12865430 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
Before I complete the UML diagram and the design of the job manager, I would like to discuss my plan with you. Overall, the job management system of BSP is resemble to that of MR. The below summary includes not only a job management but also an overview of bsp cluster. It is not completed yet, and I'll update this summary later.
----
Summary:
One BSP cluster consists of one BSPMaster, multiple GroomServer and one or more zookeeper servers in a network environment. Like JobTracker of MR, the bsp master plays the following roles:
* Maintaining groom server status.
* Controlling supersteps in a cluster.
* Maintaining job progress information.
* Assining tasks to groom servers.
* Job Scheduing
* Controlling fault.
* Providing users with the cluster control interface.
Like TaskTracker of MR, a Groom Server (shortly referred to as groom) is a process that performs bsp tasks assigned by BSPMaster. Each groom contacts the BSPMaster, and it takes assigned tasks and reports its status by means of periodical piggybacks with BSPMaster. Each groom is designed to run with HDFS or other distributed storages. Basically, a groom server and a data node should be run on one physical node.
A BSP Master and multiple grooms are started by the script. Then, the bsp master starts up with a RPC server for groom servers. Groom servers starts up with a BSPPeer instance - later, BSPPeer needs to be integrated with GroomServer - and a RPC proxy to contact the bsp master. After started, each groom periodically sends a heartbeat message that encloses its groom server status, including maximum task capacity, unused memory, and so on.
Each time the bsp master receives a heartbeat message, it brings up-to-date groom server status - the bsp master makes use of groom servers' status in order to effectively assign tasks to idle groom servers - and returns a heartbeat response that contains assigned tasks and others actions that a groom server has to do. For now, I'll develop a FIFO job scheduler and very simple task assignment algorithms.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856287#action_12856287 ]
Edward J. Yoon commented on HAMA-251:
-------------------------------------
HAMA-244 aimed to start the discuss from the user stories, as a TDD style.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856283#action_12856283 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
I think that this issues is usually related to BSP package internal. It seems to be not relevant to programming interface which disclose to users. If you don't agree, HAMA-244 is too general.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856285#action_12856285 ]
Edward J. Yoon commented on HAMA-251:
-------------------------------------
It may be so but, IMO, the design of the Job Object and the submit API also should be discussed during consider about overall architecture. If you already have some plans, I would suggest you to work firstly on UML diagrams.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856282#action_12856282 ]
Edward J. Yoon commented on HAMA-251:
-------------------------------------
This issue belongs to HAMA-244, and HAMA-247
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867427#action_12867427 ]
Edward J. Yoon commented on HAMA-251:
-------------------------------------
>> Ok, I'll move BSPExample to test package.
Quick question.
If you move BSPExample to test package, then what is the difference with HAMA-244 patch?
If, when some changes occured on BSPJob object during work for HAMA-244, how to merge them?
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867321#action_12867321 ]
Edward J. Yoon commented on HAMA-251:
-------------------------------------
Nice patch.
1) Would be nice if BSPExample is moved to another location e.g., test package. or it could be combined with HAMA-244.
2) I suggest you to stop the patch getting bigger. You can leave the method implementation empty (w/ only comment) and assign to volunteers after break down issue.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867434#action_12867434 ]
Edward J. Yoon commented on HAMA-251:
-------------------------------------
Hmm.
1) How to programming BSP with HAMA to processing data on HDFS?
2) How to configure its BSP job?
3) And, How to submit Job through client interface?
4) Then, Who manage the Job execution?
Have we disccused these before? Do you still think these are not related?
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyunsik Choi updated HAMA-251:
------------------------------
Attachment: HAMA-251_01.patch
I attached my progress. It is still working.
After executed start-dfs.sh of MR and start-bsp.sh of hama sequentially, you can submit a test job to the bsp cluster as follows:
bin/hama example
org.apache.hama.examples.BSPExample is the example class that the above command executes. You can see information of the submitted job from hama-bspmaster's log.
In addition, you can check groom servers' status from hama-bspmaster's log.
If you get any error or have any question, please report to here.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: Sub-task
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867790#action_12867790 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
1) How to programming BSP with HAMA to process data on HDFS?
A task is paired with an input split. The input split is a HDFS block. The more HDFS block is close to the reader, the more network cost is reduced. Thus, the scheduler should assign as many tasks paired with local block as possible. For this, BSPMaster needs to hold network topology and gets groom servers (with a data node) in which certain DFS blocks reside during assigning tasks to groom servers. This explain is scope of this issue.
2) How to configure its BSP job?
In my progress, job configuration is performed in BSPJob and JobContext. BSPjob is a subclass of JobContext. JobContext has a Configuration instance. Like new MapReduce API, BSPJob is a writable job instance, and JobContext is readonly view. Parameters that users set are contained in the Configuration instance in JobContext. As you know, A Configuration instance can be written into a xml file, and it can be restored to a Configuration instance.
You can check the writeXml() in Configuration, BSPJob and JobContext.
3) How to submit Job through client interface?
Do you mean command line interface as the below comment?
https://issues.apache.org/jira/browse/HAMA-244?focusedCommentId=12857607&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12857607
If so, it is somewhat trivial. If we have internal implementation, we can make any type interfaces. In addition, I made a really simple job submission interface for test as you can see BSPJob and BSPExample. Right now, it is enough to test job submission.
If you mean how jobs is disseminated, the job submission is through BSPJob.java. In order to submit job, a user firstly needs to set some parameters (e.g., Input Path, Output Path, and class names for java reflection) to a BSPJob instance; my progress supports only to set a class name for BSP computation. When the user submit the BSPJob instance, BSPJob contacts the BSPMaster, and then it get new job id from BSPMaster. The BSPJob packs configurations with JobID to a xml file and copies it to BSPMaster's local disk via DFS remote copy. A Jar file that contains the job class is also copied to BSPMaster's local disk.
Upon receiving the submitted job, BSPMaster adds the job to a simple scheduler. The scheduler assigns tasks to groom servers via responses of heartbeat. So far, I have designed that each groom server has only one task. For optimal scheduling, BSPMaster needs status of groom servers. The status of groom servers have to include the max task capacity and the number of running task for each groom servers. For this, BSPMaster collects the statuses of all groom servers that join the BSP cluster through heartbeat. Finally, the task information is transmitted to groom servers via the response of heartbeat.
You can check the heartbeat(), submitJob() in BSPJob, and submitJobInternal() in JobClient.
4) Then, Who manage the Job execution?
In overall, BSPMaster controls all groom servers. BSPMaster sends a lunch action to groom servers through responses of heartbeat. Then, each groom runs BSPPeer with a job class. During processing job, each groom sends status of a running task to BSPMaster via heartbeat. BSPMaster keeps status of all running tasks in a map data structure. Later, this information can be provided to users via any kind interfaces, such as terminal and web. If a user kills certain job, the user can send a kill action via BSPJob in the same way. These communications are performed with Hadoop RPC. The patch already contains some of these interaction.
You can check the heartbeat () in BSPMaster and transmitHeartBeat in GroomServer.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-251) Add job manager to BSPMaster
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867432#action_12867432 ]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
Merging codes may be not easy while we proceed with two issues respectively.
It would be better to commit each patch individually. Then, we should create new issue to merge them. Actually, I only need BSPExample for job submission test right now. If it will be removed later, I'm ok.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.