You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2010/04/02 08:29:27 UTC
[jira] Created: (HAMA-244) Consider more simplified BSP programming
interface
Consider more simplified BSP programming interface
--------------------------------------------------
Key: HAMA-244
URL: https://issues.apache.org/jira/browse/HAMA-244
Project: Hama
Issue Type: Improvement
Components: bsp
Affects Versions: 0.2.0
Reporter: Edward J. Yoon
Fix For: 0.2.0
Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
However, currently there is no interface layer so users have to do all the works w/o any guide.
Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871115#action_12871115 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 20 new or modified tests.
javadoc -1. The javadoc tool appears to have generated 1 warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to introduce 18 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
http://hudson.zones.apache.org/hudson/job/Hama-Patch/ws/trunk/build/reports/findbugs/newPatchFindbugsWarnings.html
18 findbugs warnings occured but most of warnings are 'Unread field'.
I just committed this patch.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, HAMA-244_v03.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867779#action_12867779 ]
Hyunsik Choi commented on HAMA-244:
-----------------------------------
Actually, I know what is programming interface that you intend a few days ago. I already said that HAMA-244 is too general in April 12 because your description is insufficient.
https://issues.apache.org/jira/browse/HAMA-251?focusedCommentId=12856283&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12856283
And, your first detailed description is posted in May 13.
https://issues.apache.org/jira/browse/HAMA-244?focusedCommentId=12866958&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12866958
Before you posted the above comment, nobody knew what to do in HAMA-244. Thus, there was no order and dependency. In addition, I submitted my progress of HAMA-251.patch in May 13.
At least, I posted some detailed plan of HAMA-251 three times.
http://www.mail-archive.com/hama-dev@incubator.apache.org/msg02458.html
https://issues.apache.org/jira/browse/HAMA-251
https://issues.apache.org/jira/browse/HAMA-251?focusedCommentId=12865430&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12865430
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Chanwit Kaewkasi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867103#action_12867103 ]
Chanwit Kaewkasi commented on HAMA-244:
---------------------------------------
It seems now there are some rooms for me to tackle simplification.
Considering this, it would be possible to have a declarative wrapper for BSP to help people easier to adopt it.
Probably a DSL like,
def bsp = bsp(config: HAMA) {
jobName = "BSP test job"
inputPath = "/abcd"
outputPath = "/abcd"
inputFormat = MyInputFormat
outputFormat = MyOutputFormat
carriage = BSPProgram
}
run(bsp)
would do the job.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867341#action_12867341 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
>> I still don't know why HAMA-244 must be first. Do you mean that we have to solve issues in the order that you determine?
Nope.
At least, we'll need to provide an examples of how to programming matrix/graph operations using BSP on 0.2 release. And they should be considered from user standpoint. If it is figured out, a detail todo-list will be created. e.g., BSP programming interface, in/output formatter, job client API, job management system, and BSP examples. In other words, The work step of the roadmap and work break-down structure will be done.
To consider about them, I made this issue. and I think this should be first. Is it wrong?
But, you opened HAMA-251, and you doing work on 'job manager' and 'mater-slave deamon' w/ some BSP job handling example now. If I didn't ask about this and you to upload your progress yesterday, perhaps i still don't know what are you doing now.
Did you read it? http://www.mail-archive.com/hama-dev@incubator.apache.org/msg02453.html
I don't know why you want to implement all the detail things alone.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868186#action_12868186 ]
Hyunsik Choi commented on HAMA-244:
-----------------------------------
What is difference between you and me?
You added the patch that is just a slight modification of HAMA-251_1.patch (except for LocalBSPCluster) that you dismiss yesterday.
I cannot understand you, and I decided to leave the hama.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Attachment: javadocwarnings.patch
- Added getClusterStatus() method
- Removed 'test' command
- Removed all javadocs warnings
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, HAMA-244_v03.patch, javadocwarnings.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyunsik Choi updated HAMA-244:
------------------------------
Attachment: HAMA-244_v03.patch
I attached the patch. This patch includes HAMA-244's patches and HAMA-251_01.patch. In addition, this patch includes the following modifications:
* Renamed BSPJobConf to BSPJob.
* Removed setInputPath, setOutputPath in order to enable concrete classes of both InputFormat and OutputFormat to designate various targets.
* Some trivial things.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, HAMA-244_v03.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867073#action_12867073 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
I mean, this issue should be first. Rest are later issue.
for example,
As you see this http://blog.udanax.org/2010/05/pagerank-implementation-using-bsp.html, the code is divided into two programs. Perhaps, there is no way to combine them w/o some kind of parser. In BSPLib's case, they wrapping the parallel part with bsp_begin() and bsp_end() methods. So, their code always look like a serial code. Then, should we add this functionality to provide simple API?
We should figure out these things. So, I made this issue and moved your issue to a sub-task. Do you understand that?
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857607#action_12857607 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
Hopefully, it will possible to run BSP programs with any executable or script in the future; like a hadoop streaming. for example,
$HAMA_HOME/bin/hama jar ./hama-bsprun.jar -input myInputDirs -output myOutputDir ... myBSP.class
So in conclusion, i'd like to say that we have to design the job configuration/submission interface like their style and overall ambience.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Fix For: 0.2.0
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867070#action_12867070 ]
Hyunsik Choi commented on HAMA-244:
-----------------------------------
http://markmail.org/search/?q=To+committers+list:org.apache.incubator.hama-dev#query:To%20committers%20list%3Aorg.apache.incubator.hama-dev+page:1+mid:set4leqwvsbhg4ys+state:results
We already discussed 0.2 plan. The issues that we discussed was written in http://wiki.apache.org/hama/RoadMap. Subscribers of hama-dev mailling list may know what we are doing for v0.2. I also wrote my plan for HAMA-251. But, I have not received any comment.
https://issues.apache.org/jira/browse/HAMA-251?focusedCommentId=12865430&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12865430
If you mean that documentation is not sufficient. In open source communities, the following sequence is not bad: 1) discussion 2) implementation 3) documentation. It is because 1) many people contribute to open source projects due to only fun 2) actually many issues in open sources are addressed in that sequence.
If we have some problem, that may be we are lack of communication.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867963#action_12867963 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
Regarding in/output format for BSP computing framework, I think we should consider about compatibility with M/R framework.
And it must added to 0.2 plan.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872763#action_12872763 ]
Hyunsik Choi commented on HAMA-244:
-----------------------------------
+1
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, HAMA-244_v03.patch, javadocwarnings.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Status: Patch Available (was: Open)
This patch seems getting bigger.. if tests run well, let's commit to trunk.
Testing HAMA-244_v03.patch through hudson.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, HAMA-244_v03.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-244:
-----------------------------------
Assignee: Edward J. Yoon
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Attachment: HAMA-244_v01.patch
Below is my imagination of how to configure the BSP job and launching it.
To avoid some collision with mapreduce.JobConf, I would like to rename the JobConf to BSPJobConf.
----
HamaConfiguration conf = new HamaConfiguration();
// BSP job configuration
BSPJobConf bsp = new BSPJobConf();
// Set the job name
bsp.setJobName("BSP test job");
// Set in/output path and formatter
bsp.setInputPath(conf, new Path("input path"));
bsp.setOutputPath(conf, new Path("output path"));
bsp.setInputFormat(MyInputFormat.class);
bsp.setOutputFormat(MyOutputFormat.class);
// Set the BSP code
bsp.setCarriage(BSPProgram.class);
BSPJobClient.runJob(bsp);
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
Javadoc warnings also removed.
As described wiki/Architecture, User Interface is defined.
I just closed this issue.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, HAMA-244_v03.patch, javadocwarnings.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867962#action_12867962 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
OK, I see.
But, We haven't decided the architecture or the architecture is not fully shared. I can't accept your 'Job Manager' patch. -1
And, I don't mean technological milestones or whatever, but I'm talking about collaborative development. The main problem is I don't see the cooperative spirit from you.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Attachment: HAMA-244_v02.patch
attach trivial patch before changing seat.
Hyunsik:
in addition, below user scenario of job configuration and job submission show why we need to add job manager HAMA-251. doesn't it?
Strictly speaking, You should have to work for it before considering about Master and GroomServer. All design consideration will be depended on this.
P.S. We need to consider about how to break down the issue.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868446#action_12868446 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
I think all things of BSP architecture and design should be shared/figured out before commit them. The final output of this issue will be a document of architecture and design. (NOT core implementations)
----
I requested you to upload/share your progress several times (via mail privately and via IRC) but you didn't, until I assigned this issue to me, and decided to touch the BSP territory to make some progress.
Hyunsik, I don't think It is important who is leading BSP progress or who is contributing more. As I mentioned above, I don't mean technological milestones or whatever.
>> I cannot understand you, and I decided to leave the hama.
Perhaps, IMO, 'build your own BSP project' will be better than contribution to ASF if you want to assert your authority/independence/rights.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867062#action_12867062 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
so i asked whether it is duplicated when you opened that issue.
before implementing JMS (Job Management System) or detailed others, I think we should figure out what is essential things for 0.2 plan. currently, I see the copy&pasted codes from mapreduce project but, there is no plan, document, and designed interface. then nobody can know what are we doing now, how to help us and how to collaborating with us.
as i mentioned HAMA-251, if we figure out how can use Hama BSP based on TDD style development method, the essential things, skeleton code and interface will be clarified.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867134#action_12867134 ]
Hyunsik Choi commented on HAMA-244:
-----------------------------------
https://issues.apache.org/jira/browse/HAMA-251?focusedCommentId=12865430&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12865430
As I mentioned in the above comment, BSPMaster is responsible to do the following:
1. Maintaining groom server status.
2. Controlling super steps in a cluster.
3. Maintaining job progress information.
4. Assigning tasks to groom servers.
5. Scheduling Job
6. Controlling fault.
7. Providing users with the cluster control interface.
8. Disseminating execution class across groom servers. (newly added)
Among them, now I plan to implement 1,3,4, 5 and 8. For these works, I made only the Work class (i.e., a dummy class for BSP computing) that has no method. A user needs to set only the class to a simple job instance in order to submit a job. It is enough for me to implement above features. Among above 5 features that I plan in HAMA-251, I already implemented most features although they are simple and some of the classes are borrowed from MapReduce.
For instance, the 8th feature plays a role to copy both job's meta data submitted and a jar file (containing an executable BSP class and its related classes) to a bsp master's disk from clients's disk and then to copy the bsp.jar to all groom servers' disk. This way has used in many cluster architectures, such as Condor, Clustera, and MapReduce.
I still don't know why HAMA-244 must be first. Do you mean that we have to solve issues in the order that you determine?
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Attachment: 0524.patch
attach patch include my local PC changes
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: 0524.patch, HAMA-244_v01.patch, HAMA-244_v02.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-244) Consider more simplified BSP programming
interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-244:
--------------------------------
Attachment: patch.txt
attach my patch before moving seat.
LocalBSPCluster and MiniCluster are added.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch, patch.txt
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867111#action_12867111 ]
Edward J. Yoon commented on HAMA-244:
-------------------------------------
Thanks For nice information!!
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch, HAMA-244_v02.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-244) Consider more simplified BSP
programming interface
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866966#action_12866966 ]
Hyunsik Choi commented on HAMA-244:
-----------------------------------
Good! This work is somewhat overlapped to small parts of HAMA-251 that I've worked.
However, I think that designing interface should be discussed in this issue.
Today evening, I'll submit my progress of HAMA-251. Then, let's discuss job submission interface.
> Consider more simplified BSP programming interface
> --------------------------------------------------
>
> Key: HAMA-244
> URL: https://issues.apache.org/jira/browse/HAMA-244
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.2.0
>
> Attachments: HAMA-244_v01.patch
>
>
> Perhaps, every BSP program will be consists of three steps such as, initialization, SPMD code implementation, and finalization from developer angle.
> However, currently there is no interface layer so users have to do all the works w/o any guide.
> Would be nice if it allow to use a more simplified interface.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.