You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2010/08/10 07:34:16 UTC
[jira] Created: (HAMA-282) Add superstep counter
Add superstep counter
---------------------
Key: HAMA-282
URL: https://issues.apache.org/jira/browse/HAMA-282
Project: Hama
Issue Type: Sub-task
Reporter: Edward J. Yoon
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HAMA-282) Add superstep counter
Posted by Filipe David Manana <fd...@apache.org>.
On Fri, Aug 20, 2010 at 8:04 AM, Edward J. Yoon <ed...@apache.org>wrote:
> Dear fdmanana, Thanks for your interest.
>
You're welcome.
It's been a while now that I got interested in Hama. However I haven't yet
looked carefully to its code.
>
> > Is this a good feature to start with?
>
> Yes, I think so. BTW, you need to wait few days for this. The reason
> is that the number of supersteps should be counted by the master
> server but I'm heavily touching code of master/slave server for
> HAMA-276 now. Moreover, there is a lot of hard coded path and TODO
> things.
>
> However, It is OK to contribute only part of superstep counting.
> Please see the part of BSPPeer.enterbarrier() method and heartbeat
> messages between the master/slave servers.
> ----
>
> And, let me explain more detail about the current status of HAMA TRUNK ....
>
> When user submits a BSP job, the mechanism is almost same with the M/R
> framework. The basic flow is as below:
>
> 1) User submits a BSP job to Hama BSP cluster.
> 2) Then, the BSPMaster will manage jobs that user have submitted to
> the BSP cluster.
> 2-1) Adding job to the scheduler.
> 2-2) Assign Task to slaves.
> 3) the GroomServer (slave) copies job/jar file to local and launches
> tasks on its host.
>
> And, Hadoop RPC will be used for the BSP send, get and sync
> communication among BSPPeer.
>
> Now the no. 3) is not implemented yet.
>
Thanks for explanation :)
I'm finishing reading the BSP paper from Leslie.
I'll then jump into the code.
cheers
>
> Thanks.
>
> On Thu, Aug 19, 2010 at 8:38 PM, Filipe Manana (JIRA) <ji...@apache.org>
> wrote:
> >
> > [
> https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900255#action_12900255]
> >
> > Filipe Manana commented on HAMA-282:
> > ------------------------------------
> >
> > Hello,
> >
> > I would like to start contributing to Hama.
> > Is this a good feature to start with?
> >
> > cheers
> >
> >> Add superstep counter
> >> ---------------------
> >>
> >> Key: HAMA-282
> >> URL: https://issues.apache.org/jira/browse/HAMA-282
> >> Project: Hama
> >> Issue Type: Sub-task
> >> Reporter: Edward J. Yoon
> >>
> >
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> edwardyoon@apache.org
> http://blog.udanax.org
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: [jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon" <ed...@apache.org>.
Dear fdmanana, Thanks for your interest.
> Is this a good feature to start with?
Yes, I think so. BTW, you need to wait few days for this. The reason
is that the number of supersteps should be counted by the master
server but I'm heavily touching code of master/slave server for
HAMA-276 now. Moreover, there is a lot of hard coded path and TODO
things.
However, It is OK to contribute only part of superstep counting.
Please see the part of BSPPeer.enterbarrier() method and heartbeat
messages between the master/slave servers.
----
And, let me explain more detail about the current status of HAMA TRUNK ....
When user submits a BSP job, the mechanism is almost same with the M/R
framework. The basic flow is as below:
1) User submits a BSP job to Hama BSP cluster.
2) Then, the BSPMaster will manage jobs that user have submitted to
the BSP cluster.
2-1) Adding job to the scheduler.
2-2) Assign Task to slaves.
3) the GroomServer (slave) copies job/jar file to local and launches
tasks on its host.
And, Hadoop RPC will be used for the BSP send, get and sync
communication among BSPPeer.
Now the no. 3) is not implemented yet.
Thanks.
On Thu, Aug 19, 2010 at 8:38 PM, Filipe Manana (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900255#action_12900255 ]
>
> Filipe Manana commented on HAMA-282:
> ------------------------------------
>
> Hello,
>
> I would like to start contributing to Hama.
> Is this a good feature to start with?
>
> cheers
>
>> Add superstep counter
>> ---------------------
>>
>> Key: HAMA-282
>> URL: https://issues.apache.org/jira/browse/HAMA-282
>> Project: Hama
>> Issue Type: Sub-task
>> Reporter: Edward J. Yoon
>>
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929459#action_12929459 ]
Edward J. Yoon commented on HAMA-282:
-------------------------------------
Also, why don't we add the count of superstep to JobStatus and BSPJobClient, as a part of report?
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Issue Type: New Feature (was: Sub-task)
Parent: (was: HAMA-276)
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Fix For: 0.2.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929873#action_12929873 ]
Edward J. Yoon commented on HAMA-282:
-------------------------------------
There must be a misunderstanding. :) The BSP consists of a sequence of supersteps. If all the tasks have completed a superstep, the system proceeds to next superstep. Therefore, the total number of supersteps performed by all the tasks is the same.
For e.g., Pi estimator consists of only one superstep. the job's superstep count should be 1, not "1 + 1 + ... + 1 = the number of tasks".
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Filipe Manana updated HAMA-282:
-------------------------------
Attachment: hama-282.patch
Hi,
The following simple patch adds the super step counter.
cheers
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Filipe Manana updated HAMA-282:
-------------------------------
Attachment: hama-316_fix_tests.patch
Hi,
I made a mistake with the peer names in the test TestBSPPeer. The names should have ':' instead of '_' characters.
The following patch fixes it.
cheers
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-316_fix_tests.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Filipe Manana updated HAMA-282:
-------------------------------
Attachment: (was: hama-316_fix_tests.patch)
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Status: Patch Available (was: Open)
running patch through hudson.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-282:
-----------------------------------
Assignee: Filipe Manana
When all state of BSPPeer is entered to barrier, we'll need to increase the super-step count.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925234#action_12925234 ]
Filipe Manana commented on HAMA-282:
------------------------------------
Ops, wrong ticket. Moving patch to HAMA-316.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929435#action_12929435 ]
Hudson commented on HAMA-282:
-----------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12459029/HAMA-282_v02.patch
against trunk revision 1030802.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/285/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hama-Patch/285/findbugsResult/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/285/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hama-Patch/285/console
This message is automatically generated.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929984#action_12929984 ]
Edward J. Yoon commented on HAMA-282:
-------------------------------------
I just committed hama-282_v05.patch.
If you find some bug please report it to here!!
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch, hama-282_v05.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925273#action_12925273 ]
Edward J. Yoon commented on HAMA-282:
-------------------------------------
Patch looks nice!
I'll test it tonight.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Attachment: hama-282_v03.patch
added getSuperstepCount() to interface.
{code}
/**
* @return the count of current super-step
*/
public long getSuperstepCount();
{code}
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Fix Version/s: 0.2.0
Affects Version/s: 0.2.0
Component/s: bsp
update issue info.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Fix For: 0.2.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900255#action_12900255 ]
Filipe Manana commented on HAMA-282:
------------------------------------
Hello,
I would like to start contributing to Hama.
Is this a good feature to start with?
cheers
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: Sub-task
> Reporter: Edward J. Yoon
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Attachment: hama-282_v04.patch
{code}
10/11/08 15:58:49 INFO bsp.BSPJobClient: Running job: job_201011081558_0001
10/11/08 15:59:06 INFO bsp.BSPJobClient: Job complete: job_201011081558_0001
10/11/08 15:59:06 INFO bsp.BSPJobClient: The total number of supersteps: 2
{code}
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Status: Open (was: Patch Available)
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch, hama-282_v05.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon resolved HAMA-282.
---------------------------------
Resolution: Fixed
Then I'm resolving this issue now. Thanks Filipe! and good night :)
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch, hama-282_v05.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930289#action_12930289 ]
Filipe Manana commented on HAMA-282:
------------------------------------
Right right, my bad :(
Superstep count for all tasks is the same.
I'll not write patches or reply to tickets anymore after midnight :P
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch, hama-282_v05.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929445#action_12929445 ]
Edward J. Yoon commented on HAMA-282:
-------------------------------------
{code}
superstepCounter -= tip.getTaskStatus(status.getTaskId()).getSuperstepCount();
{code}
I just wondered what's meaning of above code line.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929751#action_12929751 ]
Filipe Manana commented on HAMA-282:
------------------------------------
Hi Edward,
Thanks for looking into it.
Yes, the failling test went unnoticed before the addition of the GroboTestingJUnit library.
About the line:
superstepCounter -= tip.getTaskStatus(status.getTaskId()).getSuperstepCount();
My reasoning was the following:
A job in progress has one or more tasks (1 task per peer/groom server), which means the job's superstep count matches the sum of all its tasks' superstep counts. Right?
TaskInProgress seems to represent more than one task (the current task attempt and previous (failed) attempts, like Hadoop MapReduce). So I thought:
In getSuperstepCounter() of JobInProgress, iterate over the list of TaskInProgress, get the TaskStatus of each TaskInProgress, get the superstep counter from each TaskStatus and sum them all.
The problem with this is that in order to get the TaskStatus of a TaskInProgress, I need to supply the TaskStatus ID, which I only seem to know when JobInProgress.updateTaskStatus() is invoked.
Also, I thought that to update the superstep count of the job, I need to decrement it by the number of supersteps of the previous TaskStatus and then increment it with the number of supersteps in the new TaskStatus - an alternative would be to set to zero the superstep count of each TaskStatus (in GroomServer / Peer) after each heartbeat.
Also, in your approach, you do:
public void updateTaskStatus(TaskInProgress tip, TaskStatus status) {
tip.updateStatus(status); // update tip
superstepCounter = status.getSuperstepCount();
}
Like this, I think the superstep count of the job will be the superstep count of the last completed task.
For e.g., if you have 2 tasks for a job (each one running on a different groom/peer), and each one has 2 supersteps done when they finish, the job's superstep count should be 2+2=4 right?
With that approach (your suggestion) I think it will end up as 2.
But correct me if I am wrong :)
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Attachment: hama-282_v05.patch
Re-attach patch.
I added few comments.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch, hama-282_v05.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929441#action_12929441 ]
Edward J. Yoon commented on HAMA-282:
-------------------------------------
not work on distibuted system with below message.
{code}
2010-11-08 11:49:40,599 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 40000, call heartbeat(org.apache.hama.bsp.GroomServerStatus@670fe2b9, false, false, true, 2, 1) from 192.168.123.138:34583: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hama.bsp.JobInProgress.updateTaskStatus(JobInProgress.java:191)
at org.apache.hama.bsp.JobInProgress.completedTask(JobInProgress.java:162)
at org.apache.hama.bsp.BSPMaster.updateTaskStatuses(BSPMaster.java:418)
at org.apache.hama.bsp.BSPMaster.heartbeat(BSPMaster.java:372)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
{code}
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-282) Add superstep counter
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-282:
--------------------------------
Attachment: HAMA-282_v02.patch
this patch fixes the unit test failures.
> Add superstep counter
> ---------------------
>
> Key: HAMA-282
> URL: https://issues.apache.org/jira/browse/HAMA-282
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Filipe Manana
> Fix For: 0.2.0
>
> Attachments: hama-282.patch, HAMA-282_v02.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.