You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (Created) (JIRA)" <ji...@apache.org> on 2011/09/28 14:31:45 UTC
[jira] [Created] (HAMA-445) Make configurable checkpointing
Make configurable checkpointing
-------------------------------
Key: HAMA-445
URL: https://issues.apache.org/jira/browse/HAMA-445
Project: Hama
Issue Type: Improvement
Reporter: Edward J. Yoon
Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "Suraj Menon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189760#comment-13189760 ]
Suraj Menon commented on HAMA-445:
----------------------------------
Does checkpoint interval here imply the number of supersteps before we initiate a checkpoint process? Should this be done within barrier synchronization period or should we have a Checkpointer daemon as we have for backing up namenode in Hadoop. With the second option, we might loose the determinism in finding/assuming at an instant of time (how many supersteps) or (the last superstep) that have been completely checkpointed. The first approach might make it slower but would have better determinism in checkpoint recovery. Please let me know if I have the correct understanding.
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Labels: newbie
> Fix For: 0.5.0
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Suraj Menon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-445:
-----------------------------
Status: Patch Available (was: Open)
Hello Please note that this patch contains fix for issue HAMA-498
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Assignee: Suraj Menon
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HAMA-498-445.patch
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-445) Make configurable checkpointing
Posted by "Suraj Menon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon reassigned HAMA-445:
--------------------------------
Assignee: Suraj Menon
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Assignee: Suraj Menon
> Labels: newbie
> Fix For: 0.5.0
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117062#comment-13117062 ]
Edward J. Yoon commented on HAMA-445:
-------------------------------------
+1
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Reporter: Edward J. Yoon
>
> Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-445:
---------------------------------
Description:
We should extend the BSPJob to let the user set the checkpoint intervals.
bq. job.setCheckpointInterval(5);
This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
Plus points:
If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
was:Currently it works? It would be nice if it is configurable for a user.
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Labels: newbie
> Fix For: 0.4.0
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Edward J. Yoon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-445:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks Suraj!
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Assignee: Suraj Menon
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HAMA-498-445.patch
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Edward J. Yoon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-445:
--------------------------------
Affects Version/s: (was: 0.3.0)
0.4.0
Fix Version/s: (was: 0.4.0)
0.5.0
move to 0.5
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Labels: newbie
> Fix For: 0.5.0
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116992#comment-13116992 ]
ChiaHung Lin commented on HAMA-445:
-----------------------------------
Although this task is simple, it would be more meaningful that master can 1.) ensure if checkpointed data is complete 2.) obtain info from metrics/ resource system in deciding to which groom new tasks will be scheduled before making checkpoint configurable. So in my personal viewpoint, priority of HAMA-440 #2 and HAMA-363 would be higher than making checkpoint configurable.
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Reporter: Edward J. Yoon
>
> Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190428#comment-13190428 ]
Thomas Jungblut commented on HAMA-445:
--------------------------------------
bq. Does checkpoint interval here imply the number of supersteps before we initiate a checkpoint process?
Yes.
bq. Should this be done within barrier synchronization period
There is already a part in the sync barrier that will do the checkpointing. (arround line 250)
{noformat}
if (conf.getBoolean("bsp.checkpoint.enabled", false)) {
checkpoint(checkpointedPath(), bundle);
}
{noformat}
I guess it is enough to do somekind of modulo checking,
{noformat}
if(!disabled && getSuperStep() % interval == 0)
doCheckpoint
{noformat}
bq.Please let me know if I have the correct understanding.
Yes you have ;)
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Labels: newbie
> Fix For: 0.5.0
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140023#comment-13140023 ]
ChiaHung Lin commented on HAMA-445:
-----------------------------------
We can put messages to queue for checkpointer to periodically pick up and to save data to hdfs.
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
>
> Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139242#comment-13139242 ]
Edward J. Yoon commented on HAMA-445:
-------------------------------------
ChiaHung,
I don't think that the basic checkpoint/recovery are heavily related with HAMA-440 and HAMA-363.
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
>
> Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-445) Make configurable checkpointing
Posted by "Edward J. Yoon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-445:
-----------------------------------
Assignee: (was: Edward J. Yoon)
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Labels: newbie
> Fix For: 0.4.0
>
>
> Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Edward J. Yoon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-445:
--------------------------------
Component/s: bsp
Affects Version/s: 0.3.0
Fix Version/s: 0.4.0
Assignee: Edward J. Yoon
I'm scheduling to 0.4 and taking this task.
The checkpoint interval also should be configurable so that user can set the optimal "Interval" value for each jobs as below.
{code}
BSPJob job = ...
job.setCheckpointInterval(5);
{code}
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
>
> Currently it works? It would be nice if it is configurable for a user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Suraj Menon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-445:
-----------------------------
Attachment: HAMA-498-445.patch
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Assignee: Suraj Menon
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HAMA-498-445.patch
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-445) Make configurable checkpointing
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-445:
---------------------------------
Issue Type: Sub-task (was: Improvement)
Parent: HAMA-505
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Labels: newbie
> Fix For: 0.5.0
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-445) Make configurable checkpointing
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222397#comment-13222397 ]
ChiaHung Lin commented on HAMA-445:
-----------------------------------
Can we refactor checkpoint() function to be executed in another thread?
If I remember correctly, original checkpoint() execution makes use of main thread to save message bundle to hdfs. So if message bundle size is too large, this might delay the whole process. Even message bundle size is not huge, during sync() the process still need to wait after message bundle is saved to hdfs. Then
it.remove();
messenger.transfer(addr, bundle);
can happen.
> Make configurable checkpointing
> -------------------------------
>
> Key: HAMA-445
> URL: https://issues.apache.org/jira/browse/HAMA-445
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.4.0
> Reporter: Edward J. Yoon
> Assignee: Suraj Menon
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HAMA-498-445.patch
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq. job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira