You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "ChiaHung Lin (JIRA)" <ji...@apache.org> on 2011/09/19 04:50:08 UTC
[jira] [Created] (HAMA-440) Features required in recovery
procedure.
Features required in recovery procedure.
-----------------------------------------
Key: HAMA-440
URL: https://issues.apache.org/jira/browse/HAMA-440
Project: Hama
Issue Type: Improvement
Components: bsp
Reporter: ChiaHung Lin
Assignee: ChiaHung Lin
Features are required while performing recovery procedure:
1. checkpoint process needs to be restartable.
2. integrity of checkpointed data.
3. refactor bsp() in allowing checkpointed messages to be recovered.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-440) Integrity of checkpointed data
Posted by "Suraj Menon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon reassigned HAMA-440:
--------------------------------
Assignee: Suraj Menon (was: ChiaHung Lin)
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: Suraj Menon
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HAMA-440) Integrity of checkpointed
data
Posted by "ChiaHung Lin (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226775#comment-13226775 ]
ChiaHung Lin edited comment on HAMA-440 at 3/10/12 8:49 AM:
------------------------------------------------------------
This issue concerns if we have a complete snapshot of all tasks (involved in a job) per superstep so that in the later stage when requiring to do recovery, master can find the latest snapshot for bsp tasks.
was (Author: chl501):
This issue concerns if we have a complete snapshot of all tasks so that in the later stage when requiring to do recovery, master can find the latest snapshot for bsp tasks.
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: ChiaHung Lin
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-440) Integrity of checkpointed data
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226775#comment-13226775 ]
ChiaHung Lin commented on HAMA-440:
-----------------------------------
This issue concerns if we have a complete snapshot of all tasks so that in the later stage when requiring to do recovery, master can find the latest snapshot for bsp tasks.
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: ChiaHung Lin
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-440) Integrity of checkpointed data
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424965#comment-13424965 ]
Suraj Menon commented on HAMA-440:
----------------------------------
The aforesaid requirement is already met in HAMA-557 patch. It is also tested in the coming version 2 of the patch. Also as a measure, we store the message count received along with superstep progress. This count is verified with the count of the messages read during recovery for each task.
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Reporter: ChiaHung Lin
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-440) Integrity of checkpointed data
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-440:
---------------------------------
Description: Data must be consistent, so we may check it for integrity before rolling back in a superstep. (was: Features are required while performing recovery procedure:
-1. checkpoint process needs to be restartable.-
2. integrity of checkpointed data.
3. refactor bsp() in allowing checkpointed messages to be recovered. )
Summary: Integrity of checkpointed data (was: Features required in recovery procedure. )
Turned this into 2. integrity of checkpointed data.
Split the 3. point into another issue HAMA-503.
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: ChiaHung Lin
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-440) Features required in recovery
procedure.
Posted by "ChiaHung Lin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ChiaHung Lin updated HAMA-440:
------------------------------
Description:
Features are required while performing recovery procedure:
-1. checkpoint process needs to be restartable.-
2. integrity of checkpointed data.
3. refactor bsp() in allowing checkpointed messages to be recovered.
was:
Features are required while performing recovery procedure:
1. checkpoint process needs to be restartable.
2. integrity of checkpointed data.
3. refactor bsp() in allowing checkpointed messages to be recovered.
> Features required in recovery procedure.
> -----------------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: ChiaHung Lin
>
> Features are required while performing recovery procedure:
> -1. checkpoint process needs to be restartable.-
> 2. integrity of checkpointed data.
> 3. refactor bsp() in allowing checkpointed messages to be recovered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-440) Integrity of checkpointed data
Posted by "Suraj Menon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225945#comment-13225945 ]
Suraj Menon commented on HAMA-440:
----------------------------------
Isn't HDFS already ensuring integrity of data? Does this mean we have to compute checksum of all the messages (or each message) before we checkpoint and then verify while reading them back?
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: ChiaHung Lin
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-440) Integrity of checkpointed data
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-440:
---------------------------------
Issue Type: Sub-task (was: Improvement)
Parent: HAMA-505
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: ChiaHung Lin
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-440) Integrity of checkpointed data
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon resolved HAMA-440.
------------------------------
Resolution: Duplicate
Fix Version/s: 0.6.0
Duplicate of HAMA-557
> Integrity of checkpointed data
> ------------------------------
>
> Key: HAMA-440
> URL: https://issues.apache.org/jira/browse/HAMA-440
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Reporter: ChiaHung Lin
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
>
> Data must be consistent, so we may check it for integrity before rolling back in a superstep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira