You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Suraj Menon (Created) (JIRA)" <ji...@apache.org> on 2012/04/18 13:31:41 UTC
[jira] [Created] (HAMA-557) Implement Checkpointing service in Hama
Implement Checkpointing service in Hama
---------------------------------------
Key: HAMA-557
URL: https://issues.apache.org/jira/browse/HAMA-557
Project: Hama
Issue Type: New Feature
Components: bsp
Affects Versions: 0.6.0
Reporter: Suraj Menon
Assignee: Suraj Menon
Fix For: 0.6.0
Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
- Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
- BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
- MessageManager should notify Checkpointer of a new message at BSPPeer.
- Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426522#comment-13426522 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
Great, can we fix that the client don't print our superstep numbers less than the one before?
This seems to be very annoying.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452815#comment-13452815 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
bq.Should we merge "bsp.ft.enabled" and "bsp.checkpoint.enabled" into one?
+1
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417344#comment-13417344 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
Very nice, thank you Suraj. I'm tired today, but I have taken a holiday on friday and going to check this completely.
Just to know: Do we want to open a reviewboard entry for that?
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452725#comment-13452725 ]
Edward J. Yoon commented on HAMA-557:
-------------------------------------
And here's my quick performance test results
{code}
== FT disabled ==
[oracle@kr1-bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar bench 100 100 50
12/09/10 20:09:43 INFO bsp.BSPJobClient: Running job: job_201209102009_0001
12/09/10 20:09:46 INFO bsp.BSPJobClient: Current supersteps number: 8
12/09/10 20:09:49 INFO bsp.BSPJobClient: Current supersteps number: 49
12/09/10 20:09:52 INFO bsp.BSPJobClient: Current supersteps number: 50
12/09/10 20:09:52 INFO bsp.BSPJobClient: The total number of supersteps: 50
12/09/10 20:09:52 INFO bsp.BSPJobClient: Counters: 8
12/09/10 20:09:52 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/09/10 20:09:52 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/10 20:09:52 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/10 20:09:52 INFO bsp.BSPJobClient: SUPERSTEPS=50
12/09/10 20:09:52 INFO bsp.BSPJobClient: SUPERSTEP_SUM=4250
12/09/10 20:09:52 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=20677838
12/09/10 20:09:52 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=101865
12/09/10 20:09:52 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=20677838
12/09/10 20:09:52 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=850000
12/09/10 20:09:52 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=425000
Job Finished in 9.317 seconds
[oracle@kr1-bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar bench 100 10000 50
12/09/10 20:10:24 INFO bsp.BSPJobClient: Running job: job_201209102009_0002
12/09/10 20:10:27 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/10 20:10:30 INFO bsp.BSPJobClient: Current supersteps number: 17
12/09/10 20:10:33 INFO bsp.BSPJobClient: Current supersteps number: 33
12/09/10 20:10:36 INFO bsp.BSPJobClient: Current supersteps number: 50
12/09/10 20:10:39 INFO bsp.BSPJobClient: The total number of supersteps: 50
12/09/10 20:10:39 INFO bsp.BSPJobClient: Counters: 8
12/09/10 20:10:39 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/09/10 20:10:39 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/10 20:10:39 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/10 20:10:39 INFO bsp.BSPJobClient: SUPERSTEPS=50
12/09/10 20:10:39 INFO bsp.BSPJobClient: SUPERSTEP_SUM=4250
12/09/10 20:10:39 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=234746978
12/09/10 20:10:39 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=204266
12/09/10 20:10:39 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=234746978
12/09/10 20:10:39 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=85000000
12/09/10 20:10:39 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=42500000
Job Finished in 15.279 seconds
[oracle@kr1-bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar pagerank /user/oracle/edward/web-Google.txt edward/testout
12/09/10 20:13:58 INFO bsp.FileInputFormat: Total input paths to process : 1
12/09/10 20:13:58 INFO bsp.FileInputFormat: Total # of splits: 85
12/09/10 20:13:58 INFO bsp.BSPJobClient: Running job: job_201209102013_0001
12/09/10 20:14:01 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/10 20:14:04 INFO bsp.BSPJobClient: Current supersteps number: 4
12/09/10 20:14:07 INFO bsp.BSPJobClient: Current supersteps number: 11
12/09/10 20:14:07 INFO bsp.BSPJobClient: The total number of supersteps: 11
12/09/10 20:14:07 INFO bsp.BSPJobClient: Counters: 10
12/09/10 20:14:07 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/09/10 20:14:07 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/10 20:14:07 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/10 20:14:07 INFO bsp.BSPJobClient: SUPERSTEPS=11
12/09/10 20:14:07 INFO bsp.BSPJobClient: SUPERSTEP_SUM=935
12/09/10 20:14:07 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=96714132
12/09/10 20:14:07 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=96246
12/09/10 20:14:07 INFO bsp.BSPJobClient: IO_BYTES_READ=75380115
12/09/10 20:14:07 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=96714132
12/09/10 20:14:07 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=14662796
12/09/10 20:14:07 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=5105043
12/09/10 20:14:07 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=7331398
Job Finished in 9.343 seconds
== FT enabled ==
[oracle@kr1-bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar bench 100 100 50
12/09/10 19:26:28 INFO bsp.BSPJobClient: Running job: job_201209101921_0004
12/09/10 19:26:31 INFO bsp.BSPJobClient: Current supersteps number: 1
12/09/10 19:26:34 INFO bsp.BSPJobClient: Current supersteps number: 5
12/09/10 19:26:37 INFO bsp.BSPJobClient: Current supersteps number: 13
12/09/10 19:26:40 INFO bsp.BSPJobClient: Current supersteps number: 17
12/09/10 19:26:43 INFO bsp.BSPJobClient: Current supersteps number: 26
12/09/10 19:26:46 INFO bsp.BSPJobClient: Current supersteps number: 30
12/09/10 19:26:49 INFO bsp.BSPJobClient: Current supersteps number: 38
12/09/10 19:26:52 INFO bsp.BSPJobClient: Current supersteps number: 42
12/09/10 19:26:55 INFO bsp.BSPJobClient: Current supersteps number: 50
12/09/10 19:26:58 INFO bsp.BSPJobClient: The total number of supersteps: 50
12/09/10 19:26:58 INFO bsp.BSPJobClient: Counters: 8
12/09/10 19:26:58 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/09/10 19:26:58 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/10 19:26:58 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/10 19:26:58 INFO bsp.BSPJobClient: SUPERSTEPS=50
12/09/10 19:26:58 INFO bsp.BSPJobClient: SUPERSTEP_SUM=4250
12/09/10 19:26:58 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=20682590
12/09/10 19:26:58 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=168254
12/09/10 19:26:58 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=20682590
12/09/10 19:26:58 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=850000
12/09/10 19:26:58 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=425000
Job Finished in 30.278 seconds
[oracle@kr1-bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar bench 100 10000 50
12/09/10 20:11:23 INFO bsp.BSPJobClient: Running job: job_201209102011_0001
12/09/10 20:11:26 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/10 20:11:29 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/10 20:11:32 INFO bsp.BSPJobClient: Current supersteps number: 5
12/09/10 20:11:35 INFO bsp.BSPJobClient: Current supersteps number: 12
12/09/10 20:11:38 INFO bsp.BSPJobClient: Current supersteps number: 15
12/09/10 20:11:41 INFO bsp.BSPJobClient: Current supersteps number: 21
12/09/10 20:11:44 INFO bsp.BSPJobClient: Current supersteps number: 23
12/09/10 20:11:47 INFO bsp.BSPJobClient: Current supersteps number: 29
12/09/10 20:11:50 INFO bsp.BSPJobClient: Current supersteps number: 32
12/09/10 20:11:53 INFO bsp.BSPJobClient: Current supersteps number: 37
12/09/10 20:11:56 INFO bsp.BSPJobClient: Current supersteps number: 40
12/09/10 20:11:59 INFO bsp.BSPJobClient: Current supersteps number: 47
12/09/10 20:12:02 INFO bsp.BSPJobClient: Current supersteps number: 49
12/09/10 20:12:05 INFO bsp.BSPJobClient: Current supersteps number: 50
12/09/10 20:12:05 INFO bsp.BSPJobClient: The total number of supersteps: 50
12/09/10 20:12:05 INFO bsp.BSPJobClient: Counters: 8
12/09/10 20:12:05 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/09/10 20:12:05 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/10 20:12:05 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/10 20:12:05 INFO bsp.BSPJobClient: SUPERSTEPS=50
12/09/10 20:12:05 INFO bsp.BSPJobClient: SUPERSTEP_SUM=4250
12/09/10 20:12:05 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=234747503
12/09/10 20:12:05 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=174675
12/09/10 20:12:05 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=234747503
12/09/10 20:12:05 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=85000000
12/09/10 20:12:05 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=42500000
Job Finished in 42.34 seconds
[oracle@kr1-bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar pagerank /user/oracle/edward/web-Google.txt edward/testout
12/09/10 20:12:47 INFO bsp.FileInputFormat: Total input paths to process : 1
12/09/10 20:12:47 INFO bsp.FileInputFormat: Total # of splits: 85
12/09/10 20:12:48 INFO bsp.BSPJobClient: Running job: job_201209102011_0002
12/09/10 20:12:51 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/10 20:12:54 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/10 20:13:00 INFO bsp.BSPJobClient: Current supersteps number: 7
12/09/10 20:13:03 INFO bsp.BSPJobClient: Current supersteps number: 11
12/09/10 20:13:06 INFO bsp.BSPJobClient: The total number of supersteps: 11
12/09/10 20:13:06 INFO bsp.BSPJobClient: Counters: 10
12/09/10 20:13:06 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/09/10 20:13:06 INFO bsp.BSPJobClient: LAUNCHED_TASKS=85
12/09/10 20:13:06 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/09/10 20:13:06 INFO bsp.BSPJobClient: SUPERSTEPS=11
12/09/10 20:13:06 INFO bsp.BSPJobClient: SUPERSTEP_SUM=935
12/09/10 20:13:06 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=96730041
12/09/10 20:13:06 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=205603
12/09/10 20:13:06 INFO bsp.BSPJobClient: IO_BYTES_READ=75380115
12/09/10 20:13:06 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=96730041
12/09/10 20:13:06 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=14662796
12/09/10 20:13:06 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=5105043
12/09/10 20:13:06 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=7331398
Job Finished in 18.297 seconds
{code}
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Tommaso Teofili (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426531#comment-13426531 ]
Tommaso Teofili commented on HAMA-557:
--------------------------------------
nice and job! and +1 to Thomas's last comment :)
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-557) Implement Checkpointing service in Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-557:
-----------------------------
Attachment: HAMA-505-557-610-611-v1.patch
HAMA-505-557-610-611-v1.patch is the current state of the patch. I am still in the process of adding new test cases and improving the documentation of the affected classes. The next version would have fixes if any once all the test-cases pass. This patch is to show how the patches in HAMA-610, HAMA-611, HAMA-587 and HAMA-557 work with each other.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-557) Implement Checkpointing service in Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-557:
-----------------------------
Status: Patch Available (was: Open)
Tested the patch for Rand Bench example in my 3 VM cluster. The following output shows the recovery in superstep 2, 3 and 4.
hama jar hama-examples-0.5.0-SNAPSHOT.jar bench 5 5 5
12/08/01 06:28:56 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir: hdfs://hadoop-1:54310/tmp/hama-hduser/bsp/system/submit_vjh6g
12/08/01 06:28:57 INFO bsp.BSPJobClient: Running job: job_201208010625_0002
12/08/01 06:29:00 INFO bsp.BSPJobClient: Current supersteps number: 0
12/08/01 06:29:45 INFO bsp.BSPJobClient: Current supersteps number: 1
12/08/01 06:29:48 INFO bsp.BSPJobClient: Current supersteps number: 2
12/08/01 06:29:54 INFO bsp.BSPJobClient: Current supersteps number: 0
12/08/01 06:30:09 INFO bsp.BSPJobClient: Current supersteps number: 3
12/08/01 06:30:15 INFO bsp.BSPJobClient: Current supersteps number: 0
12/08/01 06:30:30 INFO bsp.BSPJobClient: Current supersteps number: 4
12/08/01 06:30:42 INFO bsp.BSPJobClient: Current supersteps number: 0
12/08/01 06:30:57 INFO bsp.BSPJobClient: Current supersteps number: 5
12/08/01 06:30:57 INFO bsp.BSPJobClient: The total number of supersteps: 5
12/08/01 06:30:57 DEBUG bsp.Counters: Adding SUPERSTEPS
12/08/01 06:30:57 INFO bsp.BSPJobClient: Counters: 8
12/08/01 06:30:57 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/08/01 06:30:57 INFO bsp.BSPJobClient: LAUNCHED_TASKS=3
12/08/01 06:30:57 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/08/01 06:30:57 INFO bsp.BSPJobClient: SUPERSTEPS=5
12/08/01 06:30:57 INFO bsp.BSPJobClient: COMPRESSED_BYTES_SENT=718
12/08/01 06:30:57 INFO bsp.BSPJobClient: SUPERSTEP_SUM=15
12/08/01 06:30:57 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=712
12/08/01 06:30:57 INFO bsp.BSPJobClient: COMPRESSED_BYTES_RECEIVED=474
12/08/01 06:30:57 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=150
12/08/01 06:30:57 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=46
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HAMA-557) Implement Checkpointing service
in Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426522#comment-13426522 ]
Thomas Jungblut edited comment on HAMA-557 at 8/1/12 11:07 AM:
---------------------------------------------------------------
Great, can we fix that the client don't print our superstep numbers less than the one before?
This seems to be very annoying.
+ A counter for restarted tasks (complete rollbacks and just a simple task restart) would be cool ;)
was (Author: thomas.jungblut):
Great, can we fix that the client don't print our superstep numbers less than the one before?
This seems to be very annoying.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417349#comment-13417349 ]
Suraj Menon commented on HAMA-557:
----------------------------------
Take your time, this is a part of a big patch. I shall upload it shortly, I am still testing it. 90% of patch was tested before I re-factored the code. I have few more unit tests to work on and have to verify different situations on my small VM cluster before I release it to others. I shall upload the current state of patch for a sneak-peek. Any effective way of reviewing is fine with me.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428931#comment-13428931 ]
Hudson commented on HAMA-557:
-----------------------------
Integrated in Hama-Nightly #633 (See [https://builds.apache.org/job/Hama-Nightly/633/])
Committing the merge from HAMA-505-branch. Contains changes for HAMA-557 HAMA-587 HAMA-610 HAMA-611 (Revision 1369575)
Result = FAILURE
surajsmenon :
Files :
* /hama/trunk
* /hama/trunk/conf/hama-default.xml
* /hama/trunk/core/src/main/java/org/apache/hama/Constants.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJobClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPMaster.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPTask.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/GroomServer.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/GroomServerAction.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/JobInProgress.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/JobInProgressListener.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/JobStatus.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/LaunchTaskAction.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/LocalBSPRunner.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/RecoverTaskAction.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/SimpleTaskScheduler.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/TaskInProgress.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/TaskRunner.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/TaskStatus.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/UpdatePeerAction.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/AsyncRcvdMsgCheckpointImpl.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/BSPFaultTolerantService.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/FaultTolerantMasterService.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/FaultTolerantPeerService.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AbstractMessageManager.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AvroMessageManagerImpl.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManager.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManagerImpl.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageEventListener.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManager.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/BSPMasterSyncClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/BSPPeerSyncClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/MasterSyncClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/PeerSyncClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/SyncClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/SyncEvent.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/SyncEventListener.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/SyncServiceFactory.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZKSyncBSPMasterClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZKSyncClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZKSyncEventFactory.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZKSyncEventListener.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZooKeeperSyncClientImpl.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/taskallocation
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/taskallocation/BSPResource.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/taskallocation/BestEffortDataLocalTaskAllocator.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/taskallocation/RawSplitResource.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/taskallocation/TaskAllocationStrategy.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestBSPTaskFaults.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestCheckpoint.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestTaskAllocation.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestZooKeeper.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/sync/TestSyncServiceFactory.java
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-557) Implement Checkpointing service in Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-557:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452724#comment-13452724 ]
Edward J. Yoon commented on HAMA-557:
-------------------------------------
{code}
edward@edward-VirtualBox:~/workspace/hama-trunk$ bin/hama jar examples/target/hama-examples-0.6.0-SNAPSHOT.jar bench 5 5 5
12/09/11 14:04:27 INFO bsp.BSPJobClient: Running job: job_201209111359_0002
12/09/11 14:04:30 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/11 14:04:36 INFO bsp.BSPJobClient: Current supersteps number: 2
12/09/11 14:04:45 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/11 14:04:51 INFO bsp.BSPJobClient: Current supersteps number: 3
12/09/11 14:05:06 INFO bsp.BSPJobClient: Current supersteps number: 0
12/09/11 14:05:12 INFO bsp.BSPJobClient: Current supersteps number: 4
12/09/11 14:05:24 INFO bsp.BSPJobClient: Current supersteps number: 0
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:host.name=edward-VirtualBox
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_06
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-oracle/jre
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/edward/workspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:/home/edward/workspace/hama-trunk/bin/../core/target/classes:/home/edward/workspace/hama-trunk/bin/../graph/target/classes:/home/edward/workspace/hama-trunk/bin/../hama-**.jar:/home/edward/workspace/hama-trunk/bin/../lib/ant-1.7.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/ant-launcher-1.7.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/avro-1.6.0.jar:/home/edward/workspace/hama-trunk/bin/../lib/avro-ipc-1.6.0.jar:/home/edward/workspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/edward/workspace/hama-trunk/bin/../lib/commons-configuration-1.7.jar:/home/edward/workspace/hama-trunk/bin/../lib/commons-httpclient-3.0.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/commons-lang-2.6.jar:/home/edward/workspace/hama-trunk/bin/../lib/commons-logging-1.1.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/commons-math3-3.0.jar:/home/edward/workspace/hama-trunk/bin/../lib/guava-10.0.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/hadoop-core-1.0.0.jar:/home/edward/workspace/hama-trunk/bin/../lib/hadoop-test-1.0.0.jar:/home/edward/workspace/hama-trunk/bin/../lib/jackson-core-asl-1.9.2.jar:/home/edward/workspace/hama-trunk/bin/../lib/jackson-mapper-asl-1.9.2.jar:/home/edward/workspace/hama-trunk/bin/../lib/jetty-6.1.14.jar:/home/edward/workspace/hama-trunk/bin/../lib/jetty-annotations-6.1.14.jar:/home/edward/workspace/hama-trunk/bin/../lib/jetty-util-6.1.14.jar:/home/edward/workspace/hama-trunk/bin/../lib/jsp-2.1-6.1.14.jar:/home/edward/workspace/hama-trunk/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/edward/workspace/hama-trunk/bin/../lib/junit-4.8.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/log4j-1.2.16.jar:/home/edward/workspace/hama-trunk/bin/../lib/netty-3.2.6.Final.jar:/home/edward/workspace/hama-trunk/bin/../lib/servlet-api-6.0.32.jar:/home/edward/workspace/hama-trunk/bin/../lib/slf4j-api-1.5.8.jar:/home/edward/workspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.8.jar:/home/edward/workspace/hama-trunk/bin/../lib/snappy-java-1.0.4.1.jar:/home/edward/workspace/hama-trunk/bin/../lib/zookeeper-3.3.3.jar::/tmp/hadoop-edward/bsp/local/groomServer/attempt_201209111359_0002_000002_2/work/classes:/tmp/hadoop-edward/bsp/local/groomServer/attempt_201209111359_0002_000002_2/work
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-29-generic
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:user.name=edward
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/edward
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Client environment:user.dir=/tmp/hadoop-edward/bsp/local/groomServer/attempt_201209111359_0002_000002_2/work
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=edward-VirtualBox:21810 sessionTimeout=1200000 watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@e33ad7
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ClientCnxn: Opening socket connection to server edward-VirtualBox/127.0.1.1:21810
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ClientCnxn: Socket connection established to edward-VirtualBox/127.0.1.1:21810, initiating session
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO sync.ZKSyncClient: Initializing ZK Sync Client
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At edward-VirtualBox/127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:07 INFO zookeeper.ClientCnxn: Session establishment complete on server edward-VirtualBox/127.0.1.1:21810, sessionid = 0x139b3b269e10013, negotiated timeout = 1200000
attempt_201209111359_0002_000002_2: 12/09/11 14:05:08 INFO ipc.NettyTransceiver: Connecting to edward-VirtualBox/127.0.1.1:61003
attempt_201209111359_0002_000002_2: 12/09/11 14:05:08 INFO ipc.NettyTransceiver: [id: 0x15ee470d] OPEN
attempt_201209111359_0002_000002_2: 12/09/11 14:05:08 INFO ipc.NettyTransceiver: [id: 0x15ee470d, /127.0.0.1:51614 => edward-VirtualBox/127.0.1.1:61003] BOUND: /127.0.0.1:51614
attempt_201209111359_0002_000002_2: 12/09/11 14:05:08 INFO ipc.NettyTransceiver: [id: 0x15ee470d, /127.0.0.1:51614 => edward-VirtualBox/127.0.1.1:61003] CONNECTED: edward-VirtualBox/127.0.1.1:61003
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x73e3df54, /127.0.0.1:50522 => /127.0.1.1:61004] OPEN
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: Connecting to edward-VirtualBox/127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x02190419] OPEN
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x385c0662, /127.0.0.1:50523 => /127.0.1.1:61004] OPEN
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x73e3df54, /127.0.0.1:50522 => /127.0.1.1:61004] BOUND: /127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x73e3df54, /127.0.0.1:50522 => /127.0.1.1:61004] CONNECTED: /127.0.0.1:50522
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x385c0662, /127.0.0.1:50523 => /127.0.1.1:61004] BOUND: /127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x385c0662, /127.0.0.1:50523 => /127.0.1.1:61004] CONNECTED: /127.0.0.1:50523
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ft.AsyncRcvdMsgCheckpointImpl: Creating path checkpoint/job_201209111359_0002/4/2
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x02190419, /127.0.0.1:50523 => edward-VirtualBox/127.0.1.1:61004] BOUND: /127.0.0.1:50523
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x02190419, /127.0.0.1:50523 => edward-VirtualBox/127.0.1.1:61004] CONNECTED: edward-VirtualBox/127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: Connecting to edward-VirtualBox/127.0.1.1:61002
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x4362b702] OPEN
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x05dd06de, /127.0.0.1:50524 => /127.0.1.1:61004] OPEN
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x05dd06de, /127.0.0.1:50524 => /127.0.1.1:61004] BOUND: /127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x05dd06de, /127.0.0.1:50524 => /127.0.1.1:61004] CONNECTED: /127.0.0.1:50524
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x4362b702, /127.0.0.1:55795 => edward-VirtualBox/127.0.1.1:61002] BOUND: /127.0.0.1:55795
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x4362b702, /127.0.0.1:55795 => edward-VirtualBox/127.0.1.1:61002] CONNECTED: edward-VirtualBox/127.0.1.1:61002
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO sync.ZKSyncClient: Writing data /bsp/job_201209111359_0002/checkpoint/2
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ft.AsyncRcvdMsgCheckpointImpl: Enabled = true checkPointInterval = 1 lastCheckPointStep = 4 getSuperstepCount() = 4
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ft.AsyncRcvdMsgCheckpointImpl: checkpointNext = true checkpointMessageCount = 0
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
attempt_201209111359_0002_000002_2: java.lang.RuntimeException: Error generated to test by peer 2
attempt_201209111359_0002_000002_2: at org.apache.hama.examples.RandBench$RandBSP.compute(RandBench.java:76)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.SuperstepBSP.bsp(SuperstepBSP.java:69)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1271)
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO zookeeper.ZooKeeper: Session: 0x139b3b269e10013 closed
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO zookeeper.ClientCnxn: EventThread shut down
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x73e3df54, /127.0.0.1:50522 :> /127.0.1.1:61004] DISCONNECTED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x73e3df54, /127.0.0.1:50522 :> /127.0.1.1:61004] UNBOUND
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x73e3df54, /127.0.0.1:50522 :> /127.0.1.1:61004] CLOSED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x385c0662, /127.0.0.1:50523 :> /127.0.1.1:61004] DISCONNECTED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x385c0662, /127.0.0.1:50523 :> /127.0.1.1:61004] UNBOUND
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x385c0662, /127.0.0.1:50523 :> /127.0.1.1:61004] CLOSED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x02190419, /127.0.0.1:50523 :> edward-VirtualBox/127.0.1.1:61004] DISCONNECTED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x02190419, /127.0.0.1:50523 :> edward-VirtualBox/127.0.1.1:61004] UNBOUND
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: [id: 0x02190419, /127.0.0.1:50523 :> edward-VirtualBox/127.0.1.1:61004] CLOSED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: Remote peer edward-VirtualBox/127.0.1.1:61004 closed connection.
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyTransceiver: Disconnecting from edward-VirtualBox/127.0.1.1:61004
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x05dd06de, /127.0.0.1:50524 :> /127.0.1.1:61004] DISCONNECTED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x05dd06de, /127.0.0.1:50524 :> /127.0.1.1:61004] UNBOUND
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 INFO ipc.NettyServer: [id: 0x05dd06de, /127.0.0.1:50524 :> /127.0.1.1:61004] CLOSED
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 ERROR bsp.BSPTask: Shutting down ping service.
attempt_201209111359_0002_000002_2: 12/09/11 14:05:09 FATAL bsp.GroomServer: Error running child
attempt_201209111359_0002_000002_2: java.lang.RuntimeException: Error generated to test by peer 2
attempt_201209111359_0002_000002_2: at org.apache.hama.examples.RandBench$RandBSP.compute(RandBench.java:76)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.SuperstepBSP.bsp(SuperstepBSP.java:69)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1271)
attempt_201209111359_0002_000002_2: java.lang.RuntimeException: Error generated to test by peer 2
attempt_201209111359_0002_000002_2: at org.apache.hama.examples.RandBench$RandBSP.compute(RandBench.java:76)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.SuperstepBSP.bsp(SuperstepBSP.java:69)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
attempt_201209111359_0002_000002_2: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1271)
12/09/11 14:05:24 INFO bsp.BSPJobClient: Job failed.
{code}
Wanted to test your RandBench but, always fails (at final Exception). Am I missed something?
I used TRUNK.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426511#comment-13426511 ]
Edward J. Yoon commented on HAMA-557:
-------------------------------------
Cool
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419420#comment-13419420 ]
Suraj Menon commented on HAMA-557:
----------------------------------
Thanks for the detailed review Thomas.
I actually continued working from the point of this patch.
> Compile Problem TestCheckpoint overrides the method replayMessages, but it does not exist, should it exist?
Sorry, I have fixed the compile problem. Caught me doing mvn install -DskipTests = true.
> In hama-core there is a folder created called "nullzookeeper", from what testcase does that come from?
I think it should be from the TestSyncService, I shall look into it.
> Can we remove the tilde's ~ from debug output?
The logs with ~s are going to be removed completely. I have it for quick check on progress.
> Do you think we should stick with defining interfaces with "I" in front? I'm naming interfaces without them and call the concrete implementations *Impl. What do you think is the best?
I can change it and we can continue the naming convention that is already there.
> Now we have a lot of services, we could extract init and close to a superinterface, WDYT? Don't know about the usage then if they can be composed.
Most of our services get initialized by ReflectionUtils.newInstance. Until now we don't have a common service that validates the interaction between each of these services. Let's look into this once we have such a requirement. I also feel inits may have different signatures for different services.
Thanks, your additional notes had some good catches. I have fixed few of those already. In my final version, I am planning to put more unit test cases along with some fixes for issues I am encountering on my cluster. Hopefully, you were happy with the way refactored code interacted with each other. I wanted these to be building blocks for future work.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-557) Implement Checkpointing service in Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-557:
-----------------------------
Attachment: HAMA-557-ft-framework.patch
HAMA-557-ft-framework.patch shows the fault-tolerance design framework and the checkpointing service is implemented using the class design.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-557) Implement Checkpointing service in Hama
Posted by "Suraj Menon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-557:
-----------------------------
Issue Type: Sub-task (was: New Feature)
Parent: HAMA-505
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256778#comment-13256778 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
Please wait for my refactoring to be committed in HAMA-521. It makes a huge refactoring in the message managers.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Suraj Menon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256648#comment-13256648 ]
Suraj Menon commented on HAMA-557:
----------------------------------
Reasons for HAMA-557 requirements are as followed in descending order of their importance:
1. Make checkpointing asynchronous. Would reduce the delay during sync. The checkpoint function you mentioned would then be optional and could be used only if there is a need to sync with the ending of checkpointing process for that superstep. The design would be similar to how spilling is implemented in Hadoop.
2. Update BSPMaster with the checkpointing status of each task with superstep count and checkpoint file. (We can live without this as we have a convention for naming checkpointing files)
3. Tighter integration of Checkpointer with MessageManager.
I would be putting up a design document for HAMA-551.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-557) Implement Checkpointing service in Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suraj Menon updated HAMA-557:
-----------------------------
Attachment: HAMA-505-557-610-611-v2.patch
Version 2 of the patch.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426606#comment-13426606 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
It's a nice feature ;)
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452797#comment-13452797 ]
Edward J. Yoon commented on HAMA-557:
-------------------------------------
Should we merge "bsp.ft.enabled" and "bsp.checkpoint.enabled" into one?
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419341#comment-13419341 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
Took a bit longer because I had to checkout the branch. Sorry for that.
Questions:
-Compile Problem TestCheckpoint overrides the method replayMessages, but it does not exist, should it exist?
-In hama-core there is a folder created called "nullzookeeper", from what testcase does that come from?
-Can we remove the tilde's ~ from debug output?
-Do you think we should stick with defining interfaces with "I" in front? I'm naming interfaces without them and call the concrete implementations *Impl. What do you think is the best?
-Now we have a lot of services, we could extract init and close to a superinterface, WDYT? Don't know about the usage then if they can be composed.
Additional notes:
-A few imports could be organized
-Leave out the abstract declarations in interfaces (MasterSyncClient for example)
-In JobInProgress you are using newInstance several times with new Object[0], I think this is not needed allocation, the method is null save as far as I know, so you can simply pass NULL
-Several // TODO Auto-generated catch block comments
(this is actually the most important note)
-in checkpoint received messages, you are writing the classname with each message, I think we should combine messages by class and then write it. Something like the message bundles does.
Testcases are fine. Thanks for the great work Suraj :)
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426602#comment-13426602 ]
Suraj Menon commented on HAMA-557:
----------------------------------
Thanks,
I have to figure out why the superstep number is shown as 0, eventhough the recovery is from 2, 3 and 4.
Also, what is your opinion on the following display? :
12/08/01 06:29:00 INFO bsp.BSPJobClient: Current supersteps number: 0 : RUNNING
12/08/01 06:29:45 INFO bsp.BSPJobClient: Current supersteps number: 1 : RUNNING
12/08/01 06:29:48 INFO bsp.BSPJobClient: Current supersteps number: 2 : RUNNING
12/08/01 06:29:54 INFO bsp.BSPJobClient: Current supersteps number: 2 : RECOVERING
12/08/01 06:30:09 INFO bsp.BSPJobClient: Current supersteps number: 3 : RUNNING
12/08/01 06:30:15 INFO bsp.BSPJobClient: Current supersteps number: 3 : RECOVERING
12/08/01 06:30:30 INFO bsp.BSPJobClient: Current supersteps number: 4 : RUNNING
12/08/01 06:30:42 INFO bsp.BSPJobClient: Current supersteps number: 4 : RECOVERING
12/08/01 06:30:57 INFO bsp.BSPJobClient: Current supersteps number: 5 : RUNNING
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419751#comment-13419751 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
bq.Most of our services get initialized by ReflectionUtils.newInstance. Until now we don't have a common service that validates the interaction between each of these services. Let's look into this once we have such a requirement. I also feel inits may have different signatures for different services.
Actually we could break down inits to use configuration as parameters. But let's talk about it when we need this.
bq. Hopefully, you were happy with the way refactored code interacted with each other. I wanted these to be building blocks for future work.
this is really great code :)
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-557) Implement Checkpointing service in
Hama
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452782#comment-13452782 ]
Thomas Jungblut commented on HAMA-557:
--------------------------------------
I think we can iron out a few performance problems there. But it is very great that it works now!
The randbench error is interesting.
> Implement Checkpointing service in Hama
> ---------------------------------------
>
> Key: HAMA-557
> URL: https://issues.apache.org/jira/browse/HAMA-557
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.6.0
> Reporter: Suraj Menon
> Assignee: Suraj Menon
> Fix For: 0.6.0
>
> Attachments: HAMA-505-557-610-611-v1.patch, HAMA-505-557-610-611-v2.patch, HAMA-557-ft-framework.patch
>
>
> Implement checkpointing service in Apache Hama. My patches for HAMA-533 and HAMA-534 are blocked on this.
> - Checkpointing should be done as messages are either sent or received. I prefer while receiving messages, as we can achieve some parallelism with asynchronous messages. Please comment if you differ.
> - BSPMaster should hold the checkpoint status for each task. Checkpoint status includes superstep count and file information for which checkpointing is complete
> - MessageManager should notify Checkpointer of a new message at BSPPeer.
> - Implement/Reuse MessageBundle class as splitClass in BSPPeerImpl for recovery in initInput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira