You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Tamas Sarlos (JIRA)" <ji...@apache.org> on 2009/12/17 00:39:18 UTC
[jira] Created: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
[MUMAK] Randomize the arrival of heartbeat responses
----------------------------------------------------
Key: MAPREDUCE-1306
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/mumak
Affects Versions: 0.21.0, 0.22.0
Reporter: Tamas Sarlos
Assignee: Tamas Sarlos
Fix For: 0.21.0, 0.22.0
We propose to make the following changes to mumak, MAPREDUCE-728
- make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
- randomize the startup time of task trackers in a fixed interval
- remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1306:
---------------------------------
Attachment: MAPREDUCE-1306-20100308-hong.patch
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843727#action_12843727 ]
Tamas Sarlos commented on MAPREDUCE-1306:
-----------------------------------------
+1 for Hong's change, it is clearly better that way.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843455#action_12843455 ]
Hong Tang commented on MAPREDUCE-1306:
--------------------------------------
I think we should avoid using conf objects to pass the seeds from SimulatorEngine to ZombieJobStoryProducer. This creates a hidden dependency from SimulatorEngine to ZJSP, and thus is harder to maintain in the long run. I revised your patch such that SE passes a seed to SimulatorJobStoryProducer (SJSP) which in turn passes down to ZJSP.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844132#action_12844132 ]
Hudson commented on MAPREDUCE-1306:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #255 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/255/])
. Randomize the arrival of heartbeat responses in Mumak.
Contributed by Tamas Sarlos
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Attachment: MAPREDUCE-1306-20100108.patch
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843484#action_12843484 ]
Hadoop QA commented on MAPREDUCE-1306:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438267/MAPREDUCE-1306-20100308.patch
against trunk revision 921230.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/515/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/515/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/515/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/515/console
This message is automatically generated.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Status: Patch Available (was: Open)
Resubmitting latest patch to Hudson as failing test runs fine locally.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Release Note: [MUMAK] Randomize the arrival of heartbeat responses
Status: Patch Available (was: Open)
Attaching patch implementing the proposed improvements.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801052#action_12801052 ]
Hong Tang commented on MAPREDUCE-1306:
--------------------------------------
+1 on Chris's suggestion. Re-playability is important.
@tamas, The extra work in ZombieJob and ZombieJobProducer seems quite minor. I really hope we could just get it done here instead of having another jira to address it.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843469#action_12843469 ]
Hong Tang commented on MAPREDUCE-1306:
--------------------------------------
Forgot to say that the patch is otherwise +1 from me.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843016#action_12843016 ]
Hadoop QA commented on MAPREDUCE-1306:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438267/MAPREDUCE-1306-20100308.patch
against trunk revision 920250.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/console
This message is automatically generated.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Status: Open (was: Patch Available)
Canceling patch so that I can resubmit it
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801010#action_12801010 ]
Hong Tang commented on MAPREDUCE-1306:
--------------------------------------
Patch looks good. +1.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798022#action_12798022 ]
Hadoop QA commented on MAPREDUCE-1306:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429734/MAPREDUCE-1306-20100108.patch
against trunk revision 897118.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/256/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/256/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/256/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/256/console
This message is automatically generated.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Attachment: MAPREDUCE-1306-20100308.patch
Attaching the updated patch that implements the random seeding option for mumak and rumen.
In order to make the simulation deterministic HashSets and HashMaps need to be replaced with different collection classes since they make no guarantees as to the order of iteration. E.g. JobInProgress iterates over the JobTrackers's nodesAtMaxLevel HashSet, the order of this iteration influences the scheduling of non-local maps. Using AspectJ all HashSets and HashMaps are replaced with LinkedHashSets and LinkedHashMaps, whose iteration order is determined by the order of insertions. This solution needs to be revisited if mumak ever becomes multi-threaded. An added functional test verifies deterministic replay by comparing the job history files.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801025#action_12801025 ]
Chris Douglas commented on MAPREDUCE-1306:
------------------------------------------
It would help if the SimulatedTaskTracker RNG were seeded deterministically from a configurable source. If the STT cstr took a {{seed}} param, a single source of randomness could seed each STT and allow for an exact replay.
Otherwise +1
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Status: Open (was: Patch Available)
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801043#action_12801043 ]
Tamas Sarlos commented on MAPREDUCE-1306:
-----------------------------------------
Re: Chris: random seed
I agree, i just consciously followed the (wrong) convention that mumak has no random seed config option. This also needs to be fixed in SimulatorEngine.java and in org.apache.hadoop.tools.{ZombieJob, ZombieJobProducer}.java (see getNextJob()) to make the entire simulation replayable. I suggest to change all these together in a separate patch. Do you agree?
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Status: Patch Available (was: Open)
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843888#action_12843888 ]
Hudson commented on MAPREDUCE-1306:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #272 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/272/])
. Randomize the arrival of heartbeat responses in Mumak.
Contributed by Tamas Sarlos
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1306:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I committed this. Thanks, Tamas!
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Tamas Sarlos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Sarlos updated MAPREDUCE-1306:
------------------------------------
Status: Patch Available (was: Open)
Submitting patch attached earlier, it passes all test-patch tests on my dev machine.
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of
heartbeat responses
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1306:
-------------------------------------
Status: Open (was: Patch Available)
> [MUMAK] Randomize the arrival of heartbeat responses
> ----------------------------------------------------
>
> Key: MAPREDUCE-1306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Tamas Sarlos
> Assignee: Tamas Sarlos
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1306-20100108.patch
>
>
> We propose to make the following changes to mumak, MAPREDUCE-728
> - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them
> - randomize the startup time of task trackers in a fixed interval
> - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.