You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Anirban Dasgupta (JIRA)" <ji...@apache.org> on 2009/12/01 02:21:24 UTC
[jira] Created: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Making Mumak work with Capacity-Scheduler
-----------------------------------------
Key: MAPREDUCE-1253
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/mumak
Affects Versions: 0.21.0, 0.22.0
Reporter: Anirban Dasgupta
Assignee: Anirban Dasgupta
Priority: Minor
In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1253:
---------------------------------
Status: Patch Available (was: Open)
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch, MAPREDUCE-1253-20100804.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1253:
---------------------------------
Status: Patch Available (was: Open)
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1253:
---------------------------------
Priority: Major (was: Minor)
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1253:
-------------------------------------
Fix Version/s: 0.22.0
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch, MAPREDUCE-1253-20100804.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897380#action_12897380 ]
Hong Tang commented on MAPREDUCE-1253:
--------------------------------------
I have reviewed Anirban's earlier and I forgot to comment with +1.
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch, MAPREDUCE-1253-20100804.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1253:
---------------------------------
Attachment: MAPREDUCE-1253-20100726-2.patch
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Priority: Minor
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1253:
---------------------------------
Attachment: MAPREDUCE-1253-20100804.patch
Fixed a findbugs warning.
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch, MAPREDUCE-1253-20100804.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1253:
-------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
+1 for the patch. I just committed this. ant test for mumak pass.
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch, MAPREDUCE-1253-20100804.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1253:
---------------------------------
Status: Open (was: Patch Available)
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Anirban Dasgupta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anirban Dasgupta updated MAPREDUCE-1253:
----------------------------------------
Attachment: MAPREDUCE-1253-20100406.patch
I am attaching a partial patch at this point. In offline testing I am observing some javadoc and findbugs warnings. It seems that the main build.xml has to be changed to remove these.
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Priority: Minor
> Attachments: MAPREDUCE-1253-20100406.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784627#action_12784627 ]
Vinod K V commented on MAPREDUCE-1253:
--------------------------------------
No idea how the Mumak test code is written, but various test-cases in {{TestCapacityScheduler}} already replace the initialization thread with a {{ControlledInitializationPoller}} for the sake of deterministic testing. Please see {{testJobInitialization()}} if it helps.
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Priority: Minor
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895621#action_12895621 ]
Hong Tang commented on MAPREDUCE-1253:
--------------------------------------
Both test-patch and mumak unit tests pass on my local machine (I forgot to copy the console output of test-patch results).
{noformat}
test:
[echo] contrib: mumak
[delete] Deleting directory /grid/0/htang/hadoop-mapreduce/build/contrib/mumak/test/logs
[mkdir] Created dir: /grid/0/htang/hadoop-mapreduce/build/contrib/mumak/test/logs
[junit] WARNING: multiple versions of ant detected in path for junit
[junit] jar:file:/homes/htang/installed/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit] and jar:file:/homes/htang/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
[junit] Running org.apache.hadoop.mapred.TestRemoveIpsFromLoggedNetworkTopology
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.586 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorDeterministicReplay
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 38.469 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorEndToEnd
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 107.055 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorEngine
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.162 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorEventQueue
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.02 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorJobClient
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.063 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorJobTracker
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.123 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorSerialJobSubmission
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 139.603 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorStressJobSubmission
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 100.492 sec
[junit] Running org.apache.hadoop.mapred.TestSimulatorTaskTracker
[junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 0.397 sec
{noformat}
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch, MAPREDUCE-1253-20100804.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1253) Making Mumak work with
Capacity-Scheduler
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892690#action_12892690 ]
Hong Tang commented on MAPREDUCE-1253:
--------------------------------------
All mumak unit tests passed on my local machine after applying patch mr-1834-20100727.patch from MAPREDUCE-1834.
> Making Mumak work with Capacity-Scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Anirban Dasgupta
> Assignee: Anirban Dasgupta
> Priority: Minor
> Attachments: MAPREDUCE-1253-20100406.patch, MAPREDUCE-1253-20100726-2.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.