You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Matei Zaharia (JIRA)" <ji...@apache.org> on 2009/01/09 18:51:59 UTC
[jira] Created: (HADOOP-5005) Create a mock MapReduce cluster
simulator to test schedulers
Create a mock MapReduce cluster simulator to test schedulers
------------------------------------------------------------
Key: HADOOP-5005
URL: https://issues.apache.org/jira/browse/HADOOP-5005
Project: Hadoop Core
Issue Type: Test
Components: test
Reporter: Matei Zaharia
Currently the Hadoop schedulers use a FakeTaskTrackerManager to run tests which is both messy and doesn't really simulate trackers going up and down, tasks finishing at different times, faiulres, etc. It would be nice to have a simulated MapReduce cluster where tasks really do take different amounts of (simulated) time, trackers may be slow, tasks can be made to fail, etc. The existing TaskTrackerManager interface given to the schedulers, plus perhaps a mockable clock (e.g. the FairScheduler.Clock class) should be enough to do all this. The end result will be easier-to-write and more complex scheduler tests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5005) Create a mock MapReduce cluster
simulator to test schedulers
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662634#action_12662634 ]
Matei Zaharia commented on HADOOP-5005:
---------------------------------------
I actually saw that, but this is meant to be a cluster that just simulates the nodes rather than running tasktrackers in their own threads. This should allow for faster tests and more tests. It's an idea I was talking about to Owen, Sameer and Eric the other day, but not necessarily a short-term goal.
> Create a mock MapReduce cluster simulator to test schedulers
> ------------------------------------------------------------
>
> Key: HADOOP-5005
> URL: https://issues.apache.org/jira/browse/HADOOP-5005
> Project: Hadoop Core
> Issue Type: Test
> Components: test
> Reporter: Matei Zaharia
>
> Currently the Hadoop schedulers use a FakeTaskTrackerManager to run tests which is both messy and doesn't really simulate trackers going up and down, tasks finishing at different times, faiulres, etc. It would be nice to have a simulated MapReduce cluster where tasks really do take different amounts of (simulated) time, trackers may be slow, tasks can be made to fail, etc. The existing TaskTrackerManager interface given to the schedulers, plus perhaps a mockable clock (e.g. the FairScheduler.Clock class) should be enough to do all this. The end result will be easier-to-write and more complex scheduler tests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5005) Create a mock MapReduce cluster
simulator to test schedulers
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662612#action_12662612 ]
Hemanth Yamijala commented on HADOOP-5005:
------------------------------------------
Matei, we recently built a MiniMR based cluster for the capacity scheduler. Please look at HADOOP-4830. Can you see if that's along the lines you wanted ? If yes, maybe we can explore changing that to support schedulers generically.
> Create a mock MapReduce cluster simulator to test schedulers
> ------------------------------------------------------------
>
> Key: HADOOP-5005
> URL: https://issues.apache.org/jira/browse/HADOOP-5005
> Project: Hadoop Core
> Issue Type: Test
> Components: test
> Reporter: Matei Zaharia
>
> Currently the Hadoop schedulers use a FakeTaskTrackerManager to run tests which is both messy and doesn't really simulate trackers going up and down, tasks finishing at different times, faiulres, etc. It would be nice to have a simulated MapReduce cluster where tasks really do take different amounts of (simulated) time, trackers may be slow, tasks can be made to fail, etc. The existing TaskTrackerManager interface given to the schedulers, plus perhaps a mockable clock (e.g. the FairScheduler.Clock class) should be enough to do all this. The end result will be easier-to-write and more complex scheduler tests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.