[jira] Updated: (HADOOP-4830) Have end to end tests based on MiniMRCluster to verify that queue capacities are honoured.

["Vinod K V (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4830:
------------------------------

    Attachment: HADOOP-4830-20090106-2-svn.txt

Attaching a new patch. Incorporated the above review comments. Notes about particular points follow:

bq. We are iterating over the task list to get the number of running tasks in ControlledMapReduceJob.getRunningTasksCount(). We check if the task is running using TaskInProgress.isRunning(). This method of computation seems like it would not be different from JobInProgress.runningMaps() or JobInProgress.runningReduces(). Can you please check if there is a difference ?
As pointed out, there is no real difference. The original intention was to ensure that so and so number of tasks are really running on TaskTrackers. But on retrospection, I came to conclude that the number of tasks scheduled(JIP.runningTasks()) should suffice. This is because the number of tasks scheduled is same as the number of tasks running in the test environment(no lost trackers). Made changes to use JIP.runningTasks() regarding the same.

bq. This may be that we need to fix something in ivy.xml of the capacity scheduler. Copying the ivy.xml from streaming ran the test successfully.
Made changes to ivy.xml in capacity scheduler src to include the needed jars. But, as Hemanth also concurs during a discussion, it may become cumbersome in future to add every new jar that capacity scheduler might not need directly but still needed to be included because of the dependency on underlying projects/modules say mapred/hdfs. Will file a new issue to see if this can be addressed generally.

While running tests, found that some tests were timing out. The actual reason turned out to be HADOOP-4977. The test TestQueueCapacities might fail sometimes because of that and till that gets fixed.

ant test-patch results:
{code}
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
{code}

> Have end to end tests based on MiniMRCluster to verify that queue capacities are honoured.
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4830
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4830
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-4830-20081222-svn.2, HADOOP-4830-20081229-svn.txt, HADOOP-4830-20090106-2-svn.txt
>
>
> At present, we only have unit tests that make use of FakeTaskManager and that only test the proper functionality of capacity scheduler in isolation. Many issues unearthed recently proved that this is not enough and that it is required to have end-to-end tests so that real JT is brought into the picture and with that the interaction of the scheduler with JT. This issue along with few other related jiras should automate and replace the end-to-end tests that are now manually done by QA, using MiniMRCluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.