You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2008/12/24 07:48:44 UTC

[jira] Created: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-4939
                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
             Project: Hadoop Core
          Issue Type: Sub-task
          Components: mapred, test
            Reporter: Devaraj Das
            Assignee: Devaraj Das
             Fix For: 0.20.0


Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4939:
--------------------------------

    Attachment: 4939.1.patch

Attached patch with the concerns addressed

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4939:
--------------------------------

    Attachment: 4939.2.patch

Attached is the updated patch. 

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.2.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661532#action_12661532 ] 

Vinod K V commented on HADOOP-4939:
-----------------------------------

bq. We can report progress of the jobs every once in a while when running the tests. Now it just stays dumb till the progress reaches the threshold values.
Sorry about that, log4j jar was missing on my classpath. It actually DOES print the job progress.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665758#action_12665758 ] 

Vinod K V commented on HADOOP-4939:
-----------------------------------

+1 for the patch.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.2.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660023#action_12660023 ] 

Devaraj Das commented on HADOOP-4939:
-------------------------------------

Arun, the PID file is looked at if it exists. I have the 'ps | grep | etc' to take care of the case where the test is run using HOD (as is normally the case in Yahoo!). In that case, the PID file is not written. Also, note that stopping/resuming daemons through 'ps | grep | etc' will stop/resume only those processes that had been launched by the user running the test.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660016#action_12660016 ] 

Arun C Murthy commented on HADOOP-4939:
---------------------------------------

I only started looking through this, a nit: we shouldn't do 'ps | grep | ... ' etc. for suspend/resume. Rather we should use the pid file to get the daemon's pid.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660082#action_12660082 ] 

Vinod K V commented on HADOOP-4939:
-----------------------------------

In the minimum, we should do our best and ensure that we are sending signals to the right process. For this, we might want to grep process list for "java" AND the full class name, instead of just searching for the daemons' names. We are limiting to one user anyways, and further we are just stopping and continuing the process, and not really destroying them.

Gone throught the patch, a few code comments:
 - runSleepJobTest, runRandomWriterTest and runSortTest can be refactored, they share much code.
 - If we can somehow get TASKTRACKER_EXPIRY_INTERVAL from JT - either via a public API or may be via clusterStatus - it would make the tests better as compared to just relying on the user input.
 - In KillTrackerThread.{startTaskTrackers|stopTaskTrackers}, the output of the shellCommand is currently discarded. That, along with the return code will give more information about the success of the signal sent.
 - If configuration is not setup(mapred-site.xml), local jobrunner would be used and the test fails with little error reporting. I think we can check for jt configuration in the minimum.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4939:
--------------------------------

    Attachment: 4939.patch

Attaching a patch for review. I am still tuning some parameters in the tests but the patch can be reviewed as such.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das resolved HADOOP-4939.
---------------------------------

      Resolution: Fixed
    Release Note: This adds a test for doing reliability testing. Many big jobs are started and random tasks are failed/killed as also the tasktrackers. The test is to see that the jobs succeed in the event of these large scale failures.
    Hadoop Flags: [Reviewed]

I just committed this. 

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.2.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661503#action_12661503 ] 

Vinod K V commented on HADOOP-4939:
-----------------------------------

Few comments:
 - The patch is breaking compilation because of the change in ClusterStatus constructor:
  -- src/mapred/org/apache/hadoop/mapred/LocalJobRunner.java +389
  -- src/test/org/apache/hadoop/mapred/TestJobQueueTaskScheduler.java:138
 - When sleepJob(or rather examples) are not on the path, it fails but with the output as follows:
        {code}
            JOB org.apache.hadoop.examples.SleepJob failed to run
            Waiting for the job org.apache.hadoop.examples.SleepJob to start
        {code}
    We should avoid the last line, *if* we can.
 - We can report progress of the jobs every once in a while when running the tests. Now it just stays dumb till the progress reaches the threshold values.
 - I think writing statements to a LOG is better than printing on standard output.
 - With a HOD allocation, the lost TaskTrackers simulating testcase fails even though keys are setup. This is because hadoop-daemons.sh tries on remote nodes to change to the non-existend directory HADOOP_HOME.
        {code}
            exec "$bin/slaves.sh" --config $HADOOP_CONF_DIR cd "$HADOOP_HOME" \; "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@"
        {code}
    A simple solution would be to throw away all changes from hadoop-daemon.sh and hadoop-daemons.sh and simpley use slaves.sh as follows:
        {code}
            HOSTLIST=conf/_reliability_test_slaves_file_ ./bin/slaves.sh ls
        {code}
 - the -ww flag to ps (ps auxw -ww) is not available on cygwin. It only modifies screen output and can be avoided. A side nit that I observed is that SIGCONT doesn't seem to work on cygwin. That would make the lost tasktracker simulation test completely useless on cygwin.
 - The randomness of failures is pretty peculiar in the tests. Though it can be admitted that it can be changed later if need be.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4939) Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-4939:
------------------------------------

    Release Note:   (was: This adds a test for doing reliability testing. Many big jobs are started and random tasks are failed/killed as also the tasktrackers. The test is to see that the jobs succeed in the event of these large scale failures.)

Edit release note for publication; tests not user-facing.

> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4939
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred, test
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4939.1.patch, 4939.2.patch, 4939.patch
>
>
> Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.