You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinay Kumar Thota (JIRA)" <ji...@apache.org> on 2010/03/30 12:21:27 UTC
[jira] Created: (MAPREDUCE-1646) Task Killing tests
Task Killing tests
------------------
Key: MAPREDUCE-1646
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
Project: Hadoop Map/Reduce
Issue Type: Task
Components: test
Reporter: Vinay Kumar Thota
The following tasks covered in the test.
1. In a running job, kill a task and verify the job succeeds.
2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
output/_temporary/_attempt-id directory is cleaned up.
3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: TaskKilling_1646.patch
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852531#action_12852531 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
One more thing: the class has about 10 unused import statement. Please optimize the import list.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853536#action_12853536 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
bq. I have checked your log file and the first test(testFailedTaskJobStatus ) failed due to task has not been started for almost 1 min, so it got failed.And the other two tests are failing due to error in configuration object. I am suspecting the failures might occurred due to environmental issue.
- Shall the limit be raised slightly? Say 2 mins?
- 'Good citizen' test should be unrelated to environment as much as possible. If you can provide some special settings in the test config - you should do so, If test requires a particular environment to exist at the moment of a test execution and such env. setting doesn't exist then the test in question should fail with proper and meaningful error message. E.g. a person who runs tests shouldn't be guessing the required environment.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854904#action_12854904 ]
Vinay Kumar Thota commented on MAPREDUCE-1646:
----------------------------------------------
Shall the limit be raised slightly? Say 2 mins?
I agreed with you for increasing the time slightly 2 mins. However, If a task is not started almost 1min, in that case there might be some issue with task tracker.As per my opinion whenever job started, atleast one task should be started with in fraction of seconds. Overall I am saying, in these condition situations we failing the test and giving the proper message instead of increasing the time.
'Good citizen' test should be unrelated to environment as much as possible. If you can provide some special settings in the test config - you should do so, If test requires a particular environment to exist at the moment of a test execution and such env. setting doesn't exist then the test in question should fail with proper and meaningful error message. E.g. a person who runs tests shouldn't be guessing the required environment.
I agreed with you, however currently I have checked all the possible conditions in the test. As per your log,it say some class is missing (com.hadoop.compression.lzo.LzoCodec not found) and it seems to me some library file is missing at your environment,so that test is failing.
I had been running the test couple of times on 5 node multi cluster and it passes consistently every time. Please check the attached log file(TEST-org.apache.hadoop.mapred.TestTaskKilling.txt).
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: TaskKilling_1646.patch
All the comments incorporated into the latest patch.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: (was: TaskKilling_1646.patch)
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: TaskKilling_1646.patch
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852526#action_12852526 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
- It seems like you have some issues with the IDE configuration.
{noformat}
+ while (counter < 240) {
+ UtilsForTests.waitFor(1000);
+ counter ++;
+ }
+ if (counter == 240 ) {
+ throw new IOException();
+ }
+ }
{noformat}
and many places like this. Please fix.
- also, I've tried to run it and it failed (see attached log)
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851352#action_12851352 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1646:
----------------------------------------------------
You can have look at TestTaskFail, which covers the cases for different types of failures.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851999#action_12851999 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
Patch looks mostly good. Except a couple of nits:
- uses a mix of JUnit v3 and v4 APIs, i.e.
{noformat}
+import junit.framework.Assert;
+import org.junit.BeforeClass;
+import org.junit.AfterClass;
+import org.junit.Test;
{noformat}
- indentation isn't uniformed and is anywhere from 1 to 14 spaces (!!!)
{noformat}
+ if (taskInfo.getTaskStatus()[0].getRunState()
+ == TaskStatus.State.RUNNING) {
+ break;
...
+ if (!taskinfo.isSetupOrCleanup()) {
+ taskInfo = taskinfo;
...
+ while (counter < 60) {
+ if (jInfo.getStatus().getRunState() == JobStatus.RUNNING) {
+ break;
+ } else {
+ UtilsForTests.waitFor(1000);
+ jInfo = wovenClient.getJobInfo(id);
+ }
{noformat}
- seems like the LOG statement is out of place:
{noformat}
+ while (!jInfo.getStatus().isJobComplete()) {
+ UtilsForTests.waitFor(100);
+ jInfo = wovenClient.getJobInfo(id);
+ }
+ LOG.info("Waiting till the job is completed...");
{noformat}
Or the the message has to be changed to "The job is completed".
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895921#action_12895921 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
Looks like there wasn't been any activity on this one for quite some time. What's the status ? Is it ready for trunk commit yet?
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1646) Task Killing tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal reassigned MAPREDUCE-1646:
-----------------------------------------
Assignee: Vinay Kumar Thota
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852529#action_12852529 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
Well, looks like {noformat} did a bad job in the above snippet case. What I've meant is that in say line 128 all indentation is done with whitespaces, but in line 129 and other 3 after it the indentation is tabs + whitespace. Different IDEs might be configured differently about the size of the tabs. I.e. for Hadoop style the size of tab is 2. Thus in some people editors tabs will be looking differently which might lead to unnecessary reformatting of the code.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855854#action_12855854 ]
Vinay Kumar Thota commented on MAPREDUCE-1646:
----------------------------------------------
Both the functions are two different tests and I am thinking its not good idea to combine two functions into one because both the tests has two different ways for killing and failing the task. For failing the task, we are defining a mapper to fail the task by itself but for killing the task , there is no user define mapper and killing the task by passing the task attempt id. So I don't think for combining the both the functions into one is a good idea .Also both are two different scenarios,so finally one more thing,if you combine both the functions into one, it will become tedious to debug for user also. In my opinion maintaining two different functions, i am mean two different tests would be better.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: (was: TaskKilling_1646.patch)
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851593#action_12851593 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
Looking good. Some small items:
- "... has not been started till 1 min". Would it be more appropriate to say "...has not been started
for 1 min." ?
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: (was: TaskKilling_1646.patch)
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Boudnik updated MAPREDUCE-1646:
------------------------------------------
Attachment: TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay Kumar Thota updated MAPREDUCE-1646:
-----------------------------------------
Attachment: TaskKilling_1646.patch
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853343#action_12853343 ]
Vinay Kumar Thota commented on MAPREDUCE-1646:
----------------------------------------------
You are absolutely true, I have an issue with IDE configuration.Even If i set the tab size 2 and aligning the code accordingly, but it reformatting the code while generating the patch file so that you are seeing the indentation issues.Now I have given the spaces instead of tabs so that it behaves correctly across the IDE's. I hope this time you won't find any indentation issues in the latest patch.
also, I've tried to run it and it failed (see attached log)
Vinay: I have run all 3 tests on multi cluster couple of times and I didn't see any issues and all the tests are passing consistently.Please check the attached log file.
I have checked your log file and the first test(testFailedTaskJobStatus ) failed due to task has not been started for almost 1 min, so it got failed.And the other two tests are failing due to error in configuration object. I am suspecting the failures might occurred due to environmental issue.
One more thing: the class has about 10 unused import statement. Please optimize the import list.
Vinay : I have optimized the import list.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851502#action_12851502 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
An appropriate JIRA number would be much helpful aside from the test name.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855263#action_12855263 ]
Konstantin Boudnik commented on MAPREDUCE-1646:
-----------------------------------------------
Ok, sounds good to me. Most probably the issue I'm seeing is caused by the lack of LZO codec. If there's no other comments from anyone please commit it to the internal Y20 branch and I'll commit it to the trunk as soon as HADOOP-6332 is ready.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Balaji Rajagopalan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855849#action_12855849 ]
Balaji Rajagopalan commented on MAPREDUCE-1646:
-----------------------------------------------
In the next iteration try to combine the functions testDirCleanupAfterTaskFailed and testDirCleanupAfterTaskKilled into one function, since I see lot of commonality in these two functions.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Assignee: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch, TaskKilling_1646.patch, TaskKilling_1646.patch, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TEST-org.apache.hadoop.mapred.TestTaskKilling.txt
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1646) Task Killing tests
Posted by "Vinay Kumar Thota (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851715#action_12851715 ]
Vinay Kumar Thota commented on MAPREDUCE-1646:
----------------------------------------------
I have changed the error message based on you review comments given earlier.The patch which has attached to this ticket has appropriate error messages.
> Task Killing tests
> ------------------
>
> Key: MAPREDUCE-1646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1646
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
> Reporter: Vinay Kumar Thota
> Attachments: TaskKilling_1646.patch
>
>
> The following tasks covered in the test.
> 1. In a running job, kill a task and verify the job succeeds.
> 2. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Kill the task. After the task is killed, make sure that the
> output/_temporary/_attempt-id directory is cleaned up.
> 3. Setup a job with long running tasks that write some output to HDFS. When one of the tasks is running, ensure that
> the output/_temporary/_attempt-id directory is created. Fail the task by simulating the map. After the task is failed,
> make sure that the output/_temporary/_attempt-id directory is cleaned up. The important difference we are trying to
> check is btw kill and fail, there would a subtle difference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.