You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Iyappan Srinivasan (JIRA)" <ji...@apache.org> on 2010/07/12 10:27:50 UTC
[jira] Created: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Create automated testcase for tasktracker dealing with corrupted disk.
----------------------------------------------------------------------
Key: MAPREDUCE-1933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Reporter: Iyappan Srinivasan
Assignee: Iyappan Srinivasan
After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Iyappan Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated MAPREDUCE-1933:
------------------------------------------
Attachment: MAPREDUCE-1933.patch
cancel.delegation token string literal replaced with constant. I couldnt find a costant for mapred.local.dir.
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Iyappan Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated MAPREDUCE-1933:
------------------------------------------
Attachment: 1933-ydist-security-patch.txt
patch for 20.1.xxx
review comments addressed.
1) String literals not used.
2) JTClient::isJobStopped used.
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Balaji Rajagopalan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887690#action_12887690 ]
Balaji Rajagopalan commented on MAPREDUCE-1933:
-----------------------------------------------
You have attached the java file, remember to create a patch.
prop.put("mapred.local.dir",
"/grid/0/dev/tmp/mapred/mapred-local,/grid/1/dev/tmp/mapred/mapred-local,/grid/2/dev/tmp/mapred/mapred-local,/grid/3/dev/tmp/mapred/mapred-local");
Please don't hard code cluster dependent values, have them come from config file.
//Making sure that the job is complete.
while (jInfo != null && !jInfo.getStatus().isJobComplete()) {
Thread.sleep(10000);
jInfo = remoteJTClient.getJobInfo(id);
jStatus = jInfo.getStatus();
}
Please use the building block JTClient::isJobStopped.
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888468#action_12888468 ]
Konstantin Boudnik commented on MAPREDUCE-1933:
-----------------------------------------------
bq. prop.put("mapred.local.dir", "/grid/0/dev/tmp/mapred/mapred-local,/grid/1/dev/tmp/mapred/mapred-local,/grid/2/dev/tmp/mapred/mapred-local,/grid/3/dev/tmp/mapred/mapred-local");
Absolutely, besides this particular parameter should be set by a normal MR config already.
Also, please don't use string literals for configuration parameters. There was a significant effort in 0.21 to have all configuration keys refactored to named constants. Use them instead.
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Iyappan Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated MAPREDUCE-1933:
------------------------------------------
Attachment: MAPREDUCE-1933.patch
patch for trunk
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Iyappan Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated MAPREDUCE-1933:
------------------------------------------
Attachment: 1933-ydist-security-patch.txt
the same patch with only a change of path in RemoteExecution
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891725#action_12891725 ]
Konstantin Boudnik commented on MAPREDUCE-1933:
-----------------------------------------------
Let's see...
- {{find src -name *java | xargs grep 'mapred.*.local.dir'}} shows that
{noformat}
src/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java:
Configuration.addDeprecation("mapred.local.dir",
{noformat}
- also it finds
{noformat}
src/java/org/apache/hadoop/mapreduce/MRConfig.java:
public static final String LOCAL_DIR = "mapreduce.cluster.local.dir";
{noformat}
Hope it helps.
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Iyappan Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated MAPREDUCE-1933:
------------------------------------------
Attachment: TestCorruptedDiskJob.java
Patch for 20.1.xxx
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Balaji Rajagopalan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891468#action_12891468 ]
Balaji Rajagopalan commented on MAPREDUCE-1933:
-----------------------------------------------
+1
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893905#action_12893905 ]
Konstantin Boudnik commented on MAPREDUCE-1933:
-----------------------------------------------
- I still see that string "mapred.local.dir" is used all over the place in the trunk patch. First, don't use string literals where there's a named constant for the same; Second: this is a deprecated parameter: javac will curse and raise the level of warnings.
- Shall these two be swapped, actually?
{noformat}
+ cluster.tearDown();
+ cluster.restart();
{noformat}
Perhaps, cleaning the folders should also be done before the {{tearDown()}} call?
- Is there defined constants for something like "test.randomwrite.bytes_per_map" ? If not - define a private member of the test class instead of using string literals.
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1933) Create automated testcase for
tasktracker dealing with corrupted disk.
Posted by "Iyappan Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated MAPREDUCE-1933:
------------------------------------------
Attachment: MAPREDUCE-1933.patch
trunk patch
> Create automated testcase for tasktracker dealing with corrupted disk.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-1933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Reporter: Iyappan Srinivasan
> Assignee: Iyappan Srinivasan
> Attachments: 1933-ydist-security-patch.txt, 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
>
>
> After the TaskTracker has already run some tasks successfully, "corrupt" a disk by making the corresponding mapred.local.dir unreadable/unwritable.
> Make sure that jobs continue to succeed even though some tasks scheduled there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.