[jira] [Resolved] (HADOOP-2462) MiniMRCluster does not utilize multiple local directories in "mapred.local.dir"

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved HADOOP-2462.
--------------------------------------

    Resolution: Incomplete

Stale.

> MiniMRCluster does not utilize multiple local directories in "mapred.local.dir"
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-2462
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2462
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.15.0
>            Reporter: Konstantin Shvachko
>
> My hadoop-site.xml specifies 4 local directories
> {code}
> <property>
>   <name>mapred.local.dir</name>
>   <value>${hadoop.tmp.dir}/mapred/local1, ${hadoop.tmp.dir}/mapred/local2, 
>          ${hadoop.tmp.dir}/mapred/local3, ${hadoop.tmp.dir}/mapred/local4</value>
> </property>
> {code}
> and I am looking at MiniMRCluster.TaskTrackerRunner
> There are several things here:
> # localDirBase value is set to
> {code}
> "/tmp/h/mapred/local1, /tmp/h/mapred/local2, /tmp/h/mapred/local3, /tmp/h/mapred/local4"
> {code}
> and I get a hierarchy of directories with commas and spaces in the names. 
> I think this was not designed to work with multiple dirs.
> # Further down, all new directories are generated with the same name
> {code}
>         File ttDir = new File(localDirBase, 
>                               Integer.toString(trackerId) + "_" + 0);
> {code}
> So in fact only one directory is created. I think the intension was to have i instead of 0
> {code}
>         File ttDir = new File(localDirBase, 
>                               Integer.toString(trackerId) + "_" + i);
> {code}
> # On windows MiniMRCluster.TaskTrackerRunner in this case throws an IOException, 
> which is silently ignored by all but the TestMiniMRMapRedDebugScript   MiniMR tests.
> {code}
> java.io.IOException: Mkdirs failed to create 
> /tmp/h/mapred/local1, /tmp/h/mapred/local2, /tmp/h/mapred/local3, /tmp/h/mapred/local4/0_0
> 	at org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.<init>(MiniMRCluster.java:124)
> 	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:293)
> 	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:244)
> 	at org.apache.hadoop.mapred.TestMiniMRClasspath.testClassPath(TestMiniMRClasspath.java:163)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:585)
> 	at junit.framework.TestCase.runTest(TestCase.java:154)
> 	at junit.framework.TestCase.runBare(TestCase.java:127)
> 	at junit.framework.TestResult$1.protect(TestResult.java:106)
> 	at junit.framework.TestResult.runProtected(TestResult.java:124)
> 	at junit.framework.TestResult.run(TestResult.java:109)
> 	at junit.framework.TestCase.run(TestCase.java:118)
> 	at junit.framework.TestSuite.runTest(TestSuite.java:208)
> 	at junit.framework.TestSuite.run(TestSuite.java:203)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:478)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:344)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
> {code}
> I am marking it as "Major" because we actually do not test multiple local directories.
> Looks like it was introduced rather recently by HADOOP-1819.



--
This message was sent by Atlassian JIRA
(v6.2#6252)