[jira] Commented: (HADOOP-6029) TestReduceFetch failed.

["Chris Douglas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720392#action_12720392 ] 

Chris Douglas commented on HADOOP-6029:
---------------------------------------

bq. In ReduceTask.ReduceCopier.ShuffleRamManager, a memory reservation is done for in-memory shuffle, and that uses Runtime.getRuntime().maxMemory(). The return value of this seems to be machine-dependent.

This should be rigged by TestReduceFetch in mapred.child.java.opts, and match {{-Xmx128m}}.

{{testReduceFromPartialMem}} is an awkward test to write. Its intent is to configure the reduce so that- presented with a set of crafted map outputs- it will make a particular guess about how to optimize its I/O. If we can't rig the total memory because \-Xmx has a machine-dependent interpretation, then writing such a test will be a real pain with our current set of configuration options. We could add a parameter for the memory reservation that defaults to querying the runtime; that would let us be certain of our memory limit, but not burden the user with setting it. I'm really surprised that setting \-Xmx doesn't work, though...

The failure for {{testReduceFromMem}} suggests this should use the counters added in HADOOP-2774 rather than the FileSystem counters to validate the result.

> TestReduceFetch failed.
> -----------------------
>
>                 Key: HADOOP-6029
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6029
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Tsz Wo (Nicholas), SZE
>         Attachments: FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, TEST-org.apache.hadoop.mapred.TestReduceFetch.txt
>
>
> {noformat}
> Testcase: testReduceFromMem took 23.625 sec
> 	FAILED
> Non-zero read from local: 83
> junit.framework.AssertionFailedError: Non-zero read from local: 83
> 	at org.apache.hadoop.mapred.TestReduceFetch.testReduceFromMem(TestReduceFetch.java:289)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
> 	at junit.extensions.TestSetup.run(TestSetup.java:27)
> {noformat}
> Ran TestReduceFetch a few times on a clean trunk.  It failed consistently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.