[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557502#comment-13557502 ] 

Karthik Kambatla commented on HADOOP-9230:
------------------------------------------

checkAgainstLegacy() compares the generated splits against a legacy split generation. I don't quite understand the purpose behind this check. Can anyone who knows this better throw some light on why we need the test.

I noticed a conflict in the math between UniformSizeInputFormat split generation and the legacy generation:

Current:
{code}
long nBytesPerSplit = (long) Math.ceil(totalSizeBytes * 1.0 / numSplits);
{code}

Legacy:
{code}
final long targetsize = totalFileSize / numSplits;
{code}

I would expect the math discrepancy to lead to more than 10% failure rate though.
                
> TestUniformSizeInputFormat fails intermittently
> -----------------------------------------------
>
>                 Key: HADOOP-9230
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9230
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 2.0.2-alpha
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>              Labels: distcp
>
> TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times and noticed 5 failures.
> Haven't noticed any particular pattern to which runs fail.
> A sample stack trace is as follows:
> {noformat}
> java.lang.AssertionError: expected:<1944> but was:<1820>
>         at org.junit.Assert.fail(Assert.java:91)
>         at org.junit.Assert.failNotEquals(Assert.java:645)
>         at org.junit.Assert.assertEquals(Assert.java:126)
>         at org.junit.Assert.assertEquals(Assert.java:470)
>         at org.junit.Assert.assertEquals(Assert.java:454)
>         at org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
>         at org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
>         at org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira