[jira] Updated: (HADOOP-1556) 9 unit test failures: file.out.index already exists

["Devaraj Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1556:
--------------------------------

    Attachment: 1556.patch

I did some analysis but couldn't pinpoint the exact cause of this issue. I did notice that all the failures emanate out of the LocalJobRunner. What seems to happen is that files are left over from a previous job (test) and hence rename fails for the current test, and all tests thereafter. 
I have attached a patch that makes sure that files are deleted at the end of a LocalJob run. It also handles the case of #reduces=0.

> 9 unit test failures: file.out.index already exists
> ---------------------------------------------------
>
>                 Key: HADOOP-1556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1556
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Nigel Daley
>             Fix For: 0.14.0
>
>         Attachments: 1556.patch
>
>
> In my nightly unit testing, 9 unit tests failed June 25 and again July 2, all with this exception:
> java.io.IOException: Target build/test/mapred/local/map_0005/file.out.index already exists
> 	at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:246)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:93)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:207)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:476)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:490)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:593)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:190)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:131)
> I'm concerned this could be a race conditrion.  The unit tests are:
> org.apache.hadoop.fs.TestDFSIO.testIOs
> org.apache.hadoop.fs.TestFileSystem.testFs
> org.apache.hadoop.mapred.TestAggregates.testAggregates
> org.apache.hadoop.mapred.TestFieldSelection.testFieldSelection
> org.apache.hadoop.mapred.TestLocalMRNotification.testMR
> org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch
> org.apache.hadoop.mapred.TestMapRed.testMapred
> org.apache.hadoop.mapred.TestMapRed.testCompression
> org.apache.hadoop.record.TestRecordMR.testMapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.