[jira] [Commented] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat

["Harsh J (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13563853#comment-13563853 ] 

Harsh J commented on MAPREDUCE-1347:
------------------------------------

Since this patch makes a safe change with a properly working test (that fails without the fix, exposing the issue cleanly), and has been awaiting further review for long (has already addressed previous comments), I'll go ahead and commit it in by tomorrow EOD unless anyone else has any objections. Thanks! :)
                
> Missing synchronization in MultipleOutputFormat
> -----------------------------------------------
>
>                 Key: MAPREDUCE-1347
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 1.0.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J
>              Labels: concurrency
>         Attachments: MAPREDUCE-1347.r10.diff, mapreduce.1347.r1.diff, MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff, MAPREDUCE-1347.r9.diff
>
>
> MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams.
> From what I can tell, the new API's MultipleOutputs seems not to have this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira