You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2014/07/29 23:43:40 UTC
[jira] [Updated] (MAPREDUCE-1347) Missing synchronization in
MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-1347:
----------------------------------------
Target Version/s: (was: )
Status: Open (was: Patch Available)
> Missing synchronization in MultipleOutputFormat
> -----------------------------------------------
>
> Key: MAPREDUCE-1347
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 1.0.0
> Reporter: Todd Lipcon
> Assignee: Harsh J
> Labels: concurrency
> Attachments: MAPREDUCE-1347.r10.diff, MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff, MAPREDUCE-1347.r9.diff, mapreduce.1347.r1.diff
>
>
> MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams.
> From what I can tell, the new API's MultipleOutputs seems not to have this issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)