You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/12/31 01:31:29 UTC

[jira] Created: (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat

Missing synchronization in MultipleOutputFormat
-----------------------------------------------

                 Key: MAPREDUCE-1347
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 0.20.2, 0.21.0, 0.22.0
            Reporter: Todd Lipcon


MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams.

>From what I can tell, the new API's MultipleOutputs seems not to have this issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.