You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Harsh J <ha...@cloudera.com> on 2011/06/24 21:06:46 UTC
Review Request: MAPREDUCE-1347. Missing synchronization in
MultipleOutputFormat.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/953/
-----------------------------------------------------------
Review request for hadoop-mapreduce and Todd Lipcon.
Summary
-------
Used the makeComputingMap from Guava's MapMaker to provide a thread safe way of creating a RecordWriter cache.
For some reason, the map is not really caching it and is instead trying to apply() over and over again for the same key-value pairs.
This addresses bug MAPREDUCE-1347.
http://issues.apache.org/jira/browse/MAPREDUCE-1347
Diffs
-----
mapreduce/ivy.xml 85ee014
mapreduce/ivy/libraries.properties 9d40aaa
mapreduce/src/java/org/apache/hadoop/mapred/lib/MultipleOutputFormat.java b8944f1
mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMultipleTextOutputFormat.java 14c097d
Diff: https://reviews.apache.org/r/953/diff
Testing
-------
Added a test case, but it fails with the current behavior of MapMaker's makeComputingMap() (would pass if its alright)
Thanks,
Harsh
Re: Review Request: MAPREDUCE-1347. Missing synchronization in
MultipleOutputFormat.
Posted by Harsh J <ha...@cloudera.com>.
> On 2011-06-24 19:39:07, Todd Lipcon wrote:
> > I think your issue with the test case is that you're calling tMOF.getRecordWriter() inside the thread. So, each thread has its own record writer and hence has its own computing map. You should call that from outside the thread, and just have the threads do the writing.
/me bangs head on the wall.
Thanks for catching it! Tests fail/pass now (w/o and w/ old style sync).
Posting patch to JIRA in a bit, unless there are further comments beyond that?
- Harsh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/953/#review910
-----------------------------------------------------------
On 2011-06-24 19:06:46, Harsh J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/953/
> -----------------------------------------------------------
>
> (Updated 2011-06-24 19:06:46)
>
>
> Review request for hadoop-mapreduce and Todd Lipcon.
>
>
> Summary
> -------
>
> Used the makeComputingMap from Guava's MapMaker to provide a thread safe way of creating a RecordWriter cache.
>
> For some reason, the map is not really caching it and is instead trying to apply() over and over again for the same key-value pairs.
>
>
> This addresses bug MAPREDUCE-1347.
> http://issues.apache.org/jira/browse/MAPREDUCE-1347
>
>
> Diffs
> -----
>
> mapreduce/ivy.xml 85ee014
> mapreduce/ivy/libraries.properties 9d40aaa
> mapreduce/src/java/org/apache/hadoop/mapred/lib/MultipleOutputFormat.java b8944f1
> mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMultipleTextOutputFormat.java 14c097d
>
> Diff: https://reviews.apache.org/r/953/diff
>
>
> Testing
> -------
>
> Added a test case, but it fails with the current behavior of MapMaker's makeComputingMap() (would pass if its alright)
>
>
> Thanks,
>
> Harsh
>
>
Re: Review Request: MAPREDUCE-1347. Missing synchronization in
MultipleOutputFormat.
Posted by Todd Lipcon <to...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/953/#review910
-----------------------------------------------------------
I think your issue with the test case is that you're calling tMOF.getRecordWriter() inside the thread. So, each thread has its own record writer and hence has its own computing map. You should call that from outside the thread, and just have the threads do the writing.
- Todd
On 2011-06-24 19:06:46, Harsh J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/953/
> -----------------------------------------------------------
>
> (Updated 2011-06-24 19:06:46)
>
>
> Review request for hadoop-mapreduce and Todd Lipcon.
>
>
> Summary
> -------
>
> Used the makeComputingMap from Guava's MapMaker to provide a thread safe way of creating a RecordWriter cache.
>
> For some reason, the map is not really caching it and is instead trying to apply() over and over again for the same key-value pairs.
>
>
> This addresses bug MAPREDUCE-1347.
> http://issues.apache.org/jira/browse/MAPREDUCE-1347
>
>
> Diffs
> -----
>
> mapreduce/ivy.xml 85ee014
> mapreduce/ivy/libraries.properties 9d40aaa
> mapreduce/src/java/org/apache/hadoop/mapred/lib/MultipleOutputFormat.java b8944f1
> mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMultipleTextOutputFormat.java 14c097d
>
> Diff: https://reviews.apache.org/r/953/diff
>
>
> Testing
> -------
>
> Added a test case, but it fails with the current behavior of MapMaker's makeComputingMap() (would pass if its alright)
>
>
> Thanks,
>
> Harsh
>
>