svn commit: r606854 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/ReduceTask.java

[dd...@apache.org]

Author: ddas
Date: Wed Dec 26 00:37:27 2007
New Revision: 606854

URL: http://svn.apache.org/viewvc?rev=606854&view=rev
Log:
HADOOP-2486. When the list of files from the InMemoryFileSystem is obtained for merging, this patch will ensure that only those files whose checksums have also got created (renamed) are returned. Contributed by Devaraj Das.

Modified:
    lucene/hadoop/trunk/CHANGES.txt
    lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java

Modified: lucene/hadoop/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?rev=606854&r1=606853&r2=606854&view=diff
==============================================================================
--- lucene/hadoop/trunk/CHANGES.txt (original)
+++ lucene/hadoop/trunk/CHANGES.txt Wed Dec 26 00:37:27 2007
@@ -333,6 +333,10 @@
     round-robin disk selections randomly. This helps in spreading data across
     multiple partitions much better. (acmurhty)
 
+    HADOOP-2486. When the list of files from the InMemoryFileSystem is obtained
+    for merging, this patch will ensure that only those files whose checksums
+    have also got created (renamed) are returned. (ddas)
+
   IMPROVEMENTS
 
     HADOOP-2160.  Remove project-level, non-user documentation from

Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java
URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java?rev=606854&r1=606853&r2=606854&view=diff
==============================================================================
--- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java (original)
+++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java Wed Dec 26 00:37:27 2007
@@ -1386,8 +1386,11 @@
       public void run() {
         LOG.info(reduceTask.getTaskId() + " Thread started: " + getName());
         try {
+          Path[] inMemClosedFiles;
           //initiate merge
-          Path[] inMemClosedFiles = inMemFileSys.getFiles(MAP_OUTPUT_FILTER);
+          synchronized (ReduceTask.this) {
+            inMemClosedFiles = inMemFileSys.getFiles(MAP_OUTPUT_FILTER);
+          }
           //Note that the above Path[] could be of length 0 if all copies are 
           //in flight. So we make sure that we have some 'closed' map
           //output files to merge to get the benefit of in-memory merge