svn commit: r1362474 - /hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/mapreduce/loadtest/LoadTest.java

[mb...@apache.org]

Author: mbautin
Date: Tue Jul 17 12:51:11 2012
New Revision: 1362474

URL: http://svn.apache.org/viewvc?rev=1362474&view=rev
Log:
[master] Modify the load tester to indefinitely retry tasks.

Author: pritam

Summary:
The load tester has a problem where all the tasks share the
same jmx ports and hence a task might fail if two tasks run on the same
machine due to a bind exception. Now if this task fails 4 times then the
entire job fails. Modifying this such that the job handles a very large
number of task failures and retries them for a long time.

Test Plan: Compile

Reviewers: kranganathan, liyintang, mbautin

Reviewed By: mbautin

CC: hbase-eng@, pkhemani

Differential Revision: https://phabricator.fb.com/D519615

Modified:
    hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/mapreduce/loadtest/LoadTest.java

Modified: hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/mapreduce/loadtest/LoadTest.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/mapreduce/loadtest/LoadTest.java?rev=1362474&r1=1362473&r2=1362474&view=diff
==============================================================================
--- hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/mapreduce/loadtest/LoadTest.java (original)
+++ hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/mapreduce/loadtest/LoadTest.java Tue Jul 17 12:51:11 2012
@@ -59,6 +59,11 @@ public class LoadTest extends Configured
   
   public static final double DEFAULT_PROFILING_FRACTION = 0.001;
 
+  // Since all tasks share the same jmx port, some tasks might fail since
+  // they might run on the same machine and try to bind to the same jmx port.
+  // Alleviating this situation by retrying tasks as long as we can.
+  public static final int MAX_REDUCE_TASK_ATTEMPTS = Integer.MAX_VALUE;
+
   public static class Map
       extends Mapper<LongWritable, Text, LongWritable, Text> {
 
@@ -264,6 +269,7 @@ public class LoadTest extends Configured
         "-Dcom.sun.management.jmxremote.authenticate=false" +
         " -Dcom.sun.management.jmxremote.ssl=false" +
         " -Dcom.sun.management.jmxremote.port=" + jmxPort);
+    conf.setInt("mapred.reduce.max.attempts", MAX_REDUCE_TASK_ATTEMPTS);
 
     SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");
     String currentDate = dateFormat.format(new Date());