You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by Björn-Elmar Macek <ma...@cs.uni-kassel.de> on 2012/09/05 13:56:25 UTC

Re: mortbay, huge files and the ulimit

Hello again,

i just wanted to keep you updated, in case anybody reads this and is 
interested in the mortbay-issue:
I applied the wordcount example to a big input file and everything 
worked fine. I watched the TT logs all the time and no single 
mortbay-eof exception occurred. The only possible explanation for me 
seemed to be, that my code somehow produces the exceptions. So i started 
to keep my code as close to the wordcount example as possible:

* first getting rid of any members within the reducer class, that may 
take lots of memory.
* using the Context class instead of OutputCollector (in Mapper and Reducer)
* removing any Comparator (and restructuring the code so it just 
calculates the same as with the sorting)
* using standart HashPartitioner

Especially the first 2 points made my job succeed in the end, ALTHOUGH 
the problem of "n-1 mortbay-exception-throwing TTs" still remains: one 
node seems to complete all the work while the others are busy with their 
exceptions (at least from what i can see in the log).

I REALLY dont know what i can do, to make this job run through like the 
wordcount example. I dont see any big difference between my program and 
the example anymore.

I added the code here:

*Please* help me here.

Best regards,
Elmar


I added the code here:
########################################
package uni.kassel.macek.rtprep;

import java.util.Date;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class RetweetApplication extends Configured implements Tool{

     public static int DEBUG = -1;
     private Configuration jobConf;

     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new 
Date()).toString();

     public static void main(String[] args) {
          int res=-1;
         try {
             res = ToolRunner.run(new Configuration(), new 
RetweetApplication(), args);
         } catch (Exception e) {
             // TODO Auto-generated catch block
             e.printStackTrace();
         }
         System.exit(res);

     }

     public void setConf(Configuration conf) {
         jobConf = conf;

     }

     public Configuration getConf() {
         return jobConf;
     }

     public int run(String[] args) throws Exception {
         JobClient client = new JobClient();
         JobConf conf = new JobConf(RetweetApplication.class);


         if(DEBUG > 0){
             conf.set("mapred.job.tracker", "local");
             conf.set("fs.default.name", "file:///");
             conf.set("dfs.replication", "1");
         }

         //conf.set("mapred.child.java.opts","-Xmx3000m 
-XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
         conf.set("mapred.child.ulimit","100000000");

         FileInputFormat.setInputPaths(conf, new Path(args[0]));
         FileOutputFormat.setOutputPath(conf, new Path(args[1]));


         //conf.setOutputKeyClass(Text.class);
         //conf.setOutputValueClass(Text.class);
         conf.setMapOutputKeyClass(Text.class);
         conf.setMapOutputValueClass(Text.class);

         conf.setMapperClass(RetweetMapper.class);
         //conf.setPartitionerClass(TweetPartitioner.class);
//conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
//conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
         conf.setReducerClass(RetweetReducer.class);

         conf.setOutputFormat(TextOutputFormat.class);

         client.setConf(conf);
         try {
             JobClient.runJob(conf);
         } catch (Exception e) {
             e.printStackTrace();
         }
         return 0;
     }
}
*/

package uni.kassel.macek.rtprep;



import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text, 
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null && 
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new 
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new 
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor()), new 
Text(aTweet.toString()));
             } catch (InterruptedException e) {
                 // TODO Auto-generated catch block
                 e.printStackTrace();
             }

         }
     }

public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
Text> {

     public void reduce(Text atweet, Iterator<Text> tweets,
             Context context)
             throws IOException {
         List<Tweet> originals = new ArrayList<Tweet>();
         List<Tweet> retweets = new ArrayList<Tweet>();
         while(tweets.hasNext()){
             Tweet aTweet = new Tweet(tweets.toString());
             originals.add(aTweet);
             if(aTweet.getRT()>0) retweets.add(aTweet);
         }

         for(Tweet originalTweet : originals){
             for(Tweet aRetweet : retweets){
                 if(aRetweet.isRetweeted(originalTweet))
                 {
aRetweet.setReactionTime(originalTweet.getDate());
                     try {
                         context.write(
                                 atweet,
                                 new Text(aRetweet.toString()));
                     } catch (InterruptedException e) {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                     }
                 }
             }
         }
     }
}

   public static void main(String[] args) throws Exception {
     Configuration conf = new Configuration();
     String[] otherArgs = new GenericOptionsParser(conf, 
args).getRemainingArgs();
     if (otherArgs.length != 2) {
       System.err.println("Usage: wordcount <in> <out>");
       System.exit(2);
     }
     Job job = new Job(conf, "tweets");
     job.setJarByClass(RetweetApplication.class);
     job.setMapperClass(RetweetMapper1.class);
     job.setReducerClass(RetweetReducer1.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);
     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
     System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
> Hi all,
>
> i got some more information:
>
> It is always only one TT, where mortbay errors do not occur: all other 
> slaves fail relatively fast after the reduce process started: maybe it 
> is a comparator issue (since the operation is MAPRED_SHUFFLE)
> And it has nothing to do with the local configuration of the server, 
> since
> 1) if i rerun the job on the same set of servers, the ONE working 
> differs from one job to another.
> 2) the same happens even when i am using a completely different set of 
> servers.
>
> Who is capable to solve this unsolved mystery? :D
>
> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>> Hi,
>>
>> is there anybody, who knows more about this issue: it has already 
>> been recently marked here:
>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>
>> I really want to do something about it, if i knew how... i tried so 
>> many different setup parameters and JVM options and nothing did the 
>> trick...
>>
>> It was opened 2009 and gets still updated (2012): I mean - this does 
>> NOT look OK, does it? The whole TT seems like a big time waster:
>>
>> 2012-08-30 12:05:22,582 INFO 
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, 
>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, 
>> duration: 5456122852
>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: 
>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>> injecting delay2 times
>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: 
>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>> recreating selector 2 times, canceled keys 70 times
>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>> org.mortbay.jetty.EofException
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>     at 
>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: java.io.IOException: Broken pipe
>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>     at 
>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>     ... 27 more
>>
>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>> org.mortbay.jetty.EofException
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>     at 
>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: java.io.IOException: Broken pipe
>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>     at 
>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>     ... 27 more
>>
>> 2012-08-30 12:08:39,236 INFO 
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>> 141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>> duration: 3209410
>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> 2012-08-30 12:10:27,109 INFO 
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>> 141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, 
>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>> duration: 5686096517
>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>> org.mortbay.jetty.EofException
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>     at 
>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: java.io.IOException: Broken pipe
>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>     at 
>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>     ... 27 more
>>
>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>> org.mortbay.jetty.EofException
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>     at 
>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: java.io.IOException: Broken pipe
>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>     at 
>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>     ... 27 more
>>
>> 2012-08-30 12:14:29,740 INFO 
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>> 141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>> duration: 3746212
>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> 2012-08-30 12:16:28,503 INFO 
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>> 141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, 
>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>> duration: 7793391393
>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: 
>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>> injecting delay1 times
>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: 
>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>> recreating selector 1 times, canceled keys 69 times
>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>> org.mortbay.jetty.EofException
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>     at 
>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: java.io.IOException: Broken pipe
>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>     at 
>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>     ... 27 more
>>
>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>> org.mortbay.jetty.EofException
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>     at 
>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>     at 
>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: java.io.IOException: Broken pipe
>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>     at 
>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>     ... 27 more
>>
>> 2012-08-30 12:20:01,205 INFO 
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>> 141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, 
>> duration: 5405031
>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>> Hi there,
>>>
>>> i am currently running a job where i selfjoin a 63 gigabyte big csv 
>>> file on 20 physically distinct nodes with 15GB each:
>>>
>>> While the mapping works just fine and is low cost, the reducer does 
>>> the main work: holding a hashmap with elements to join with and 
>>> finding join tuples for evry incoming key-value-pair.
>>>
>>> The jobs works perfectly on small files with 2 gigabytes, but starts 
>>> to get "unstable" as the file size goes up: this becomes evident 
>>> with a look into the tasktracker's logs saying:
>>>
>>> ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>>
>>> And while it is no problem at the beginning of the reduce process, 
>>> where this happens only on a few nodes and rarely, it becomes 
>>> crucial as the progress rises. The reason for this (afaik from 
>>> reading articles), is that there are memory or file handle problems. 
>>> I addressed the memory problem by conitiously purging the map of 
>>> outdated elements evry 5 million processed key-value-pairs. And i 
>>> set mapred.child.ulimit to 100000000 (ulimit in the shell tells me 
>>> it is 400000000).
>>>
>>> Anyway i am still running into those mortbay errors and i start to 
>>> wonder, if hadoop can manage the job with this algorithmn anyways. 
>>> By pure naive math it should be:
>>> i explicily assigned 10GB memory to each JVM on each node and set 
>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
>>> datastructures cause the GC to throw exceptions). This would naively 
>>> make 18 slave machines with 10GB each resulting in an overall memory 
>>> of 180GB - three times as much as needed... i would think. So if the 
>>> Partitioner distributes them just about equally to all nodes i 
>>> should not run into any errors, do i?
>>>
>>> Can anybody help me with this issue?
>>>
>>> Best regards,
>>> Elmar
>>>
>>>
>>
>>
>
>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

i think both heuristics dont imply anything about the sense or nonsense 
of the program:
The first version of my program used the "values" as "keys", so i could 
accomplish some sorting. But this also meant, that since each value is 
unique my iterator in the reduce method was also always of size 1. So i 
had to store possible joinpartners in a reduce-function-external 
variable and i decided to put it as a membervariable, which i regularly 
cleaned of outdated data. This data structure grew with every call of 
the reduce function and with this the size of non-garbage-collectable 
data, while the code in my class didnt do too much: since i used a 
hashmap here, there was only very few time-consuming traversing needed 
and the execution time of my code was most probably smaller than the 
execution time of the GC since a certain treshold. In fact: it is easy 
to create a program that does meaningfull stuff and still trigger the 
98%-2% rule and let the GC throw an exception.

My 2 cents,
Elmar
Am 05.09.2012 15:32, schrieb Vasco Visser:
> Now that I think about it I wonder about oracle's description.
>
> [...] if more than 98% of the total time is spent in garbage collection [...]
>
> I took that to mean 98% of CPU time for the application, but it can
> also mean that the GC is active for 98% percent of wall clock time.
>
> If it is the former it implies the application isn't doing useful
> stuff anymore (e.g., keeping a connection open). If it is the latter I
> don't know if it possible to say anything about how much the program
> is still doing useful stuff.
>
>
>
> On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Hi Vasco,
>>
>> thank you for your help!
>>
>> I can try to add the limit again (i currently have it turned off for all
>> Java processes spawned by Hadoop). Also i do not have any persistent
>> (member-) variables, that could store things that use alot of data: at the
>> very moment i only got 2 local variables in the reduce-method, that could
>> get a little larger - but those should be gc'ed easily, shouldnt they?
>>
>> Thank you for your thoughts again. I will pay with the GC a little. If i
>> know the reason, i'll let you know!
>> Best regards,
>> Elmar
>>
>>
>> Am 05.09.2012 14:38, schrieb Vasco Visser:
>>
>>> Just a guess, but could it simply be memory issue on reducer?
>>>
>>> In your adjusted program, maybe try running without
>>> -UseGCOverheadLimit and see if you still got OOM errors.
>>>
>>>   From the sun website:
>>> The parallel collector will throw an OutOfMemoryError if too much time
>>> is being spent in garbage collection: if more than 98% of the total
>>> time is spent in garbage collection and less than 2% of the heap is
>>> recovered, an OutOfMemoryError will be thrown. This feature is
>>> designed to prevent applications from running for an extended period
>>> of time while making little or no progress because the heap is too
>>> small. If necessary, this feature can be disabled by adding the option
>>> -XX:-UseGCOverheadLimit to the command line.
>>>
>>> So basically if you get a GC overhead limit exceeded OOM error it
>>> means that your app is doing nothing but garbage collection, the vm is
>>> fighting against running out of memory. If that happens it might
>>> result in a timeout of the connection fetching the map data (i don't
>>> know if it can result in timeout, but I can imagine it could).
>>>
>>> Also, note that 63 GB on disk is probably going to be inflated in
>>> memory. So in general you cant say that 60 GB on disk will need 60GB
>>> of mem. Actually, some people us a rule of thumb to do x4 to get
>>> approx mem requirement.
>>>
>>> Just some ideas, not really a solution but maybe it helps you further.
>>>
>>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>>> <ma...@cs.uni-kassel.de> wrote:
>>>> Excuse me: in my last code section was some old code included. Here is it
>>>> again stripped of deprecated code:
>>>>
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>           public void map(Object key, Text value,
>>>>               Context context)
>>>>               throws IOException {
>>>>
>>>>               String line = value.toString();
>>>>
>>>>               Tweet aTweet = new Tweet(line);
>>>>               if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                   /*aTweet.setRT(0);
>>>>                   output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                   aTweet.setRT(1);
>>>>                   try {
>>>>                       context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                   } catch (InterruptedException e) {
>>>>                       // TODO Auto-generated catch block
>>>>                       e.printStackTrace();
>>>>                   }
>>>>               }
>>>>               aTweet.setRT(0);
>>>>               try {
>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>               } catch (InterruptedException e) {
>>>>                   // TODO Auto-generated catch block
>>>>                   e.printStackTrace();
>>>>               }
>>>>
>>>>           }
>>>>       }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text>
>>>> {
>>>>
>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>               Context context)
>>>>               throws IOException {
>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>           while(tweets.hasNext()){
>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>               originals.add(aTweet);
>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>           }
>>>>
>>>>           for(Tweet originalTweet : originals){
>>>>               for(Tweet aRetweet : retweets){
>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>                   {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                       try {
>>>>                           context.write(
>>>>                                   atweet,
>>>>                                   new Text(aRetweet.toString()));
>>>>                       } catch (InterruptedException e) {
>>>>                           // TODO Auto-generated catch block
>>>>                           e.printStackTrace();
>>>>                       }
>>>>                   }
>>>>               }
>>>>           }
>>>>       }
>>>> }
>>>>
>>>>     public static void main(String[] args) throws Exception {
>>>>       Configuration conf = new Configuration();
>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>       if (otherArgs.length != 2) {
>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>         System.exit(2);
>>>>       }
>>>>       Job job = new Job(conf, "word count");
>>>>
>>>>       job.setJarByClass(RetweetApplication.class);
>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>       job.setOutputKeyClass(Text.class);
>>>>       job.setOutputValueClass(Text.class);
>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>     }
>>>> }
>>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>>
>>>>> Hello again,
>>>>>
>>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>>> interested in the mortbay-issue:
>>>>> I applied the wordcount example to a big input file and everything
>>>>> worked
>>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>>> exception
>>>>> occurred. The only possible explanation for me seemed to be, that my
>>>>> code
>>>>> somehow produces the exceptions. So i started to keep my code as close
>>>>> to
>>>>> the wordcount example as possible:
>>>>>
>>>>> * first getting rid of any members within the reducer class, that may
>>>>> take
>>>>> lots of memory.
>>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>>> Reducer)
>>>>> * removing any Comparator (and restructuring the code so it just
>>>>> calculates the same as with the sorting)
>>>>> * using standart HashPartitioner
>>>>>
>>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>>> the
>>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>>> seems to complete all the work while the others are busy with their
>>>>> exceptions (at least from what i can see in the log).
>>>>>
>>>>> I REALLY dont know what i can do, to make this job run through like the
>>>>> wordcount example. I dont see any big difference between my program and
>>>>> the
>>>>> example anymore.
>>>>>
>>>>> I added the code here:
>>>>>
>>>>> *Please* help me here.
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>> I added the code here:
>>>>> ########################################
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>> import java.util.Date;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.conf.Configured;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>>> import org.apache.hadoop.mapred.JobClient;
>>>>> import org.apache.hadoop.mapred.JobConf;
>>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>>> import org.apache.hadoop.util.Tool;
>>>>> import org.apache.hadoop.util.ToolRunner;
>>>>>
>>>>> public class RetweetApplication extends Configured implements Tool{
>>>>>
>>>>>       public static int DEBUG = -1;
>>>>>       private Configuration jobConf;
>>>>>
>>>>>       //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>>       //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>>> Date()).toString();
>>>>>
>>>>>       public static void main(String[] args) {
>>>>>            int res=-1;
>>>>>           try {
>>>>>               res = ToolRunner.run(new Configuration(), new
>>>>> RetweetApplication(), args);
>>>>>           } catch (Exception e) {
>>>>>               // TODO Auto-generated catch block
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           System.exit(res);
>>>>>
>>>>>       }
>>>>>
>>>>>       public void setConf(Configuration conf) {
>>>>>           jobConf = conf;
>>>>>
>>>>>       }
>>>>>
>>>>>       public Configuration getConf() {
>>>>>           return jobConf;
>>>>>       }
>>>>>
>>>>>       public int run(String[] args) throws Exception {
>>>>>           JobClient client = new JobClient();
>>>>>           JobConf conf = new JobConf(RetweetApplication.class);
>>>>>
>>>>>
>>>>>           if(DEBUG > 0){
>>>>>               conf.set("mapred.job.tracker", "local");
>>>>>               conf.set("fs.default.name", "file:///");
>>>>>               conf.set("dfs.replication", "1");
>>>>>           }
>>>>>
>>>>>           //conf.set("mapred.child.java.opts","-Xmx3000m
>>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>>           conf.set("mapred.child.ulimit","100000000");
>>>>>
>>>>>           FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>>           FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>>
>>>>>
>>>>>           //conf.setOutputKeyClass(Text.class);
>>>>>           //conf.setOutputValueClass(Text.class);
>>>>>           conf.setMapOutputKeyClass(Text.class);
>>>>>           conf.setMapOutputValueClass(Text.class);
>>>>>
>>>>>           conf.setMapperClass(RetweetMapper.class);
>>>>>           //conf.setPartitionerClass(TweetPartitioner.class);
>>>>>
>>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>>
>>>>>
>>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>>           conf.setReducerClass(RetweetReducer.class);
>>>>>
>>>>>           conf.setOutputFormat(TextOutputFormat.class);
>>>>>
>>>>>           client.setConf(conf);
>>>>>           try {
>>>>>               JobClient.runJob(conf);
>>>>>           } catch (Exception e) {
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           return 0;
>>>>>       }
>>>>> }
>>>>> */
>>>>>
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>>
>>>>>
>>>>> import gnu.trove.iterator.TIntIterator;
>>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>>> import gnu.trove.set.TIntSet;
>>>>>
>>>>> import java.io.IOException;
>>>>> import java.util.ArrayList;
>>>>> import java.util.Calendar;
>>>>> import java.util.Iterator;
>>>>> import java.util.List;
>>>>> import java.util.StringTokenizer;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.IntWritable;
>>>>> import org.apache.hadoop.io.LongWritable;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>>> import org.apache.hadoop.mapred.Reporter;
>>>>> import org.apache.hadoop.mapreduce.Job;
>>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>>
>>>>> public class RetweetApplication {
>>>>>
>>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>>> Text,
>>>>> Text> {
>>>>>
>>>>>           public void map(Object key, Text value,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>
>>>>>               String line = value.toString();
>>>>>
>>>>>               Tweet aTweet = new Tweet(line);
>>>>>               if(aTweet.getMention()!=null &&
>>>>> aTweet.getMention().length()>0){
>>>>>                   /*aTweet.setRT(0);
>>>>>                   output.collect(new Text(aTweet.toString()), new
>>>>> Text(aTweet.toString()));*/
>>>>>                   aTweet.setRT(1);
>>>>>                   try {
>>>>>                       context.write(new Text(aTweet.getMention()), new
>>>>> Text(aTweet.toString()));
>>>>>                   } catch (InterruptedException e) {
>>>>>                       // TODO Auto-generated catch block
>>>>>                       e.printStackTrace();
>>>>>                   }
>>>>>               }
>>>>>               aTweet.setRT(0);
>>>>>               try {
>>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>>> Text(aTweet.toString()));
>>>>>               } catch (InterruptedException e) {
>>>>>                   // TODO Auto-generated catch block
>>>>>                   e.printStackTrace();
>>>>>               }
>>>>>
>>>>>           }
>>>>>       }
>>>>>
>>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>>> Text> {
>>>>>
>>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>>           while(tweets.hasNext()){
>>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>>               originals.add(aTweet);
>>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>>           }
>>>>>
>>>>>           for(Tweet originalTweet : originals){
>>>>>               for(Tweet aRetweet : retweets){
>>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>>                   {
>>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>>                       try {
>>>>>                           context.write(
>>>>>                                   atweet,
>>>>>                                   new Text(aRetweet.toString()));
>>>>>                       } catch (InterruptedException e) {
>>>>>                           // TODO Auto-generated catch block
>>>>>                           e.printStackTrace();
>>>>>                       }
>>>>>                   }
>>>>>               }
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>>     public static void main(String[] args) throws Exception {
>>>>>       Configuration conf = new Configuration();
>>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>>> args).getRemainingArgs();
>>>>>       if (otherArgs.length != 2) {
>>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>>         System.exit(2);
>>>>>       }
>>>>>       Job job = new Job(conf, "tweets");
>>>>>       job.setJarByClass(RetweetApplication.class);
>>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>>       job.setOutputKeyClass(Text.class);
>>>>>       job.setOutputValueClass(Text.class);
>>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>>     }
>>>>> }
>>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>> Hi all,
>>>>>>
>>>>>> i got some more information:
>>>>>>
>>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>>> is a
>>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>>> And it has nothing to do with the local configuration of the server,
>>>>>> since
>>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>>> differs
>>>>>> from one job to another.
>>>>>> 2) the same happens even when i am using a completely different set of
>>>>>> servers.
>>>>>>
>>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>>
>>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>> Hi,
>>>>>>>
>>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>>> recently marked here:
>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>>
>>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>>> many
>>>>>>> different setup parameters and JVM options and nothing did the
>>>>>>> trick...
>>>>>>>
>>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>>> NOT
>>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>>
>>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay2 times
>>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay1 times
>>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>>
>>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>>> the
>>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>>> join
>>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>>
>>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>>> to
>>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>>> look
>>>>>>>> into the tasktracker's logs saying:
>>>>>>>>
>>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>>
>>>>>>>>
>>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>>> as the
>>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>>> that
>>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>>> problem by
>>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>>> processed
>>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>>> in the
>>>>>>>> shell tells me it is 400000000).
>>>>>>>>
>>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>>> pure
>>>>>>>> naive math it should be:
>>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>>> datastructures
>>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>>> machines
>>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>>> as much
>>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>>> just
>>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>>
>>>>>>>> Can anybody help me with this issue?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Elmar
>>>>>>>>
>>>>>>>>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

i think both heuristics dont imply anything about the sense or nonsense 
of the program:
The first version of my program used the "values" as "keys", so i could 
accomplish some sorting. But this also meant, that since each value is 
unique my iterator in the reduce method was also always of size 1. So i 
had to store possible joinpartners in a reduce-function-external 
variable and i decided to put it as a membervariable, which i regularly 
cleaned of outdated data. This data structure grew with every call of 
the reduce function and with this the size of non-garbage-collectable 
data, while the code in my class didnt do too much: since i used a 
hashmap here, there was only very few time-consuming traversing needed 
and the execution time of my code was most probably smaller than the 
execution time of the GC since a certain treshold. In fact: it is easy 
to create a program that does meaningfull stuff and still trigger the 
98%-2% rule and let the GC throw an exception.

My 2 cents,
Elmar
Am 05.09.2012 15:32, schrieb Vasco Visser:
> Now that I think about it I wonder about oracle's description.
>
> [...] if more than 98% of the total time is spent in garbage collection [...]
>
> I took that to mean 98% of CPU time for the application, but it can
> also mean that the GC is active for 98% percent of wall clock time.
>
> If it is the former it implies the application isn't doing useful
> stuff anymore (e.g., keeping a connection open). If it is the latter I
> don't know if it possible to say anything about how much the program
> is still doing useful stuff.
>
>
>
> On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Hi Vasco,
>>
>> thank you for your help!
>>
>> I can try to add the limit again (i currently have it turned off for all
>> Java processes spawned by Hadoop). Also i do not have any persistent
>> (member-) variables, that could store things that use alot of data: at the
>> very moment i only got 2 local variables in the reduce-method, that could
>> get a little larger - but those should be gc'ed easily, shouldnt they?
>>
>> Thank you for your thoughts again. I will pay with the GC a little. If i
>> know the reason, i'll let you know!
>> Best regards,
>> Elmar
>>
>>
>> Am 05.09.2012 14:38, schrieb Vasco Visser:
>>
>>> Just a guess, but could it simply be memory issue on reducer?
>>>
>>> In your adjusted program, maybe try running without
>>> -UseGCOverheadLimit and see if you still got OOM errors.
>>>
>>>   From the sun website:
>>> The parallel collector will throw an OutOfMemoryError if too much time
>>> is being spent in garbage collection: if more than 98% of the total
>>> time is spent in garbage collection and less than 2% of the heap is
>>> recovered, an OutOfMemoryError will be thrown. This feature is
>>> designed to prevent applications from running for an extended period
>>> of time while making little or no progress because the heap is too
>>> small. If necessary, this feature can be disabled by adding the option
>>> -XX:-UseGCOverheadLimit to the command line.
>>>
>>> So basically if you get a GC overhead limit exceeded OOM error it
>>> means that your app is doing nothing but garbage collection, the vm is
>>> fighting against running out of memory. If that happens it might
>>> result in a timeout of the connection fetching the map data (i don't
>>> know if it can result in timeout, but I can imagine it could).
>>>
>>> Also, note that 63 GB on disk is probably going to be inflated in
>>> memory. So in general you cant say that 60 GB on disk will need 60GB
>>> of mem. Actually, some people us a rule of thumb to do x4 to get
>>> approx mem requirement.
>>>
>>> Just some ideas, not really a solution but maybe it helps you further.
>>>
>>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>>> <ma...@cs.uni-kassel.de> wrote:
>>>> Excuse me: in my last code section was some old code included. Here is it
>>>> again stripped of deprecated code:
>>>>
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>           public void map(Object key, Text value,
>>>>               Context context)
>>>>               throws IOException {
>>>>
>>>>               String line = value.toString();
>>>>
>>>>               Tweet aTweet = new Tweet(line);
>>>>               if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                   /*aTweet.setRT(0);
>>>>                   output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                   aTweet.setRT(1);
>>>>                   try {
>>>>                       context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                   } catch (InterruptedException e) {
>>>>                       // TODO Auto-generated catch block
>>>>                       e.printStackTrace();
>>>>                   }
>>>>               }
>>>>               aTweet.setRT(0);
>>>>               try {
>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>               } catch (InterruptedException e) {
>>>>                   // TODO Auto-generated catch block
>>>>                   e.printStackTrace();
>>>>               }
>>>>
>>>>           }
>>>>       }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text>
>>>> {
>>>>
>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>               Context context)
>>>>               throws IOException {
>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>           while(tweets.hasNext()){
>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>               originals.add(aTweet);
>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>           }
>>>>
>>>>           for(Tweet originalTweet : originals){
>>>>               for(Tweet aRetweet : retweets){
>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>                   {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                       try {
>>>>                           context.write(
>>>>                                   atweet,
>>>>                                   new Text(aRetweet.toString()));
>>>>                       } catch (InterruptedException e) {
>>>>                           // TODO Auto-generated catch block
>>>>                           e.printStackTrace();
>>>>                       }
>>>>                   }
>>>>               }
>>>>           }
>>>>       }
>>>> }
>>>>
>>>>     public static void main(String[] args) throws Exception {
>>>>       Configuration conf = new Configuration();
>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>       if (otherArgs.length != 2) {
>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>         System.exit(2);
>>>>       }
>>>>       Job job = new Job(conf, "word count");
>>>>
>>>>       job.setJarByClass(RetweetApplication.class);
>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>       job.setOutputKeyClass(Text.class);
>>>>       job.setOutputValueClass(Text.class);
>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>     }
>>>> }
>>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>>
>>>>> Hello again,
>>>>>
>>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>>> interested in the mortbay-issue:
>>>>> I applied the wordcount example to a big input file and everything
>>>>> worked
>>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>>> exception
>>>>> occurred. The only possible explanation for me seemed to be, that my
>>>>> code
>>>>> somehow produces the exceptions. So i started to keep my code as close
>>>>> to
>>>>> the wordcount example as possible:
>>>>>
>>>>> * first getting rid of any members within the reducer class, that may
>>>>> take
>>>>> lots of memory.
>>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>>> Reducer)
>>>>> * removing any Comparator (and restructuring the code so it just
>>>>> calculates the same as with the sorting)
>>>>> * using standart HashPartitioner
>>>>>
>>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>>> the
>>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>>> seems to complete all the work while the others are busy with their
>>>>> exceptions (at least from what i can see in the log).
>>>>>
>>>>> I REALLY dont know what i can do, to make this job run through like the
>>>>> wordcount example. I dont see any big difference between my program and
>>>>> the
>>>>> example anymore.
>>>>>
>>>>> I added the code here:
>>>>>
>>>>> *Please* help me here.
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>> I added the code here:
>>>>> ########################################
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>> import java.util.Date;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.conf.Configured;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>>> import org.apache.hadoop.mapred.JobClient;
>>>>> import org.apache.hadoop.mapred.JobConf;
>>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>>> import org.apache.hadoop.util.Tool;
>>>>> import org.apache.hadoop.util.ToolRunner;
>>>>>
>>>>> public class RetweetApplication extends Configured implements Tool{
>>>>>
>>>>>       public static int DEBUG = -1;
>>>>>       private Configuration jobConf;
>>>>>
>>>>>       //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>>       //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>>> Date()).toString();
>>>>>
>>>>>       public static void main(String[] args) {
>>>>>            int res=-1;
>>>>>           try {
>>>>>               res = ToolRunner.run(new Configuration(), new
>>>>> RetweetApplication(), args);
>>>>>           } catch (Exception e) {
>>>>>               // TODO Auto-generated catch block
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           System.exit(res);
>>>>>
>>>>>       }
>>>>>
>>>>>       public void setConf(Configuration conf) {
>>>>>           jobConf = conf;
>>>>>
>>>>>       }
>>>>>
>>>>>       public Configuration getConf() {
>>>>>           return jobConf;
>>>>>       }
>>>>>
>>>>>       public int run(String[] args) throws Exception {
>>>>>           JobClient client = new JobClient();
>>>>>           JobConf conf = new JobConf(RetweetApplication.class);
>>>>>
>>>>>
>>>>>           if(DEBUG > 0){
>>>>>               conf.set("mapred.job.tracker", "local");
>>>>>               conf.set("fs.default.name", "file:///");
>>>>>               conf.set("dfs.replication", "1");
>>>>>           }
>>>>>
>>>>>           //conf.set("mapred.child.java.opts","-Xmx3000m
>>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>>           conf.set("mapred.child.ulimit","100000000");
>>>>>
>>>>>           FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>>           FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>>
>>>>>
>>>>>           //conf.setOutputKeyClass(Text.class);
>>>>>           //conf.setOutputValueClass(Text.class);
>>>>>           conf.setMapOutputKeyClass(Text.class);
>>>>>           conf.setMapOutputValueClass(Text.class);
>>>>>
>>>>>           conf.setMapperClass(RetweetMapper.class);
>>>>>           //conf.setPartitionerClass(TweetPartitioner.class);
>>>>>
>>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>>
>>>>>
>>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>>           conf.setReducerClass(RetweetReducer.class);
>>>>>
>>>>>           conf.setOutputFormat(TextOutputFormat.class);
>>>>>
>>>>>           client.setConf(conf);
>>>>>           try {
>>>>>               JobClient.runJob(conf);
>>>>>           } catch (Exception e) {
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           return 0;
>>>>>       }
>>>>> }
>>>>> */
>>>>>
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>>
>>>>>
>>>>> import gnu.trove.iterator.TIntIterator;
>>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>>> import gnu.trove.set.TIntSet;
>>>>>
>>>>> import java.io.IOException;
>>>>> import java.util.ArrayList;
>>>>> import java.util.Calendar;
>>>>> import java.util.Iterator;
>>>>> import java.util.List;
>>>>> import java.util.StringTokenizer;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.IntWritable;
>>>>> import org.apache.hadoop.io.LongWritable;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>>> import org.apache.hadoop.mapred.Reporter;
>>>>> import org.apache.hadoop.mapreduce.Job;
>>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>>
>>>>> public class RetweetApplication {
>>>>>
>>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>>> Text,
>>>>> Text> {
>>>>>
>>>>>           public void map(Object key, Text value,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>
>>>>>               String line = value.toString();
>>>>>
>>>>>               Tweet aTweet = new Tweet(line);
>>>>>               if(aTweet.getMention()!=null &&
>>>>> aTweet.getMention().length()>0){
>>>>>                   /*aTweet.setRT(0);
>>>>>                   output.collect(new Text(aTweet.toString()), new
>>>>> Text(aTweet.toString()));*/
>>>>>                   aTweet.setRT(1);
>>>>>                   try {
>>>>>                       context.write(new Text(aTweet.getMention()), new
>>>>> Text(aTweet.toString()));
>>>>>                   } catch (InterruptedException e) {
>>>>>                       // TODO Auto-generated catch block
>>>>>                       e.printStackTrace();
>>>>>                   }
>>>>>               }
>>>>>               aTweet.setRT(0);
>>>>>               try {
>>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>>> Text(aTweet.toString()));
>>>>>               } catch (InterruptedException e) {
>>>>>                   // TODO Auto-generated catch block
>>>>>                   e.printStackTrace();
>>>>>               }
>>>>>
>>>>>           }
>>>>>       }
>>>>>
>>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>>> Text> {
>>>>>
>>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>>           while(tweets.hasNext()){
>>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>>               originals.add(aTweet);
>>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>>           }
>>>>>
>>>>>           for(Tweet originalTweet : originals){
>>>>>               for(Tweet aRetweet : retweets){
>>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>>                   {
>>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>>                       try {
>>>>>                           context.write(
>>>>>                                   atweet,
>>>>>                                   new Text(aRetweet.toString()));
>>>>>                       } catch (InterruptedException e) {
>>>>>                           // TODO Auto-generated catch block
>>>>>                           e.printStackTrace();
>>>>>                       }
>>>>>                   }
>>>>>               }
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>>     public static void main(String[] args) throws Exception {
>>>>>       Configuration conf = new Configuration();
>>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>>> args).getRemainingArgs();
>>>>>       if (otherArgs.length != 2) {
>>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>>         System.exit(2);
>>>>>       }
>>>>>       Job job = new Job(conf, "tweets");
>>>>>       job.setJarByClass(RetweetApplication.class);
>>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>>       job.setOutputKeyClass(Text.class);
>>>>>       job.setOutputValueClass(Text.class);
>>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>>     }
>>>>> }
>>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>> Hi all,
>>>>>>
>>>>>> i got some more information:
>>>>>>
>>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>>> is a
>>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>>> And it has nothing to do with the local configuration of the server,
>>>>>> since
>>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>>> differs
>>>>>> from one job to another.
>>>>>> 2) the same happens even when i am using a completely different set of
>>>>>> servers.
>>>>>>
>>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>>
>>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>> Hi,
>>>>>>>
>>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>>> recently marked here:
>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>>
>>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>>> many
>>>>>>> different setup parameters and JVM options and nothing did the
>>>>>>> trick...
>>>>>>>
>>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>>> NOT
>>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>>
>>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay2 times
>>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay1 times
>>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>>
>>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>>> the
>>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>>> join
>>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>>
>>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>>> to
>>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>>> look
>>>>>>>> into the tasktracker's logs saying:
>>>>>>>>
>>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>>
>>>>>>>>
>>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>>> as the
>>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>>> that
>>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>>> problem by
>>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>>> processed
>>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>>> in the
>>>>>>>> shell tells me it is 400000000).
>>>>>>>>
>>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>>> pure
>>>>>>>> naive math it should be:
>>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>>> datastructures
>>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>>> machines
>>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>>> as much
>>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>>> just
>>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>>
>>>>>>>> Can anybody help me with this issue?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Elmar
>>>>>>>>
>>>>>>>>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

i think both heuristics dont imply anything about the sense or nonsense 
of the program:
The first version of my program used the "values" as "keys", so i could 
accomplish some sorting. But this also meant, that since each value is 
unique my iterator in the reduce method was also always of size 1. So i 
had to store possible joinpartners in a reduce-function-external 
variable and i decided to put it as a membervariable, which i regularly 
cleaned of outdated data. This data structure grew with every call of 
the reduce function and with this the size of non-garbage-collectable 
data, while the code in my class didnt do too much: since i used a 
hashmap here, there was only very few time-consuming traversing needed 
and the execution time of my code was most probably smaller than the 
execution time of the GC since a certain treshold. In fact: it is easy 
to create a program that does meaningfull stuff and still trigger the 
98%-2% rule and let the GC throw an exception.

My 2 cents,
Elmar
Am 05.09.2012 15:32, schrieb Vasco Visser:
> Now that I think about it I wonder about oracle's description.
>
> [...] if more than 98% of the total time is spent in garbage collection [...]
>
> I took that to mean 98% of CPU time for the application, but it can
> also mean that the GC is active for 98% percent of wall clock time.
>
> If it is the former it implies the application isn't doing useful
> stuff anymore (e.g., keeping a connection open). If it is the latter I
> don't know if it possible to say anything about how much the program
> is still doing useful stuff.
>
>
>
> On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Hi Vasco,
>>
>> thank you for your help!
>>
>> I can try to add the limit again (i currently have it turned off for all
>> Java processes spawned by Hadoop). Also i do not have any persistent
>> (member-) variables, that could store things that use alot of data: at the
>> very moment i only got 2 local variables in the reduce-method, that could
>> get a little larger - but those should be gc'ed easily, shouldnt they?
>>
>> Thank you for your thoughts again. I will pay with the GC a little. If i
>> know the reason, i'll let you know!
>> Best regards,
>> Elmar
>>
>>
>> Am 05.09.2012 14:38, schrieb Vasco Visser:
>>
>>> Just a guess, but could it simply be memory issue on reducer?
>>>
>>> In your adjusted program, maybe try running without
>>> -UseGCOverheadLimit and see if you still got OOM errors.
>>>
>>>   From the sun website:
>>> The parallel collector will throw an OutOfMemoryError if too much time
>>> is being spent in garbage collection: if more than 98% of the total
>>> time is spent in garbage collection and less than 2% of the heap is
>>> recovered, an OutOfMemoryError will be thrown. This feature is
>>> designed to prevent applications from running for an extended period
>>> of time while making little or no progress because the heap is too
>>> small. If necessary, this feature can be disabled by adding the option
>>> -XX:-UseGCOverheadLimit to the command line.
>>>
>>> So basically if you get a GC overhead limit exceeded OOM error it
>>> means that your app is doing nothing but garbage collection, the vm is
>>> fighting against running out of memory. If that happens it might
>>> result in a timeout of the connection fetching the map data (i don't
>>> know if it can result in timeout, but I can imagine it could).
>>>
>>> Also, note that 63 GB on disk is probably going to be inflated in
>>> memory. So in general you cant say that 60 GB on disk will need 60GB
>>> of mem. Actually, some people us a rule of thumb to do x4 to get
>>> approx mem requirement.
>>>
>>> Just some ideas, not really a solution but maybe it helps you further.
>>>
>>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>>> <ma...@cs.uni-kassel.de> wrote:
>>>> Excuse me: in my last code section was some old code included. Here is it
>>>> again stripped of deprecated code:
>>>>
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>           public void map(Object key, Text value,
>>>>               Context context)
>>>>               throws IOException {
>>>>
>>>>               String line = value.toString();
>>>>
>>>>               Tweet aTweet = new Tweet(line);
>>>>               if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                   /*aTweet.setRT(0);
>>>>                   output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                   aTweet.setRT(1);
>>>>                   try {
>>>>                       context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                   } catch (InterruptedException e) {
>>>>                       // TODO Auto-generated catch block
>>>>                       e.printStackTrace();
>>>>                   }
>>>>               }
>>>>               aTweet.setRT(0);
>>>>               try {
>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>               } catch (InterruptedException e) {
>>>>                   // TODO Auto-generated catch block
>>>>                   e.printStackTrace();
>>>>               }
>>>>
>>>>           }
>>>>       }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text>
>>>> {
>>>>
>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>               Context context)
>>>>               throws IOException {
>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>           while(tweets.hasNext()){
>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>               originals.add(aTweet);
>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>           }
>>>>
>>>>           for(Tweet originalTweet : originals){
>>>>               for(Tweet aRetweet : retweets){
>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>                   {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                       try {
>>>>                           context.write(
>>>>                                   atweet,
>>>>                                   new Text(aRetweet.toString()));
>>>>                       } catch (InterruptedException e) {
>>>>                           // TODO Auto-generated catch block
>>>>                           e.printStackTrace();
>>>>                       }
>>>>                   }
>>>>               }
>>>>           }
>>>>       }
>>>> }
>>>>
>>>>     public static void main(String[] args) throws Exception {
>>>>       Configuration conf = new Configuration();
>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>       if (otherArgs.length != 2) {
>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>         System.exit(2);
>>>>       }
>>>>       Job job = new Job(conf, "word count");
>>>>
>>>>       job.setJarByClass(RetweetApplication.class);
>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>       job.setOutputKeyClass(Text.class);
>>>>       job.setOutputValueClass(Text.class);
>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>     }
>>>> }
>>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>>
>>>>> Hello again,
>>>>>
>>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>>> interested in the mortbay-issue:
>>>>> I applied the wordcount example to a big input file and everything
>>>>> worked
>>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>>> exception
>>>>> occurred. The only possible explanation for me seemed to be, that my
>>>>> code
>>>>> somehow produces the exceptions. So i started to keep my code as close
>>>>> to
>>>>> the wordcount example as possible:
>>>>>
>>>>> * first getting rid of any members within the reducer class, that may
>>>>> take
>>>>> lots of memory.
>>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>>> Reducer)
>>>>> * removing any Comparator (and restructuring the code so it just
>>>>> calculates the same as with the sorting)
>>>>> * using standart HashPartitioner
>>>>>
>>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>>> the
>>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>>> seems to complete all the work while the others are busy with their
>>>>> exceptions (at least from what i can see in the log).
>>>>>
>>>>> I REALLY dont know what i can do, to make this job run through like the
>>>>> wordcount example. I dont see any big difference between my program and
>>>>> the
>>>>> example anymore.
>>>>>
>>>>> I added the code here:
>>>>>
>>>>> *Please* help me here.
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>> I added the code here:
>>>>> ########################################
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>> import java.util.Date;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.conf.Configured;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>>> import org.apache.hadoop.mapred.JobClient;
>>>>> import org.apache.hadoop.mapred.JobConf;
>>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>>> import org.apache.hadoop.util.Tool;
>>>>> import org.apache.hadoop.util.ToolRunner;
>>>>>
>>>>> public class RetweetApplication extends Configured implements Tool{
>>>>>
>>>>>       public static int DEBUG = -1;
>>>>>       private Configuration jobConf;
>>>>>
>>>>>       //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>>       //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>>> Date()).toString();
>>>>>
>>>>>       public static void main(String[] args) {
>>>>>            int res=-1;
>>>>>           try {
>>>>>               res = ToolRunner.run(new Configuration(), new
>>>>> RetweetApplication(), args);
>>>>>           } catch (Exception e) {
>>>>>               // TODO Auto-generated catch block
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           System.exit(res);
>>>>>
>>>>>       }
>>>>>
>>>>>       public void setConf(Configuration conf) {
>>>>>           jobConf = conf;
>>>>>
>>>>>       }
>>>>>
>>>>>       public Configuration getConf() {
>>>>>           return jobConf;
>>>>>       }
>>>>>
>>>>>       public int run(String[] args) throws Exception {
>>>>>           JobClient client = new JobClient();
>>>>>           JobConf conf = new JobConf(RetweetApplication.class);
>>>>>
>>>>>
>>>>>           if(DEBUG > 0){
>>>>>               conf.set("mapred.job.tracker", "local");
>>>>>               conf.set("fs.default.name", "file:///");
>>>>>               conf.set("dfs.replication", "1");
>>>>>           }
>>>>>
>>>>>           //conf.set("mapred.child.java.opts","-Xmx3000m
>>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>>           conf.set("mapred.child.ulimit","100000000");
>>>>>
>>>>>           FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>>           FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>>
>>>>>
>>>>>           //conf.setOutputKeyClass(Text.class);
>>>>>           //conf.setOutputValueClass(Text.class);
>>>>>           conf.setMapOutputKeyClass(Text.class);
>>>>>           conf.setMapOutputValueClass(Text.class);
>>>>>
>>>>>           conf.setMapperClass(RetweetMapper.class);
>>>>>           //conf.setPartitionerClass(TweetPartitioner.class);
>>>>>
>>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>>
>>>>>
>>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>>           conf.setReducerClass(RetweetReducer.class);
>>>>>
>>>>>           conf.setOutputFormat(TextOutputFormat.class);
>>>>>
>>>>>           client.setConf(conf);
>>>>>           try {
>>>>>               JobClient.runJob(conf);
>>>>>           } catch (Exception e) {
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           return 0;
>>>>>       }
>>>>> }
>>>>> */
>>>>>
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>>
>>>>>
>>>>> import gnu.trove.iterator.TIntIterator;
>>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>>> import gnu.trove.set.TIntSet;
>>>>>
>>>>> import java.io.IOException;
>>>>> import java.util.ArrayList;
>>>>> import java.util.Calendar;
>>>>> import java.util.Iterator;
>>>>> import java.util.List;
>>>>> import java.util.StringTokenizer;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.IntWritable;
>>>>> import org.apache.hadoop.io.LongWritable;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>>> import org.apache.hadoop.mapred.Reporter;
>>>>> import org.apache.hadoop.mapreduce.Job;
>>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>>
>>>>> public class RetweetApplication {
>>>>>
>>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>>> Text,
>>>>> Text> {
>>>>>
>>>>>           public void map(Object key, Text value,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>
>>>>>               String line = value.toString();
>>>>>
>>>>>               Tweet aTweet = new Tweet(line);
>>>>>               if(aTweet.getMention()!=null &&
>>>>> aTweet.getMention().length()>0){
>>>>>                   /*aTweet.setRT(0);
>>>>>                   output.collect(new Text(aTweet.toString()), new
>>>>> Text(aTweet.toString()));*/
>>>>>                   aTweet.setRT(1);
>>>>>                   try {
>>>>>                       context.write(new Text(aTweet.getMention()), new
>>>>> Text(aTweet.toString()));
>>>>>                   } catch (InterruptedException e) {
>>>>>                       // TODO Auto-generated catch block
>>>>>                       e.printStackTrace();
>>>>>                   }
>>>>>               }
>>>>>               aTweet.setRT(0);
>>>>>               try {
>>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>>> Text(aTweet.toString()));
>>>>>               } catch (InterruptedException e) {
>>>>>                   // TODO Auto-generated catch block
>>>>>                   e.printStackTrace();
>>>>>               }
>>>>>
>>>>>           }
>>>>>       }
>>>>>
>>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>>> Text> {
>>>>>
>>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>>           while(tweets.hasNext()){
>>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>>               originals.add(aTweet);
>>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>>           }
>>>>>
>>>>>           for(Tweet originalTweet : originals){
>>>>>               for(Tweet aRetweet : retweets){
>>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>>                   {
>>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>>                       try {
>>>>>                           context.write(
>>>>>                                   atweet,
>>>>>                                   new Text(aRetweet.toString()));
>>>>>                       } catch (InterruptedException e) {
>>>>>                           // TODO Auto-generated catch block
>>>>>                           e.printStackTrace();
>>>>>                       }
>>>>>                   }
>>>>>               }
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>>     public static void main(String[] args) throws Exception {
>>>>>       Configuration conf = new Configuration();
>>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>>> args).getRemainingArgs();
>>>>>       if (otherArgs.length != 2) {
>>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>>         System.exit(2);
>>>>>       }
>>>>>       Job job = new Job(conf, "tweets");
>>>>>       job.setJarByClass(RetweetApplication.class);
>>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>>       job.setOutputKeyClass(Text.class);
>>>>>       job.setOutputValueClass(Text.class);
>>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>>     }
>>>>> }
>>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>> Hi all,
>>>>>>
>>>>>> i got some more information:
>>>>>>
>>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>>> is a
>>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>>> And it has nothing to do with the local configuration of the server,
>>>>>> since
>>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>>> differs
>>>>>> from one job to another.
>>>>>> 2) the same happens even when i am using a completely different set of
>>>>>> servers.
>>>>>>
>>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>>
>>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>> Hi,
>>>>>>>
>>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>>> recently marked here:
>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>>
>>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>>> many
>>>>>>> different setup parameters and JVM options and nothing did the
>>>>>>> trick...
>>>>>>>
>>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>>> NOT
>>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>>
>>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay2 times
>>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay1 times
>>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>>
>>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>>> the
>>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>>> join
>>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>>
>>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>>> to
>>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>>> look
>>>>>>>> into the tasktracker's logs saying:
>>>>>>>>
>>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>>
>>>>>>>>
>>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>>> as the
>>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>>> that
>>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>>> problem by
>>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>>> processed
>>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>>> in the
>>>>>>>> shell tells me it is 400000000).
>>>>>>>>
>>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>>> pure
>>>>>>>> naive math it should be:
>>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>>> datastructures
>>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>>> machines
>>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>>> as much
>>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>>> just
>>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>>
>>>>>>>> Can anybody help me with this issue?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Elmar
>>>>>>>>
>>>>>>>>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

i think both heuristics dont imply anything about the sense or nonsense 
of the program:
The first version of my program used the "values" as "keys", so i could 
accomplish some sorting. But this also meant, that since each value is 
unique my iterator in the reduce method was also always of size 1. So i 
had to store possible joinpartners in a reduce-function-external 
variable and i decided to put it as a membervariable, which i regularly 
cleaned of outdated data. This data structure grew with every call of 
the reduce function and with this the size of non-garbage-collectable 
data, while the code in my class didnt do too much: since i used a 
hashmap here, there was only very few time-consuming traversing needed 
and the execution time of my code was most probably smaller than the 
execution time of the GC since a certain treshold. In fact: it is easy 
to create a program that does meaningfull stuff and still trigger the 
98%-2% rule and let the GC throw an exception.

My 2 cents,
Elmar
Am 05.09.2012 15:32, schrieb Vasco Visser:
> Now that I think about it I wonder about oracle's description.
>
> [...] if more than 98% of the total time is spent in garbage collection [...]
>
> I took that to mean 98% of CPU time for the application, but it can
> also mean that the GC is active for 98% percent of wall clock time.
>
> If it is the former it implies the application isn't doing useful
> stuff anymore (e.g., keeping a connection open). If it is the latter I
> don't know if it possible to say anything about how much the program
> is still doing useful stuff.
>
>
>
> On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Hi Vasco,
>>
>> thank you for your help!
>>
>> I can try to add the limit again (i currently have it turned off for all
>> Java processes spawned by Hadoop). Also i do not have any persistent
>> (member-) variables, that could store things that use alot of data: at the
>> very moment i only got 2 local variables in the reduce-method, that could
>> get a little larger - but those should be gc'ed easily, shouldnt they?
>>
>> Thank you for your thoughts again. I will pay with the GC a little. If i
>> know the reason, i'll let you know!
>> Best regards,
>> Elmar
>>
>>
>> Am 05.09.2012 14:38, schrieb Vasco Visser:
>>
>>> Just a guess, but could it simply be memory issue on reducer?
>>>
>>> In your adjusted program, maybe try running without
>>> -UseGCOverheadLimit and see if you still got OOM errors.
>>>
>>>   From the sun website:
>>> The parallel collector will throw an OutOfMemoryError if too much time
>>> is being spent in garbage collection: if more than 98% of the total
>>> time is spent in garbage collection and less than 2% of the heap is
>>> recovered, an OutOfMemoryError will be thrown. This feature is
>>> designed to prevent applications from running for an extended period
>>> of time while making little or no progress because the heap is too
>>> small. If necessary, this feature can be disabled by adding the option
>>> -XX:-UseGCOverheadLimit to the command line.
>>>
>>> So basically if you get a GC overhead limit exceeded OOM error it
>>> means that your app is doing nothing but garbage collection, the vm is
>>> fighting against running out of memory. If that happens it might
>>> result in a timeout of the connection fetching the map data (i don't
>>> know if it can result in timeout, but I can imagine it could).
>>>
>>> Also, note that 63 GB on disk is probably going to be inflated in
>>> memory. So in general you cant say that 60 GB on disk will need 60GB
>>> of mem. Actually, some people us a rule of thumb to do x4 to get
>>> approx mem requirement.
>>>
>>> Just some ideas, not really a solution but maybe it helps you further.
>>>
>>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>>> <ma...@cs.uni-kassel.de> wrote:
>>>> Excuse me: in my last code section was some old code included. Here is it
>>>> again stripped of deprecated code:
>>>>
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>           public void map(Object key, Text value,
>>>>               Context context)
>>>>               throws IOException {
>>>>
>>>>               String line = value.toString();
>>>>
>>>>               Tweet aTweet = new Tweet(line);
>>>>               if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                   /*aTweet.setRT(0);
>>>>                   output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                   aTweet.setRT(1);
>>>>                   try {
>>>>                       context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                   } catch (InterruptedException e) {
>>>>                       // TODO Auto-generated catch block
>>>>                       e.printStackTrace();
>>>>                   }
>>>>               }
>>>>               aTweet.setRT(0);
>>>>               try {
>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>               } catch (InterruptedException e) {
>>>>                   // TODO Auto-generated catch block
>>>>                   e.printStackTrace();
>>>>               }
>>>>
>>>>           }
>>>>       }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text>
>>>> {
>>>>
>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>               Context context)
>>>>               throws IOException {
>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>           while(tweets.hasNext()){
>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>               originals.add(aTweet);
>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>           }
>>>>
>>>>           for(Tweet originalTweet : originals){
>>>>               for(Tweet aRetweet : retweets){
>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>                   {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                       try {
>>>>                           context.write(
>>>>                                   atweet,
>>>>                                   new Text(aRetweet.toString()));
>>>>                       } catch (InterruptedException e) {
>>>>                           // TODO Auto-generated catch block
>>>>                           e.printStackTrace();
>>>>                       }
>>>>                   }
>>>>               }
>>>>           }
>>>>       }
>>>> }
>>>>
>>>>     public static void main(String[] args) throws Exception {
>>>>       Configuration conf = new Configuration();
>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>       if (otherArgs.length != 2) {
>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>         System.exit(2);
>>>>       }
>>>>       Job job = new Job(conf, "word count");
>>>>
>>>>       job.setJarByClass(RetweetApplication.class);
>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>       job.setOutputKeyClass(Text.class);
>>>>       job.setOutputValueClass(Text.class);
>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>     }
>>>> }
>>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>>
>>>>> Hello again,
>>>>>
>>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>>> interested in the mortbay-issue:
>>>>> I applied the wordcount example to a big input file and everything
>>>>> worked
>>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>>> exception
>>>>> occurred. The only possible explanation for me seemed to be, that my
>>>>> code
>>>>> somehow produces the exceptions. So i started to keep my code as close
>>>>> to
>>>>> the wordcount example as possible:
>>>>>
>>>>> * first getting rid of any members within the reducer class, that may
>>>>> take
>>>>> lots of memory.
>>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>>> Reducer)
>>>>> * removing any Comparator (and restructuring the code so it just
>>>>> calculates the same as with the sorting)
>>>>> * using standart HashPartitioner
>>>>>
>>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>>> the
>>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>>> seems to complete all the work while the others are busy with their
>>>>> exceptions (at least from what i can see in the log).
>>>>>
>>>>> I REALLY dont know what i can do, to make this job run through like the
>>>>> wordcount example. I dont see any big difference between my program and
>>>>> the
>>>>> example anymore.
>>>>>
>>>>> I added the code here:
>>>>>
>>>>> *Please* help me here.
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>> I added the code here:
>>>>> ########################################
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>> import java.util.Date;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.conf.Configured;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>>> import org.apache.hadoop.mapred.JobClient;
>>>>> import org.apache.hadoop.mapred.JobConf;
>>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>>> import org.apache.hadoop.util.Tool;
>>>>> import org.apache.hadoop.util.ToolRunner;
>>>>>
>>>>> public class RetweetApplication extends Configured implements Tool{
>>>>>
>>>>>       public static int DEBUG = -1;
>>>>>       private Configuration jobConf;
>>>>>
>>>>>       //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>>       //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>>> Date()).toString();
>>>>>
>>>>>       public static void main(String[] args) {
>>>>>            int res=-1;
>>>>>           try {
>>>>>               res = ToolRunner.run(new Configuration(), new
>>>>> RetweetApplication(), args);
>>>>>           } catch (Exception e) {
>>>>>               // TODO Auto-generated catch block
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           System.exit(res);
>>>>>
>>>>>       }
>>>>>
>>>>>       public void setConf(Configuration conf) {
>>>>>           jobConf = conf;
>>>>>
>>>>>       }
>>>>>
>>>>>       public Configuration getConf() {
>>>>>           return jobConf;
>>>>>       }
>>>>>
>>>>>       public int run(String[] args) throws Exception {
>>>>>           JobClient client = new JobClient();
>>>>>           JobConf conf = new JobConf(RetweetApplication.class);
>>>>>
>>>>>
>>>>>           if(DEBUG > 0){
>>>>>               conf.set("mapred.job.tracker", "local");
>>>>>               conf.set("fs.default.name", "file:///");
>>>>>               conf.set("dfs.replication", "1");
>>>>>           }
>>>>>
>>>>>           //conf.set("mapred.child.java.opts","-Xmx3000m
>>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>>           conf.set("mapred.child.ulimit","100000000");
>>>>>
>>>>>           FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>>           FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>>
>>>>>
>>>>>           //conf.setOutputKeyClass(Text.class);
>>>>>           //conf.setOutputValueClass(Text.class);
>>>>>           conf.setMapOutputKeyClass(Text.class);
>>>>>           conf.setMapOutputValueClass(Text.class);
>>>>>
>>>>>           conf.setMapperClass(RetweetMapper.class);
>>>>>           //conf.setPartitionerClass(TweetPartitioner.class);
>>>>>
>>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>>
>>>>>
>>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>>           conf.setReducerClass(RetweetReducer.class);
>>>>>
>>>>>           conf.setOutputFormat(TextOutputFormat.class);
>>>>>
>>>>>           client.setConf(conf);
>>>>>           try {
>>>>>               JobClient.runJob(conf);
>>>>>           } catch (Exception e) {
>>>>>               e.printStackTrace();
>>>>>           }
>>>>>           return 0;
>>>>>       }
>>>>> }
>>>>> */
>>>>>
>>>>> package uni.kassel.macek.rtprep;
>>>>>
>>>>>
>>>>>
>>>>> import gnu.trove.iterator.TIntIterator;
>>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>>> import gnu.trove.set.TIntSet;
>>>>>
>>>>> import java.io.IOException;
>>>>> import java.util.ArrayList;
>>>>> import java.util.Calendar;
>>>>> import java.util.Iterator;
>>>>> import java.util.List;
>>>>> import java.util.StringTokenizer;
>>>>>
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.IntWritable;
>>>>> import org.apache.hadoop.io.LongWritable;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>>> import org.apache.hadoop.mapred.Reporter;
>>>>> import org.apache.hadoop.mapreduce.Job;
>>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>>
>>>>> public class RetweetApplication {
>>>>>
>>>>>       public static class RetweetMapper1 extends Mapper<Object, Text,
>>>>> Text,
>>>>> Text> {
>>>>>
>>>>>           public void map(Object key, Text value,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>
>>>>>               String line = value.toString();
>>>>>
>>>>>               Tweet aTweet = new Tweet(line);
>>>>>               if(aTweet.getMention()!=null &&
>>>>> aTweet.getMention().length()>0){
>>>>>                   /*aTweet.setRT(0);
>>>>>                   output.collect(new Text(aTweet.toString()), new
>>>>> Text(aTweet.toString()));*/
>>>>>                   aTweet.setRT(1);
>>>>>                   try {
>>>>>                       context.write(new Text(aTweet.getMention()), new
>>>>> Text(aTweet.toString()));
>>>>>                   } catch (InterruptedException e) {
>>>>>                       // TODO Auto-generated catch block
>>>>>                       e.printStackTrace();
>>>>>                   }
>>>>>               }
>>>>>               aTweet.setRT(0);
>>>>>               try {
>>>>>                   context.write(new Text(aTweet.getAuthor()), new
>>>>> Text(aTweet.toString()));
>>>>>               } catch (InterruptedException e) {
>>>>>                   // TODO Auto-generated catch block
>>>>>                   e.printStackTrace();
>>>>>               }
>>>>>
>>>>>           }
>>>>>       }
>>>>>
>>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>>> Text> {
>>>>>
>>>>>       public void reduce(Text atweet, Iterator<Text> tweets,
>>>>>               Context context)
>>>>>               throws IOException {
>>>>>           List<Tweet> originals = new ArrayList<Tweet>();
>>>>>           List<Tweet> retweets = new ArrayList<Tweet>();
>>>>>           while(tweets.hasNext()){
>>>>>               Tweet aTweet = new Tweet(tweets.toString());
>>>>>               originals.add(aTweet);
>>>>>               if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>>           }
>>>>>
>>>>>           for(Tweet originalTweet : originals){
>>>>>               for(Tweet aRetweet : retweets){
>>>>>                   if(aRetweet.isRetweeted(originalTweet))
>>>>>                   {
>>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>>                       try {
>>>>>                           context.write(
>>>>>                                   atweet,
>>>>>                                   new Text(aRetweet.toString()));
>>>>>                       } catch (InterruptedException e) {
>>>>>                           // TODO Auto-generated catch block
>>>>>                           e.printStackTrace();
>>>>>                       }
>>>>>                   }
>>>>>               }
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>>     public static void main(String[] args) throws Exception {
>>>>>       Configuration conf = new Configuration();
>>>>>       String[] otherArgs = new GenericOptionsParser(conf,
>>>>> args).getRemainingArgs();
>>>>>       if (otherArgs.length != 2) {
>>>>>         System.err.println("Usage: wordcount <in> <out>");
>>>>>         System.exit(2);
>>>>>       }
>>>>>       Job job = new Job(conf, "tweets");
>>>>>       job.setJarByClass(RetweetApplication.class);
>>>>>       job.setMapperClass(RetweetMapper1.class);
>>>>>       job.setReducerClass(RetweetReducer1.class);
>>>>>       job.setOutputKeyClass(Text.class);
>>>>>       job.setOutputValueClass(Text.class);
>>>>>       FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>>       FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>>       System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>>     }
>>>>> }
>>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>> Hi all,
>>>>>>
>>>>>> i got some more information:
>>>>>>
>>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>>> is a
>>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>>> And it has nothing to do with the local configuration of the server,
>>>>>> since
>>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>>> differs
>>>>>> from one job to another.
>>>>>> 2) the same happens even when i am using a completely different set of
>>>>>> servers.
>>>>>>
>>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>>
>>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>> Hi,
>>>>>>>
>>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>>> recently marked here:
>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>>
>>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>>> many
>>>>>>> different setup parameters and JVM options and nothing did the
>>>>>>> trick...
>>>>>>>
>>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>>> NOT
>>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>>
>>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay2 times
>>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>>> cliID:
>>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> injecting
>>>>>>> delay1 times
>>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>>> org.mortbay.jetty.EofException
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>>       at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>       at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>>       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>>       at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>>       at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>>       at
>>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>>       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>>       ... 27 more
>>>>>>>
>>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>>> 141.51.205.130:35497,
>>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>       at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>       at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>       at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>>
>>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>>> the
>>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>>> join
>>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>>
>>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>>> to
>>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>>> look
>>>>>>>> into the tasktracker's logs saying:
>>>>>>>>
>>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>>       at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>>       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>>       at
>>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>>       at
>>>>>>>>
>>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>>
>>>>>>>>
>>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>>> as the
>>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>>> that
>>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>>> problem by
>>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>>> processed
>>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>>> in the
>>>>>>>> shell tells me it is 400000000).
>>>>>>>>
>>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>>> pure
>>>>>>>> naive math it should be:
>>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>>> datastructures
>>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>>> machines
>>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>>> as much
>>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>>> just
>>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>>
>>>>>>>> Can anybody help me with this issue?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Elmar
>>>>>>>>
>>>>>>>>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Now that I think about it I wonder about oracle's description.

[...] if more than 98% of the total time is spent in garbage collection [...]

I took that to mean 98% of CPU time for the application, but it can
also mean that the GC is active for 98% percent of wall clock time.

If it is the former it implies the application isn't doing useful
stuff anymore (e.g., keeping a connection open). If it is the latter I
don't know if it possible to say anything about how much the program
is still doing useful stuff.



On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Hi Vasco,
>
> thank you for your help!
>
> I can try to add the limit again (i currently have it turned off for all
> Java processes spawned by Hadoop). Also i do not have any persistent
> (member-) variables, that could store things that use alot of data: at the
> very moment i only got 2 local variables in the reduce-method, that could
> get a little larger - but those should be gc'ed easily, shouldnt they?
>
> Thank you for your thoughts again. I will pay with the GC a little. If i
> know the reason, i'll let you know!
> Best regards,
> Elmar
>
>
> Am 05.09.2012 14:38, schrieb Vasco Visser:
>
>> Just a guess, but could it simply be memory issue on reducer?
>>
>> In your adjusted program, maybe try running without
>> -UseGCOverheadLimit and see if you still got OOM errors.
>>
>>  From the sun website:
>> The parallel collector will throw an OutOfMemoryError if too much time
>> is being spent in garbage collection: if more than 98% of the total
>> time is spent in garbage collection and less than 2% of the heap is
>> recovered, an OutOfMemoryError will be thrown. This feature is
>> designed to prevent applications from running for an extended period
>> of time while making little or no progress because the heap is too
>> small. If necessary, this feature can be disabled by adding the option
>> -XX:-UseGCOverheadLimit to the command line.
>>
>> So basically if you get a GC overhead limit exceeded OOM error it
>> means that your app is doing nothing but garbage collection, the vm is
>> fighting against running out of memory. If that happens it might
>> result in a timeout of the connection fetching the map data (i don't
>> know if it can result in timeout, but I can imagine it could).
>>
>> Also, note that 63 GB on disk is probably going to be inflated in
>> memory. So in general you cant say that 60 GB on disk will need 60GB
>> of mem. Actually, some people us a rule of thumb to do x4 to get
>> approx mem requirement.
>>
>> Just some ideas, not really a solution but maybe it helps you further.
>>
>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>> <ma...@cs.uni-kassel.de> wrote:
>>>
>>> Excuse me: in my last code section was some old code included. Here is it
>>> again stripped of deprecated code:
>>>
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>> Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text>
>>> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "word count");
>>>
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>
>>>> Hello again,
>>>>
>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>> interested in the mortbay-issue:
>>>> I applied the wordcount example to a big input file and everything
>>>> worked
>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>> exception
>>>> occurred. The only possible explanation for me seemed to be, that my
>>>> code
>>>> somehow produces the exceptions. So i started to keep my code as close
>>>> to
>>>> the wordcount example as possible:
>>>>
>>>> * first getting rid of any members within the reducer class, that may
>>>> take
>>>> lots of memory.
>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>> Reducer)
>>>> * removing any Comparator (and restructuring the code so it just
>>>> calculates the same as with the sorting)
>>>> * using standart HashPartitioner
>>>>
>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>> the
>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>> seems to complete all the work while the others are busy with their
>>>> exceptions (at least from what i can see in the log).
>>>>
>>>> I REALLY dont know what i can do, to make this job run through like the
>>>> wordcount example. I dont see any big difference between my program and
>>>> the
>>>> example anymore.
>>>>
>>>> I added the code here:
>>>>
>>>> *Please* help me here.
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>> I added the code here:
>>>> ########################################
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>> import java.util.Date;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.conf.Configured;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>> import org.apache.hadoop.mapred.JobClient;
>>>> import org.apache.hadoop.mapred.JobConf;
>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>> import org.apache.hadoop.util.Tool;
>>>> import org.apache.hadoop.util.ToolRunner;
>>>>
>>>> public class RetweetApplication extends Configured implements Tool{
>>>>
>>>>      public static int DEBUG = -1;
>>>>      private Configuration jobConf;
>>>>
>>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>> Date()).toString();
>>>>
>>>>      public static void main(String[] args) {
>>>>           int res=-1;
>>>>          try {
>>>>              res = ToolRunner.run(new Configuration(), new
>>>> RetweetApplication(), args);
>>>>          } catch (Exception e) {
>>>>              // TODO Auto-generated catch block
>>>>              e.printStackTrace();
>>>>          }
>>>>          System.exit(res);
>>>>
>>>>      }
>>>>
>>>>      public void setConf(Configuration conf) {
>>>>          jobConf = conf;
>>>>
>>>>      }
>>>>
>>>>      public Configuration getConf() {
>>>>          return jobConf;
>>>>      }
>>>>
>>>>      public int run(String[] args) throws Exception {
>>>>          JobClient client = new JobClient();
>>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>>
>>>>
>>>>          if(DEBUG > 0){
>>>>              conf.set("mapred.job.tracker", "local");
>>>>              conf.set("fs.default.name", "file:///");
>>>>              conf.set("dfs.replication", "1");
>>>>          }
>>>>
>>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>          conf.set("mapred.child.ulimit","100000000");
>>>>
>>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>
>>>>
>>>>          //conf.setOutputKeyClass(Text.class);
>>>>          //conf.setOutputValueClass(Text.class);
>>>>          conf.setMapOutputKeyClass(Text.class);
>>>>          conf.setMapOutputValueClass(Text.class);
>>>>
>>>>          conf.setMapperClass(RetweetMapper.class);
>>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>>>
>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>
>>>>
>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>          conf.setReducerClass(RetweetReducer.class);
>>>>
>>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>>
>>>>          client.setConf(conf);
>>>>          try {
>>>>              JobClient.runJob(conf);
>>>>          } catch (Exception e) {
>>>>              e.printStackTrace();
>>>>          }
>>>>          return 0;
>>>>      }
>>>> }
>>>> */
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>          public void map(Object key, Text value,
>>>>              Context context)
>>>>              throws IOException {
>>>>
>>>>              String line = value.toString();
>>>>
>>>>              Tweet aTweet = new Tweet(line);
>>>>              if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                  /*aTweet.setRT(0);
>>>>                  output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                  aTweet.setRT(1);
>>>>                  try {
>>>>                      context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                  } catch (InterruptedException e) {
>>>>                      // TODO Auto-generated catch block
>>>>                      e.printStackTrace();
>>>>                  }
>>>>              }
>>>>              aTweet.setRT(0);
>>>>              try {
>>>>                  context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>              } catch (InterruptedException e) {
>>>>                  // TODO Auto-generated catch block
>>>>                  e.printStackTrace();
>>>>              }
>>>>
>>>>          }
>>>>      }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text> {
>>>>
>>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>>              Context context)
>>>>              throws IOException {
>>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>>          while(tweets.hasNext()){
>>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>>              originals.add(aTweet);
>>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>          }
>>>>
>>>>          for(Tweet originalTweet : originals){
>>>>              for(Tweet aRetweet : retweets){
>>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>>                  {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                      try {
>>>>                          context.write(
>>>>                                  atweet,
>>>>                                  new Text(aRetweet.toString()));
>>>>                      } catch (InterruptedException e) {
>>>>                          // TODO Auto-generated catch block
>>>>                          e.printStackTrace();
>>>>                      }
>>>>                  }
>>>>              }
>>>>          }
>>>>      }
>>>> }
>>>>
>>>>    public static void main(String[] args) throws Exception {
>>>>      Configuration conf = new Configuration();
>>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>      if (otherArgs.length != 2) {
>>>>        System.err.println("Usage: wordcount <in> <out>");
>>>>        System.exit(2);
>>>>      }
>>>>      Job job = new Job(conf, "tweets");
>>>>      job.setJarByClass(RetweetApplication.class);
>>>>      job.setMapperClass(RetweetMapper1.class);
>>>>      job.setReducerClass(RetweetReducer1.class);
>>>>      job.setOutputKeyClass(Text.class);
>>>>      job.setOutputValueClass(Text.class);
>>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>    }
>>>> }
>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> i got some more information:
>>>>>
>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>> is a
>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>> And it has nothing to do with the local configuration of the server,
>>>>> since
>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>> differs
>>>>> from one job to another.
>>>>> 2) the same happens even when i am using a completely different set of
>>>>> servers.
>>>>>
>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>
>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>> recently marked here:
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>
>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>> many
>>>>>> different setup parameters and JVM options and nothing did the
>>>>>> trick...
>>>>>>
>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>> NOT
>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>
>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay2 times
>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay1 times
>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>
>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>> the
>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>> join
>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>
>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>> to
>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>> look
>>>>>>> into the tasktracker's logs saying:
>>>>>>>
>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>>
>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>> as the
>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>> that
>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>> problem by
>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>> processed
>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>> in the
>>>>>>> shell tells me it is 400000000).
>>>>>>>
>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>> pure
>>>>>>> naive math it should be:
>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>> datastructures
>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>> machines
>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>> as much
>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>> just
>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>
>>>>>>> Can anybody help me with this issue?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Elmar
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Now that I think about it I wonder about oracle's description.

[...] if more than 98% of the total time is spent in garbage collection [...]

I took that to mean 98% of CPU time for the application, but it can
also mean that the GC is active for 98% percent of wall clock time.

If it is the former it implies the application isn't doing useful
stuff anymore (e.g., keeping a connection open). If it is the latter I
don't know if it possible to say anything about how much the program
is still doing useful stuff.



On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Hi Vasco,
>
> thank you for your help!
>
> I can try to add the limit again (i currently have it turned off for all
> Java processes spawned by Hadoop). Also i do not have any persistent
> (member-) variables, that could store things that use alot of data: at the
> very moment i only got 2 local variables in the reduce-method, that could
> get a little larger - but those should be gc'ed easily, shouldnt they?
>
> Thank you for your thoughts again. I will pay with the GC a little. If i
> know the reason, i'll let you know!
> Best regards,
> Elmar
>
>
> Am 05.09.2012 14:38, schrieb Vasco Visser:
>
>> Just a guess, but could it simply be memory issue on reducer?
>>
>> In your adjusted program, maybe try running without
>> -UseGCOverheadLimit and see if you still got OOM errors.
>>
>>  From the sun website:
>> The parallel collector will throw an OutOfMemoryError if too much time
>> is being spent in garbage collection: if more than 98% of the total
>> time is spent in garbage collection and less than 2% of the heap is
>> recovered, an OutOfMemoryError will be thrown. This feature is
>> designed to prevent applications from running for an extended period
>> of time while making little or no progress because the heap is too
>> small. If necessary, this feature can be disabled by adding the option
>> -XX:-UseGCOverheadLimit to the command line.
>>
>> So basically if you get a GC overhead limit exceeded OOM error it
>> means that your app is doing nothing but garbage collection, the vm is
>> fighting against running out of memory. If that happens it might
>> result in a timeout of the connection fetching the map data (i don't
>> know if it can result in timeout, but I can imagine it could).
>>
>> Also, note that 63 GB on disk is probably going to be inflated in
>> memory. So in general you cant say that 60 GB on disk will need 60GB
>> of mem. Actually, some people us a rule of thumb to do x4 to get
>> approx mem requirement.
>>
>> Just some ideas, not really a solution but maybe it helps you further.
>>
>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>> <ma...@cs.uni-kassel.de> wrote:
>>>
>>> Excuse me: in my last code section was some old code included. Here is it
>>> again stripped of deprecated code:
>>>
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>> Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text>
>>> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "word count");
>>>
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>
>>>> Hello again,
>>>>
>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>> interested in the mortbay-issue:
>>>> I applied the wordcount example to a big input file and everything
>>>> worked
>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>> exception
>>>> occurred. The only possible explanation for me seemed to be, that my
>>>> code
>>>> somehow produces the exceptions. So i started to keep my code as close
>>>> to
>>>> the wordcount example as possible:
>>>>
>>>> * first getting rid of any members within the reducer class, that may
>>>> take
>>>> lots of memory.
>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>> Reducer)
>>>> * removing any Comparator (and restructuring the code so it just
>>>> calculates the same as with the sorting)
>>>> * using standart HashPartitioner
>>>>
>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>> the
>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>> seems to complete all the work while the others are busy with their
>>>> exceptions (at least from what i can see in the log).
>>>>
>>>> I REALLY dont know what i can do, to make this job run through like the
>>>> wordcount example. I dont see any big difference between my program and
>>>> the
>>>> example anymore.
>>>>
>>>> I added the code here:
>>>>
>>>> *Please* help me here.
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>> I added the code here:
>>>> ########################################
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>> import java.util.Date;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.conf.Configured;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>> import org.apache.hadoop.mapred.JobClient;
>>>> import org.apache.hadoop.mapred.JobConf;
>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>> import org.apache.hadoop.util.Tool;
>>>> import org.apache.hadoop.util.ToolRunner;
>>>>
>>>> public class RetweetApplication extends Configured implements Tool{
>>>>
>>>>      public static int DEBUG = -1;
>>>>      private Configuration jobConf;
>>>>
>>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>> Date()).toString();
>>>>
>>>>      public static void main(String[] args) {
>>>>           int res=-1;
>>>>          try {
>>>>              res = ToolRunner.run(new Configuration(), new
>>>> RetweetApplication(), args);
>>>>          } catch (Exception e) {
>>>>              // TODO Auto-generated catch block
>>>>              e.printStackTrace();
>>>>          }
>>>>          System.exit(res);
>>>>
>>>>      }
>>>>
>>>>      public void setConf(Configuration conf) {
>>>>          jobConf = conf;
>>>>
>>>>      }
>>>>
>>>>      public Configuration getConf() {
>>>>          return jobConf;
>>>>      }
>>>>
>>>>      public int run(String[] args) throws Exception {
>>>>          JobClient client = new JobClient();
>>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>>
>>>>
>>>>          if(DEBUG > 0){
>>>>              conf.set("mapred.job.tracker", "local");
>>>>              conf.set("fs.default.name", "file:///");
>>>>              conf.set("dfs.replication", "1");
>>>>          }
>>>>
>>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>          conf.set("mapred.child.ulimit","100000000");
>>>>
>>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>
>>>>
>>>>          //conf.setOutputKeyClass(Text.class);
>>>>          //conf.setOutputValueClass(Text.class);
>>>>          conf.setMapOutputKeyClass(Text.class);
>>>>          conf.setMapOutputValueClass(Text.class);
>>>>
>>>>          conf.setMapperClass(RetweetMapper.class);
>>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>>>
>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>
>>>>
>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>          conf.setReducerClass(RetweetReducer.class);
>>>>
>>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>>
>>>>          client.setConf(conf);
>>>>          try {
>>>>              JobClient.runJob(conf);
>>>>          } catch (Exception e) {
>>>>              e.printStackTrace();
>>>>          }
>>>>          return 0;
>>>>      }
>>>> }
>>>> */
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>          public void map(Object key, Text value,
>>>>              Context context)
>>>>              throws IOException {
>>>>
>>>>              String line = value.toString();
>>>>
>>>>              Tweet aTweet = new Tweet(line);
>>>>              if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                  /*aTweet.setRT(0);
>>>>                  output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                  aTweet.setRT(1);
>>>>                  try {
>>>>                      context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                  } catch (InterruptedException e) {
>>>>                      // TODO Auto-generated catch block
>>>>                      e.printStackTrace();
>>>>                  }
>>>>              }
>>>>              aTweet.setRT(0);
>>>>              try {
>>>>                  context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>              } catch (InterruptedException e) {
>>>>                  // TODO Auto-generated catch block
>>>>                  e.printStackTrace();
>>>>              }
>>>>
>>>>          }
>>>>      }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text> {
>>>>
>>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>>              Context context)
>>>>              throws IOException {
>>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>>          while(tweets.hasNext()){
>>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>>              originals.add(aTweet);
>>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>          }
>>>>
>>>>          for(Tweet originalTweet : originals){
>>>>              for(Tweet aRetweet : retweets){
>>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>>                  {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                      try {
>>>>                          context.write(
>>>>                                  atweet,
>>>>                                  new Text(aRetweet.toString()));
>>>>                      } catch (InterruptedException e) {
>>>>                          // TODO Auto-generated catch block
>>>>                          e.printStackTrace();
>>>>                      }
>>>>                  }
>>>>              }
>>>>          }
>>>>      }
>>>> }
>>>>
>>>>    public static void main(String[] args) throws Exception {
>>>>      Configuration conf = new Configuration();
>>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>      if (otherArgs.length != 2) {
>>>>        System.err.println("Usage: wordcount <in> <out>");
>>>>        System.exit(2);
>>>>      }
>>>>      Job job = new Job(conf, "tweets");
>>>>      job.setJarByClass(RetweetApplication.class);
>>>>      job.setMapperClass(RetweetMapper1.class);
>>>>      job.setReducerClass(RetweetReducer1.class);
>>>>      job.setOutputKeyClass(Text.class);
>>>>      job.setOutputValueClass(Text.class);
>>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>    }
>>>> }
>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> i got some more information:
>>>>>
>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>> is a
>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>> And it has nothing to do with the local configuration of the server,
>>>>> since
>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>> differs
>>>>> from one job to another.
>>>>> 2) the same happens even when i am using a completely different set of
>>>>> servers.
>>>>>
>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>
>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>> recently marked here:
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>
>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>> many
>>>>>> different setup parameters and JVM options and nothing did the
>>>>>> trick...
>>>>>>
>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>> NOT
>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>
>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay2 times
>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay1 times
>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>
>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>> the
>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>> join
>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>
>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>> to
>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>> look
>>>>>>> into the tasktracker's logs saying:
>>>>>>>
>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>>
>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>> as the
>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>> that
>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>> problem by
>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>> processed
>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>> in the
>>>>>>> shell tells me it is 400000000).
>>>>>>>
>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>> pure
>>>>>>> naive math it should be:
>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>> datastructures
>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>> machines
>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>> as much
>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>> just
>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>
>>>>>>> Can anybody help me with this issue?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Elmar
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Now that I think about it I wonder about oracle's description.

[...] if more than 98% of the total time is spent in garbage collection [...]

I took that to mean 98% of CPU time for the application, but it can
also mean that the GC is active for 98% percent of wall clock time.

If it is the former it implies the application isn't doing useful
stuff anymore (e.g., keeping a connection open). If it is the latter I
don't know if it possible to say anything about how much the program
is still doing useful stuff.



On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Hi Vasco,
>
> thank you for your help!
>
> I can try to add the limit again (i currently have it turned off for all
> Java processes spawned by Hadoop). Also i do not have any persistent
> (member-) variables, that could store things that use alot of data: at the
> very moment i only got 2 local variables in the reduce-method, that could
> get a little larger - but those should be gc'ed easily, shouldnt they?
>
> Thank you for your thoughts again. I will pay with the GC a little. If i
> know the reason, i'll let you know!
> Best regards,
> Elmar
>
>
> Am 05.09.2012 14:38, schrieb Vasco Visser:
>
>> Just a guess, but could it simply be memory issue on reducer?
>>
>> In your adjusted program, maybe try running without
>> -UseGCOverheadLimit and see if you still got OOM errors.
>>
>>  From the sun website:
>> The parallel collector will throw an OutOfMemoryError if too much time
>> is being spent in garbage collection: if more than 98% of the total
>> time is spent in garbage collection and less than 2% of the heap is
>> recovered, an OutOfMemoryError will be thrown. This feature is
>> designed to prevent applications from running for an extended period
>> of time while making little or no progress because the heap is too
>> small. If necessary, this feature can be disabled by adding the option
>> -XX:-UseGCOverheadLimit to the command line.
>>
>> So basically if you get a GC overhead limit exceeded OOM error it
>> means that your app is doing nothing but garbage collection, the vm is
>> fighting against running out of memory. If that happens it might
>> result in a timeout of the connection fetching the map data (i don't
>> know if it can result in timeout, but I can imagine it could).
>>
>> Also, note that 63 GB on disk is probably going to be inflated in
>> memory. So in general you cant say that 60 GB on disk will need 60GB
>> of mem. Actually, some people us a rule of thumb to do x4 to get
>> approx mem requirement.
>>
>> Just some ideas, not really a solution but maybe it helps you further.
>>
>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>> <ma...@cs.uni-kassel.de> wrote:
>>>
>>> Excuse me: in my last code section was some old code included. Here is it
>>> again stripped of deprecated code:
>>>
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>> Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text>
>>> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "word count");
>>>
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>
>>>> Hello again,
>>>>
>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>> interested in the mortbay-issue:
>>>> I applied the wordcount example to a big input file and everything
>>>> worked
>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>> exception
>>>> occurred. The only possible explanation for me seemed to be, that my
>>>> code
>>>> somehow produces the exceptions. So i started to keep my code as close
>>>> to
>>>> the wordcount example as possible:
>>>>
>>>> * first getting rid of any members within the reducer class, that may
>>>> take
>>>> lots of memory.
>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>> Reducer)
>>>> * removing any Comparator (and restructuring the code so it just
>>>> calculates the same as with the sorting)
>>>> * using standart HashPartitioner
>>>>
>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>> the
>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>> seems to complete all the work while the others are busy with their
>>>> exceptions (at least from what i can see in the log).
>>>>
>>>> I REALLY dont know what i can do, to make this job run through like the
>>>> wordcount example. I dont see any big difference between my program and
>>>> the
>>>> example anymore.
>>>>
>>>> I added the code here:
>>>>
>>>> *Please* help me here.
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>> I added the code here:
>>>> ########################################
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>> import java.util.Date;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.conf.Configured;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>> import org.apache.hadoop.mapred.JobClient;
>>>> import org.apache.hadoop.mapred.JobConf;
>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>> import org.apache.hadoop.util.Tool;
>>>> import org.apache.hadoop.util.ToolRunner;
>>>>
>>>> public class RetweetApplication extends Configured implements Tool{
>>>>
>>>>      public static int DEBUG = -1;
>>>>      private Configuration jobConf;
>>>>
>>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>> Date()).toString();
>>>>
>>>>      public static void main(String[] args) {
>>>>           int res=-1;
>>>>          try {
>>>>              res = ToolRunner.run(new Configuration(), new
>>>> RetweetApplication(), args);
>>>>          } catch (Exception e) {
>>>>              // TODO Auto-generated catch block
>>>>              e.printStackTrace();
>>>>          }
>>>>          System.exit(res);
>>>>
>>>>      }
>>>>
>>>>      public void setConf(Configuration conf) {
>>>>          jobConf = conf;
>>>>
>>>>      }
>>>>
>>>>      public Configuration getConf() {
>>>>          return jobConf;
>>>>      }
>>>>
>>>>      public int run(String[] args) throws Exception {
>>>>          JobClient client = new JobClient();
>>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>>
>>>>
>>>>          if(DEBUG > 0){
>>>>              conf.set("mapred.job.tracker", "local");
>>>>              conf.set("fs.default.name", "file:///");
>>>>              conf.set("dfs.replication", "1");
>>>>          }
>>>>
>>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>          conf.set("mapred.child.ulimit","100000000");
>>>>
>>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>
>>>>
>>>>          //conf.setOutputKeyClass(Text.class);
>>>>          //conf.setOutputValueClass(Text.class);
>>>>          conf.setMapOutputKeyClass(Text.class);
>>>>          conf.setMapOutputValueClass(Text.class);
>>>>
>>>>          conf.setMapperClass(RetweetMapper.class);
>>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>>>
>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>
>>>>
>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>          conf.setReducerClass(RetweetReducer.class);
>>>>
>>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>>
>>>>          client.setConf(conf);
>>>>          try {
>>>>              JobClient.runJob(conf);
>>>>          } catch (Exception e) {
>>>>              e.printStackTrace();
>>>>          }
>>>>          return 0;
>>>>      }
>>>> }
>>>> */
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>          public void map(Object key, Text value,
>>>>              Context context)
>>>>              throws IOException {
>>>>
>>>>              String line = value.toString();
>>>>
>>>>              Tweet aTweet = new Tweet(line);
>>>>              if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                  /*aTweet.setRT(0);
>>>>                  output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                  aTweet.setRT(1);
>>>>                  try {
>>>>                      context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                  } catch (InterruptedException e) {
>>>>                      // TODO Auto-generated catch block
>>>>                      e.printStackTrace();
>>>>                  }
>>>>              }
>>>>              aTweet.setRT(0);
>>>>              try {
>>>>                  context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>              } catch (InterruptedException e) {
>>>>                  // TODO Auto-generated catch block
>>>>                  e.printStackTrace();
>>>>              }
>>>>
>>>>          }
>>>>      }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text> {
>>>>
>>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>>              Context context)
>>>>              throws IOException {
>>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>>          while(tweets.hasNext()){
>>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>>              originals.add(aTweet);
>>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>          }
>>>>
>>>>          for(Tweet originalTweet : originals){
>>>>              for(Tweet aRetweet : retweets){
>>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>>                  {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                      try {
>>>>                          context.write(
>>>>                                  atweet,
>>>>                                  new Text(aRetweet.toString()));
>>>>                      } catch (InterruptedException e) {
>>>>                          // TODO Auto-generated catch block
>>>>                          e.printStackTrace();
>>>>                      }
>>>>                  }
>>>>              }
>>>>          }
>>>>      }
>>>> }
>>>>
>>>>    public static void main(String[] args) throws Exception {
>>>>      Configuration conf = new Configuration();
>>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>      if (otherArgs.length != 2) {
>>>>        System.err.println("Usage: wordcount <in> <out>");
>>>>        System.exit(2);
>>>>      }
>>>>      Job job = new Job(conf, "tweets");
>>>>      job.setJarByClass(RetweetApplication.class);
>>>>      job.setMapperClass(RetweetMapper1.class);
>>>>      job.setReducerClass(RetweetReducer1.class);
>>>>      job.setOutputKeyClass(Text.class);
>>>>      job.setOutputValueClass(Text.class);
>>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>    }
>>>> }
>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> i got some more information:
>>>>>
>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>> is a
>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>> And it has nothing to do with the local configuration of the server,
>>>>> since
>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>> differs
>>>>> from one job to another.
>>>>> 2) the same happens even when i am using a completely different set of
>>>>> servers.
>>>>>
>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>
>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>> recently marked here:
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>
>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>> many
>>>>>> different setup parameters and JVM options and nothing did the
>>>>>> trick...
>>>>>>
>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>> NOT
>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>
>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay2 times
>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay1 times
>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>
>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>> the
>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>> join
>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>
>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>> to
>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>> look
>>>>>>> into the tasktracker's logs saying:
>>>>>>>
>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>>
>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>> as the
>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>> that
>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>> problem by
>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>> processed
>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>> in the
>>>>>>> shell tells me it is 400000000).
>>>>>>>
>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>> pure
>>>>>>> naive math it should be:
>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>> datastructures
>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>> machines
>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>> as much
>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>> just
>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>
>>>>>>> Can anybody help me with this issue?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Elmar
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Now that I think about it I wonder about oracle's description.

[...] if more than 98% of the total time is spent in garbage collection [...]

I took that to mean 98% of CPU time for the application, but it can
also mean that the GC is active for 98% percent of wall clock time.

If it is the former it implies the application isn't doing useful
stuff anymore (e.g., keeping a connection open). If it is the latter I
don't know if it possible to say anything about how much the program
is still doing useful stuff.



On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Hi Vasco,
>
> thank you for your help!
>
> I can try to add the limit again (i currently have it turned off for all
> Java processes spawned by Hadoop). Also i do not have any persistent
> (member-) variables, that could store things that use alot of data: at the
> very moment i only got 2 local variables in the reduce-method, that could
> get a little larger - but those should be gc'ed easily, shouldnt they?
>
> Thank you for your thoughts again. I will pay with the GC a little. If i
> know the reason, i'll let you know!
> Best regards,
> Elmar
>
>
> Am 05.09.2012 14:38, schrieb Vasco Visser:
>
>> Just a guess, but could it simply be memory issue on reducer?
>>
>> In your adjusted program, maybe try running without
>> -UseGCOverheadLimit and see if you still got OOM errors.
>>
>>  From the sun website:
>> The parallel collector will throw an OutOfMemoryError if too much time
>> is being spent in garbage collection: if more than 98% of the total
>> time is spent in garbage collection and less than 2% of the heap is
>> recovered, an OutOfMemoryError will be thrown. This feature is
>> designed to prevent applications from running for an extended period
>> of time while making little or no progress because the heap is too
>> small. If necessary, this feature can be disabled by adding the option
>> -XX:-UseGCOverheadLimit to the command line.
>>
>> So basically if you get a GC overhead limit exceeded OOM error it
>> means that your app is doing nothing but garbage collection, the vm is
>> fighting against running out of memory. If that happens it might
>> result in a timeout of the connection fetching the map data (i don't
>> know if it can result in timeout, but I can imagine it could).
>>
>> Also, note that 63 GB on disk is probably going to be inflated in
>> memory. So in general you cant say that 60 GB on disk will need 60GB
>> of mem. Actually, some people us a rule of thumb to do x4 to get
>> approx mem requirement.
>>
>> Just some ideas, not really a solution but maybe it helps you further.
>>
>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>> <ma...@cs.uni-kassel.de> wrote:
>>>
>>> Excuse me: in my last code section was some old code included. Here is it
>>> again stripped of deprecated code:
>>>
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>> Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text>
>>> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "word count");
>>>
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>>
>>>> Hello again,
>>>>
>>>> i just wanted to keep you updated, in case anybody reads this and is
>>>> interested in the mortbay-issue:
>>>> I applied the wordcount example to a big input file and everything
>>>> worked
>>>> fine. I watched the TT logs all the time and no single mortbay-eof
>>>> exception
>>>> occurred. The only possible explanation for me seemed to be, that my
>>>> code
>>>> somehow produces the exceptions. So i started to keep my code as close
>>>> to
>>>> the wordcount example as possible:
>>>>
>>>> * first getting rid of any members within the reducer class, that may
>>>> take
>>>> lots of memory.
>>>> * using the Context class instead of OutputCollector (in Mapper and
>>>> Reducer)
>>>> * removing any Comparator (and restructuring the code so it just
>>>> calculates the same as with the sorting)
>>>> * using standart HashPartitioner
>>>>
>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH
>>>> the
>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>>> seems to complete all the work while the others are busy with their
>>>> exceptions (at least from what i can see in the log).
>>>>
>>>> I REALLY dont know what i can do, to make this job run through like the
>>>> wordcount example. I dont see any big difference between my program and
>>>> the
>>>> example anymore.
>>>>
>>>> I added the code here:
>>>>
>>>> *Please* help me here.
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>> I added the code here:
>>>> ########################################
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>> import java.util.Date;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.conf.Configured;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.FileInputFormat;
>>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>>> import org.apache.hadoop.mapred.JobClient;
>>>> import org.apache.hadoop.mapred.JobConf;
>>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>>> import org.apache.hadoop.util.Tool;
>>>> import org.apache.hadoop.util.ToolRunner;
>>>>
>>>> public class RetweetApplication extends Configured implements Tool{
>>>>
>>>>      public static int DEBUG = -1;
>>>>      private Configuration jobConf;
>>>>
>>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>>> Date()).toString();
>>>>
>>>>      public static void main(String[] args) {
>>>>           int res=-1;
>>>>          try {
>>>>              res = ToolRunner.run(new Configuration(), new
>>>> RetweetApplication(), args);
>>>>          } catch (Exception e) {
>>>>              // TODO Auto-generated catch block
>>>>              e.printStackTrace();
>>>>          }
>>>>          System.exit(res);
>>>>
>>>>      }
>>>>
>>>>      public void setConf(Configuration conf) {
>>>>          jobConf = conf;
>>>>
>>>>      }
>>>>
>>>>      public Configuration getConf() {
>>>>          return jobConf;
>>>>      }
>>>>
>>>>      public int run(String[] args) throws Exception {
>>>>          JobClient client = new JobClient();
>>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>>
>>>>
>>>>          if(DEBUG > 0){
>>>>              conf.set("mapred.job.tracker", "local");
>>>>              conf.set("fs.default.name", "file:///");
>>>>              conf.set("dfs.replication", "1");
>>>>          }
>>>>
>>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>>          conf.set("mapred.child.ulimit","100000000");
>>>>
>>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>>
>>>>
>>>>          //conf.setOutputKeyClass(Text.class);
>>>>          //conf.setOutputValueClass(Text.class);
>>>>          conf.setMapOutputKeyClass(Text.class);
>>>>          conf.setMapOutputValueClass(Text.class);
>>>>
>>>>          conf.setMapperClass(RetweetMapper.class);
>>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>>>
>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>>
>>>>
>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>>          conf.setReducerClass(RetweetReducer.class);
>>>>
>>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>>
>>>>          client.setConf(conf);
>>>>          try {
>>>>              JobClient.runJob(conf);
>>>>          } catch (Exception e) {
>>>>              e.printStackTrace();
>>>>          }
>>>>          return 0;
>>>>      }
>>>> }
>>>> */
>>>>
>>>> package uni.kassel.macek.rtprep;
>>>>
>>>>
>>>>
>>>> import gnu.trove.iterator.TIntIterator;
>>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>>> import gnu.trove.set.TIntSet;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Calendar;
>>>> import java.util.Iterator;
>>>> import java.util.List;
>>>> import java.util.StringTokenizer;
>>>>
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapred.MapReduceBase;
>>>> import org.apache.hadoop.mapred.OutputCollector;
>>>> import org.apache.hadoop.mapred.Reporter;
>>>> import org.apache.hadoop.mapreduce.Job;
>>>> import org.apache.hadoop.mapreduce.Mapper;
>>>> import org.apache.hadoop.mapreduce.Reducer;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>
>>>> public class RetweetApplication {
>>>>
>>>>      public static class RetweetMapper1 extends Mapper<Object, Text,
>>>> Text,
>>>> Text> {
>>>>
>>>>          public void map(Object key, Text value,
>>>>              Context context)
>>>>              throws IOException {
>>>>
>>>>              String line = value.toString();
>>>>
>>>>              Tweet aTweet = new Tweet(line);
>>>>              if(aTweet.getMention()!=null &&
>>>> aTweet.getMention().length()>0){
>>>>                  /*aTweet.setRT(0);
>>>>                  output.collect(new Text(aTweet.toString()), new
>>>> Text(aTweet.toString()));*/
>>>>                  aTweet.setRT(1);
>>>>                  try {
>>>>                      context.write(new Text(aTweet.getMention()), new
>>>> Text(aTweet.toString()));
>>>>                  } catch (InterruptedException e) {
>>>>                      // TODO Auto-generated catch block
>>>>                      e.printStackTrace();
>>>>                  }
>>>>              }
>>>>              aTweet.setRT(0);
>>>>              try {
>>>>                  context.write(new Text(aTweet.getAuthor()), new
>>>> Text(aTweet.toString()));
>>>>              } catch (InterruptedException e) {
>>>>                  // TODO Auto-generated catch block
>>>>                  e.printStackTrace();
>>>>              }
>>>>
>>>>          }
>>>>      }
>>>>
>>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>>> Text> {
>>>>
>>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>>              Context context)
>>>>              throws IOException {
>>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>>          while(tweets.hasNext()){
>>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>>              originals.add(aTweet);
>>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>>          }
>>>>
>>>>          for(Tweet originalTweet : originals){
>>>>              for(Tweet aRetweet : retweets){
>>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>>                  {
>>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>>                      try {
>>>>                          context.write(
>>>>                                  atweet,
>>>>                                  new Text(aRetweet.toString()));
>>>>                      } catch (InterruptedException e) {
>>>>                          // TODO Auto-generated catch block
>>>>                          e.printStackTrace();
>>>>                      }
>>>>                  }
>>>>              }
>>>>          }
>>>>      }
>>>> }
>>>>
>>>>    public static void main(String[] args) throws Exception {
>>>>      Configuration conf = new Configuration();
>>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>>> args).getRemainingArgs();
>>>>      if (otherArgs.length != 2) {
>>>>        System.err.println("Usage: wordcount <in> <out>");
>>>>        System.exit(2);
>>>>      }
>>>>      Job job = new Job(conf, "tweets");
>>>>      job.setJarByClass(RetweetApplication.class);
>>>>      job.setMapperClass(RetweetMapper1.class);
>>>>      job.setReducerClass(RetweetReducer1.class);
>>>>      job.setOutputKeyClass(Text.class);
>>>>      job.setOutputValueClass(Text.class);
>>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>>    }
>>>> }
>>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> i got some more information:
>>>>>
>>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>>> slaves fail relatively fast after the reduce process started: maybe it
>>>>> is a
>>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>>> And it has nothing to do with the local configuration of the server,
>>>>> since
>>>>> 1) if i rerun the job on the same set of servers, the ONE working
>>>>> differs
>>>>> from one job to another.
>>>>> 2) the same happens even when i am using a completely different set of
>>>>> servers.
>>>>>
>>>>> Who is capable to solve this unsolved mystery? :D
>>>>>
>>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> is there anybody, who knows more about this issue: it has already been
>>>>>> recently marked here:
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>>
>>>>>> I really want to do something about it, if i knew how... i tried so
>>>>>> many
>>>>>> different setup parameters and JVM options and nothing did the
>>>>>> trick...
>>>>>>
>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does
>>>>>> NOT
>>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>>
>>>>>> 2012-08-30 12:05:22,582 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay2 times
>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 2 times, canceled keys 70 times
>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:08:39,236 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:10:27,109 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:14:29,740 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> 2012-08-30 12:16:28,503 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE,
>>>>>> cliID:
>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> injecting
>>>>>> delay1 times
>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>>> recreating selector 1 times, canceled keys 69 times
>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>>> org.mortbay.jetty.EofException
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>>      at
>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>>      ... 27 more
>>>>>>
>>>>>> 2012-08-30 12:20:01,205 INFO
>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>>>>> 141.51.205.130:35497,
>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>>
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>>
>>>>>>> While the mapping works just fine and is low cost, the reducer does
>>>>>>> the
>>>>>>> main work: holding a hashmap with elements to join with and finding
>>>>>>> join
>>>>>>> tuples for evry incoming key-value-pair.
>>>>>>>
>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts
>>>>>>> to
>>>>>>> get "unstable" as the file size goes up: this becomes evident with a
>>>>>>> look
>>>>>>> into the tasktracker's logs saying:
>>>>>>>
>>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>>> java.lang.IllegalStateException: Committed
>>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>>      at
>>>>>>>
>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>>      at
>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>>      at
>>>>>>>
>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>>
>>>>>>>
>>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial
>>>>>>> as the
>>>>>>> progress rises. The reason for this (afaik from reading articles), is
>>>>>>> that
>>>>>>> there are memory or file handle problems. I addressed the memory
>>>>>>> problem by
>>>>>>> conitiously purging the map of outdated elements evry 5 million
>>>>>>> processed
>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit
>>>>>>> in the
>>>>>>> shell tells me it is 400000000).
>>>>>>>
>>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By
>>>>>>> pure
>>>>>>> naive math it should be:
>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large
>>>>>>> datastructures
>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave
>>>>>>> machines
>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times
>>>>>>> as much
>>>>>>> as needed... i would think. So if the Partitioner distributes them
>>>>>>> just
>>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>>
>>>>>>> Can anybody help me with this issue?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Elmar
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

thank you for your help!

I can try to add the limit again (i currently have it turned off for all 
Java processes spawned by Hadoop). Also i do not have any persistent 
(member-) variables, that could store things that use alot of data: at 
the very moment i only got 2 local variables in the reduce-method, that 
could get a little larger - but those should be gc'ed easily, shouldnt they?

Thank you for your thoughts again. I will pay with the GC a little. If i 
know the reason, i'll let you know!
Best regards,
Elmar


Am 05.09.2012 14:38, schrieb Vasco Visser:
> Just a guess, but could it simply be memory issue on reducer?
>
> In your adjusted program, maybe try running without
> -UseGCOverheadLimit and see if you still got OOM errors.
>
>  From the sun website:
> The parallel collector will throw an OutOfMemoryError if too much time
> is being spent in garbage collection: if more than 98% of the total
> time is spent in garbage collection and less than 2% of the heap is
> recovered, an OutOfMemoryError will be thrown. This feature is
> designed to prevent applications from running for an extended period
> of time while making little or no progress because the heap is too
> small. If necessary, this feature can be disabled by adding the option
> -XX:-UseGCOverheadLimit to the command line.
>
> So basically if you get a GC overhead limit exceeded OOM error it
> means that your app is doing nothing but garbage collection, the vm is
> fighting against running out of memory. If that happens it might
> result in a timeout of the connection fetching the map data (i don't
> know if it can result in timeout, but I can imagine it could).
>
> Also, note that 63 GB on disk is probably going to be inflated in
> memory. So in general you cant say that 60 GB on disk will need 60GB
> of mem. Actually, some people us a rule of thumb to do x4 to get
> approx mem requirement.
>
> Just some ideas, not really a solution but maybe it helps you further.
>
> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Excuse me: in my last code section was some old code included. Here is it
>> again stripped of deprecated code:
>>
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>          public void map(Object key, Text value,
>>              Context context)
>>              throws IOException {
>>
>>              String line = value.toString();
>>
>>              Tweet aTweet = new Tweet(line);
>>              if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>>                  /*aTweet.setRT(0);
>>                  output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                  aTweet.setRT(1);
>>                  try {
>>                      context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                  } catch (InterruptedException e) {
>>                      // TODO Auto-generated catch block
>>                      e.printStackTrace();
>>                  }
>>              }
>>              aTweet.setRT(0);
>>              try {
>>                  context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>              } catch (InterruptedException e) {
>>                  // TODO Auto-generated catch block
>>                  e.printStackTrace();
>>              }
>>
>>          }
>>      }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
>> {
>>
>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>              Context context)
>>              throws IOException {
>>          List<Tweet> originals = new ArrayList<Tweet>();
>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>          while(tweets.hasNext()){
>>              Tweet aTweet = new Tweet(tweets.toString());
>>              originals.add(aTweet);
>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>          }
>>
>>          for(Tweet originalTweet : originals){
>>              for(Tweet aRetweet : retweets){
>>                  if(aRetweet.isRetweeted(originalTweet))
>>                  {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                      try {
>>                          context.write(
>>                                  atweet,
>>                                  new Text(aRetweet.toString()));
>>                      } catch (InterruptedException e) {
>>                          // TODO Auto-generated catch block
>>                          e.printStackTrace();
>>                      }
>>                  }
>>              }
>>          }
>>      }
>> }
>>
>>    public static void main(String[] args) throws Exception {
>>      Configuration conf = new Configuration();
>>      String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>      if (otherArgs.length != 2) {
>>        System.err.println("Usage: wordcount <in> <out>");
>>        System.exit(2);
>>      }
>>      Job job = new Job(conf, "word count");
>>
>>      job.setJarByClass(RetweetApplication.class);
>>      job.setMapperClass(RetweetMapper1.class);
>>      job.setReducerClass(RetweetReducer1.class);
>>      job.setOutputKeyClass(Text.class);
>>      job.setOutputValueClass(Text.class);
>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>    }
>> }
>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>
>>> Hello again,
>>>
>>> i just wanted to keep you updated, in case anybody reads this and is
>>> interested in the mortbay-issue:
>>> I applied the wordcount example to a big input file and everything worked
>>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>>> occurred. The only possible explanation for me seemed to be, that my code
>>> somehow produces the exceptions. So i started to keep my code as close to
>>> the wordcount example as possible:
>>>
>>> * first getting rid of any members within the reducer class, that may take
>>> lots of memory.
>>> * using the Context class instead of OutputCollector (in Mapper and
>>> Reducer)
>>> * removing any Comparator (and restructuring the code so it just
>>> calculates the same as with the sorting)
>>> * using standart HashPartitioner
>>>
>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>> seems to complete all the work while the others are busy with their
>>> exceptions (at least from what i can see in the log).
>>>
>>> I REALLY dont know what i can do, to make this job run through like the
>>> wordcount example. I dont see any big difference between my program and the
>>> example anymore.
>>>
>>> I added the code here:
>>>
>>> *Please* help me here.
>>>
>>> Best regards,
>>> Elmar
>>>
>>>
>>> I added the code here:
>>> ########################################
>>> package uni.kassel.macek.rtprep;
>>>
>>> import java.util.Date;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.FileInputFormat;
>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>> import org.apache.hadoop.mapred.JobClient;
>>> import org.apache.hadoop.mapred.JobConf;
>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>> import org.apache.hadoop.util.Tool;
>>> import org.apache.hadoop.util.ToolRunner;
>>>
>>> public class RetweetApplication extends Configured implements Tool{
>>>
>>>      public static int DEBUG = -1;
>>>      private Configuration jobConf;
>>>
>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>> Date()).toString();
>>>
>>>      public static void main(String[] args) {
>>>           int res=-1;
>>>          try {
>>>              res = ToolRunner.run(new Configuration(), new
>>> RetweetApplication(), args);
>>>          } catch (Exception e) {
>>>              // TODO Auto-generated catch block
>>>              e.printStackTrace();
>>>          }
>>>          System.exit(res);
>>>
>>>      }
>>>
>>>      public void setConf(Configuration conf) {
>>>          jobConf = conf;
>>>
>>>      }
>>>
>>>      public Configuration getConf() {
>>>          return jobConf;
>>>      }
>>>
>>>      public int run(String[] args) throws Exception {
>>>          JobClient client = new JobClient();
>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>
>>>
>>>          if(DEBUG > 0){
>>>              conf.set("mapred.job.tracker", "local");
>>>              conf.set("fs.default.name", "file:///");
>>>              conf.set("dfs.replication", "1");
>>>          }
>>>
>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>          conf.set("mapred.child.ulimit","100000000");
>>>
>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>
>>>
>>>          //conf.setOutputKeyClass(Text.class);
>>>          //conf.setOutputValueClass(Text.class);
>>>          conf.setMapOutputKeyClass(Text.class);
>>>          conf.setMapOutputValueClass(Text.class);
>>>
>>>          conf.setMapperClass(RetweetMapper.class);
>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>
>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>          conf.setReducerClass(RetweetReducer.class);
>>>
>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>
>>>          client.setConf(conf);
>>>          try {
>>>              JobClient.runJob(conf);
>>>          } catch (Exception e) {
>>>              e.printStackTrace();
>>>          }
>>>          return 0;
>>>      }
>>> }
>>> */
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "tweets");
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>> Hi all,
>>>>
>>>> i got some more information:
>>>>
>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>> And it has nothing to do with the local configuration of the server,
>>>> since
>>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>>> from one job to another.
>>>> 2) the same happens even when i am using a completely different set of
>>>> servers.
>>>>
>>>> Who is capable to solve this unsolved mystery? :D
>>>>
>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>> Hi,
>>>>>
>>>>> is there anybody, who knows more about this issue: it has already been
>>>>> recently marked here:
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>
>>>>> I really want to do something about it, if i knew how... i tried so many
>>>>> different setup parameters and JVM options and nothing did the trick...
>>>>>
>>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>
>>>>> 2012-08-30 12:05:22,582 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay2 times
>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 2 times, canceled keys 70 times
>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:10:27,109 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:16:28,503 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay1 times
>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 1 times, canceled keys 69 times
>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>> Hi there,
>>>>>>
>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>
>>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>>> tuples for evry incoming key-value-pair.
>>>>>>
>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>>> into the tasktracker's logs saying:
>>>>>>
>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>>
>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>>> shell tells me it is 400000000).
>>>>>>
>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>>> naive math it should be:
>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>
>>>>>> Can anybody help me with this issue?
>>>>>>
>>>>>> Best regards,
>>>>>> Elmar
>>>>>>
>>>>>>
>>>>>
>>>>
>>>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

thank you for your help!

I can try to add the limit again (i currently have it turned off for all 
Java processes spawned by Hadoop). Also i do not have any persistent 
(member-) variables, that could store things that use alot of data: at 
the very moment i only got 2 local variables in the reduce-method, that 
could get a little larger - but those should be gc'ed easily, shouldnt they?

Thank you for your thoughts again. I will pay with the GC a little. If i 
know the reason, i'll let you know!
Best regards,
Elmar


Am 05.09.2012 14:38, schrieb Vasco Visser:
> Just a guess, but could it simply be memory issue on reducer?
>
> In your adjusted program, maybe try running without
> -UseGCOverheadLimit and see if you still got OOM errors.
>
>  From the sun website:
> The parallel collector will throw an OutOfMemoryError if too much time
> is being spent in garbage collection: if more than 98% of the total
> time is spent in garbage collection and less than 2% of the heap is
> recovered, an OutOfMemoryError will be thrown. This feature is
> designed to prevent applications from running for an extended period
> of time while making little or no progress because the heap is too
> small. If necessary, this feature can be disabled by adding the option
> -XX:-UseGCOverheadLimit to the command line.
>
> So basically if you get a GC overhead limit exceeded OOM error it
> means that your app is doing nothing but garbage collection, the vm is
> fighting against running out of memory. If that happens it might
> result in a timeout of the connection fetching the map data (i don't
> know if it can result in timeout, but I can imagine it could).
>
> Also, note that 63 GB on disk is probably going to be inflated in
> memory. So in general you cant say that 60 GB on disk will need 60GB
> of mem. Actually, some people us a rule of thumb to do x4 to get
> approx mem requirement.
>
> Just some ideas, not really a solution but maybe it helps you further.
>
> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Excuse me: in my last code section was some old code included. Here is it
>> again stripped of deprecated code:
>>
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>          public void map(Object key, Text value,
>>              Context context)
>>              throws IOException {
>>
>>              String line = value.toString();
>>
>>              Tweet aTweet = new Tweet(line);
>>              if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>>                  /*aTweet.setRT(0);
>>                  output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                  aTweet.setRT(1);
>>                  try {
>>                      context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                  } catch (InterruptedException e) {
>>                      // TODO Auto-generated catch block
>>                      e.printStackTrace();
>>                  }
>>              }
>>              aTweet.setRT(0);
>>              try {
>>                  context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>              } catch (InterruptedException e) {
>>                  // TODO Auto-generated catch block
>>                  e.printStackTrace();
>>              }
>>
>>          }
>>      }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
>> {
>>
>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>              Context context)
>>              throws IOException {
>>          List<Tweet> originals = new ArrayList<Tweet>();
>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>          while(tweets.hasNext()){
>>              Tweet aTweet = new Tweet(tweets.toString());
>>              originals.add(aTweet);
>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>          }
>>
>>          for(Tweet originalTweet : originals){
>>              for(Tweet aRetweet : retweets){
>>                  if(aRetweet.isRetweeted(originalTweet))
>>                  {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                      try {
>>                          context.write(
>>                                  atweet,
>>                                  new Text(aRetweet.toString()));
>>                      } catch (InterruptedException e) {
>>                          // TODO Auto-generated catch block
>>                          e.printStackTrace();
>>                      }
>>                  }
>>              }
>>          }
>>      }
>> }
>>
>>    public static void main(String[] args) throws Exception {
>>      Configuration conf = new Configuration();
>>      String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>      if (otherArgs.length != 2) {
>>        System.err.println("Usage: wordcount <in> <out>");
>>        System.exit(2);
>>      }
>>      Job job = new Job(conf, "word count");
>>
>>      job.setJarByClass(RetweetApplication.class);
>>      job.setMapperClass(RetweetMapper1.class);
>>      job.setReducerClass(RetweetReducer1.class);
>>      job.setOutputKeyClass(Text.class);
>>      job.setOutputValueClass(Text.class);
>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>    }
>> }
>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>
>>> Hello again,
>>>
>>> i just wanted to keep you updated, in case anybody reads this and is
>>> interested in the mortbay-issue:
>>> I applied the wordcount example to a big input file and everything worked
>>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>>> occurred. The only possible explanation for me seemed to be, that my code
>>> somehow produces the exceptions. So i started to keep my code as close to
>>> the wordcount example as possible:
>>>
>>> * first getting rid of any members within the reducer class, that may take
>>> lots of memory.
>>> * using the Context class instead of OutputCollector (in Mapper and
>>> Reducer)
>>> * removing any Comparator (and restructuring the code so it just
>>> calculates the same as with the sorting)
>>> * using standart HashPartitioner
>>>
>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>> seems to complete all the work while the others are busy with their
>>> exceptions (at least from what i can see in the log).
>>>
>>> I REALLY dont know what i can do, to make this job run through like the
>>> wordcount example. I dont see any big difference between my program and the
>>> example anymore.
>>>
>>> I added the code here:
>>>
>>> *Please* help me here.
>>>
>>> Best regards,
>>> Elmar
>>>
>>>
>>> I added the code here:
>>> ########################################
>>> package uni.kassel.macek.rtprep;
>>>
>>> import java.util.Date;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.FileInputFormat;
>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>> import org.apache.hadoop.mapred.JobClient;
>>> import org.apache.hadoop.mapred.JobConf;
>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>> import org.apache.hadoop.util.Tool;
>>> import org.apache.hadoop.util.ToolRunner;
>>>
>>> public class RetweetApplication extends Configured implements Tool{
>>>
>>>      public static int DEBUG = -1;
>>>      private Configuration jobConf;
>>>
>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>> Date()).toString();
>>>
>>>      public static void main(String[] args) {
>>>           int res=-1;
>>>          try {
>>>              res = ToolRunner.run(new Configuration(), new
>>> RetweetApplication(), args);
>>>          } catch (Exception e) {
>>>              // TODO Auto-generated catch block
>>>              e.printStackTrace();
>>>          }
>>>          System.exit(res);
>>>
>>>      }
>>>
>>>      public void setConf(Configuration conf) {
>>>          jobConf = conf;
>>>
>>>      }
>>>
>>>      public Configuration getConf() {
>>>          return jobConf;
>>>      }
>>>
>>>      public int run(String[] args) throws Exception {
>>>          JobClient client = new JobClient();
>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>
>>>
>>>          if(DEBUG > 0){
>>>              conf.set("mapred.job.tracker", "local");
>>>              conf.set("fs.default.name", "file:///");
>>>              conf.set("dfs.replication", "1");
>>>          }
>>>
>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>          conf.set("mapred.child.ulimit","100000000");
>>>
>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>
>>>
>>>          //conf.setOutputKeyClass(Text.class);
>>>          //conf.setOutputValueClass(Text.class);
>>>          conf.setMapOutputKeyClass(Text.class);
>>>          conf.setMapOutputValueClass(Text.class);
>>>
>>>          conf.setMapperClass(RetweetMapper.class);
>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>
>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>          conf.setReducerClass(RetweetReducer.class);
>>>
>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>
>>>          client.setConf(conf);
>>>          try {
>>>              JobClient.runJob(conf);
>>>          } catch (Exception e) {
>>>              e.printStackTrace();
>>>          }
>>>          return 0;
>>>      }
>>> }
>>> */
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "tweets");
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>> Hi all,
>>>>
>>>> i got some more information:
>>>>
>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>> And it has nothing to do with the local configuration of the server,
>>>> since
>>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>>> from one job to another.
>>>> 2) the same happens even when i am using a completely different set of
>>>> servers.
>>>>
>>>> Who is capable to solve this unsolved mystery? :D
>>>>
>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>> Hi,
>>>>>
>>>>> is there anybody, who knows more about this issue: it has already been
>>>>> recently marked here:
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>
>>>>> I really want to do something about it, if i knew how... i tried so many
>>>>> different setup parameters and JVM options and nothing did the trick...
>>>>>
>>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>
>>>>> 2012-08-30 12:05:22,582 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay2 times
>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 2 times, canceled keys 70 times
>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:10:27,109 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:16:28,503 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay1 times
>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 1 times, canceled keys 69 times
>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>> Hi there,
>>>>>>
>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>
>>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>>> tuples for evry incoming key-value-pair.
>>>>>>
>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>>> into the tasktracker's logs saying:
>>>>>>
>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>>
>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>>> shell tells me it is 400000000).
>>>>>>
>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>>> naive math it should be:
>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>
>>>>>> Can anybody help me with this issue?
>>>>>>
>>>>>> Best regards,
>>>>>> Elmar
>>>>>>
>>>>>>
>>>>>
>>>>
>>>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

thank you for your help!

I can try to add the limit again (i currently have it turned off for all 
Java processes spawned by Hadoop). Also i do not have any persistent 
(member-) variables, that could store things that use alot of data: at 
the very moment i only got 2 local variables in the reduce-method, that 
could get a little larger - but those should be gc'ed easily, shouldnt they?

Thank you for your thoughts again. I will pay with the GC a little. If i 
know the reason, i'll let you know!
Best regards,
Elmar


Am 05.09.2012 14:38, schrieb Vasco Visser:
> Just a guess, but could it simply be memory issue on reducer?
>
> In your adjusted program, maybe try running without
> -UseGCOverheadLimit and see if you still got OOM errors.
>
>  From the sun website:
> The parallel collector will throw an OutOfMemoryError if too much time
> is being spent in garbage collection: if more than 98% of the total
> time is spent in garbage collection and less than 2% of the heap is
> recovered, an OutOfMemoryError will be thrown. This feature is
> designed to prevent applications from running for an extended period
> of time while making little or no progress because the heap is too
> small. If necessary, this feature can be disabled by adding the option
> -XX:-UseGCOverheadLimit to the command line.
>
> So basically if you get a GC overhead limit exceeded OOM error it
> means that your app is doing nothing but garbage collection, the vm is
> fighting against running out of memory. If that happens it might
> result in a timeout of the connection fetching the map data (i don't
> know if it can result in timeout, but I can imagine it could).
>
> Also, note that 63 GB on disk is probably going to be inflated in
> memory. So in general you cant say that 60 GB on disk will need 60GB
> of mem. Actually, some people us a rule of thumb to do x4 to get
> approx mem requirement.
>
> Just some ideas, not really a solution but maybe it helps you further.
>
> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Excuse me: in my last code section was some old code included. Here is it
>> again stripped of deprecated code:
>>
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>          public void map(Object key, Text value,
>>              Context context)
>>              throws IOException {
>>
>>              String line = value.toString();
>>
>>              Tweet aTweet = new Tweet(line);
>>              if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>>                  /*aTweet.setRT(0);
>>                  output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                  aTweet.setRT(1);
>>                  try {
>>                      context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                  } catch (InterruptedException e) {
>>                      // TODO Auto-generated catch block
>>                      e.printStackTrace();
>>                  }
>>              }
>>              aTweet.setRT(0);
>>              try {
>>                  context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>              } catch (InterruptedException e) {
>>                  // TODO Auto-generated catch block
>>                  e.printStackTrace();
>>              }
>>
>>          }
>>      }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
>> {
>>
>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>              Context context)
>>              throws IOException {
>>          List<Tweet> originals = new ArrayList<Tweet>();
>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>          while(tweets.hasNext()){
>>              Tweet aTweet = new Tweet(tweets.toString());
>>              originals.add(aTweet);
>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>          }
>>
>>          for(Tweet originalTweet : originals){
>>              for(Tweet aRetweet : retweets){
>>                  if(aRetweet.isRetweeted(originalTweet))
>>                  {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                      try {
>>                          context.write(
>>                                  atweet,
>>                                  new Text(aRetweet.toString()));
>>                      } catch (InterruptedException e) {
>>                          // TODO Auto-generated catch block
>>                          e.printStackTrace();
>>                      }
>>                  }
>>              }
>>          }
>>      }
>> }
>>
>>    public static void main(String[] args) throws Exception {
>>      Configuration conf = new Configuration();
>>      String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>      if (otherArgs.length != 2) {
>>        System.err.println("Usage: wordcount <in> <out>");
>>        System.exit(2);
>>      }
>>      Job job = new Job(conf, "word count");
>>
>>      job.setJarByClass(RetweetApplication.class);
>>      job.setMapperClass(RetweetMapper1.class);
>>      job.setReducerClass(RetweetReducer1.class);
>>      job.setOutputKeyClass(Text.class);
>>      job.setOutputValueClass(Text.class);
>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>    }
>> }
>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>
>>> Hello again,
>>>
>>> i just wanted to keep you updated, in case anybody reads this and is
>>> interested in the mortbay-issue:
>>> I applied the wordcount example to a big input file and everything worked
>>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>>> occurred. The only possible explanation for me seemed to be, that my code
>>> somehow produces the exceptions. So i started to keep my code as close to
>>> the wordcount example as possible:
>>>
>>> * first getting rid of any members within the reducer class, that may take
>>> lots of memory.
>>> * using the Context class instead of OutputCollector (in Mapper and
>>> Reducer)
>>> * removing any Comparator (and restructuring the code so it just
>>> calculates the same as with the sorting)
>>> * using standart HashPartitioner
>>>
>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>> seems to complete all the work while the others are busy with their
>>> exceptions (at least from what i can see in the log).
>>>
>>> I REALLY dont know what i can do, to make this job run through like the
>>> wordcount example. I dont see any big difference between my program and the
>>> example anymore.
>>>
>>> I added the code here:
>>>
>>> *Please* help me here.
>>>
>>> Best regards,
>>> Elmar
>>>
>>>
>>> I added the code here:
>>> ########################################
>>> package uni.kassel.macek.rtprep;
>>>
>>> import java.util.Date;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.FileInputFormat;
>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>> import org.apache.hadoop.mapred.JobClient;
>>> import org.apache.hadoop.mapred.JobConf;
>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>> import org.apache.hadoop.util.Tool;
>>> import org.apache.hadoop.util.ToolRunner;
>>>
>>> public class RetweetApplication extends Configured implements Tool{
>>>
>>>      public static int DEBUG = -1;
>>>      private Configuration jobConf;
>>>
>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>> Date()).toString();
>>>
>>>      public static void main(String[] args) {
>>>           int res=-1;
>>>          try {
>>>              res = ToolRunner.run(new Configuration(), new
>>> RetweetApplication(), args);
>>>          } catch (Exception e) {
>>>              // TODO Auto-generated catch block
>>>              e.printStackTrace();
>>>          }
>>>          System.exit(res);
>>>
>>>      }
>>>
>>>      public void setConf(Configuration conf) {
>>>          jobConf = conf;
>>>
>>>      }
>>>
>>>      public Configuration getConf() {
>>>          return jobConf;
>>>      }
>>>
>>>      public int run(String[] args) throws Exception {
>>>          JobClient client = new JobClient();
>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>
>>>
>>>          if(DEBUG > 0){
>>>              conf.set("mapred.job.tracker", "local");
>>>              conf.set("fs.default.name", "file:///");
>>>              conf.set("dfs.replication", "1");
>>>          }
>>>
>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>          conf.set("mapred.child.ulimit","100000000");
>>>
>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>
>>>
>>>          //conf.setOutputKeyClass(Text.class);
>>>          //conf.setOutputValueClass(Text.class);
>>>          conf.setMapOutputKeyClass(Text.class);
>>>          conf.setMapOutputValueClass(Text.class);
>>>
>>>          conf.setMapperClass(RetweetMapper.class);
>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>
>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>          conf.setReducerClass(RetweetReducer.class);
>>>
>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>
>>>          client.setConf(conf);
>>>          try {
>>>              JobClient.runJob(conf);
>>>          } catch (Exception e) {
>>>              e.printStackTrace();
>>>          }
>>>          return 0;
>>>      }
>>> }
>>> */
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "tweets");
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>> Hi all,
>>>>
>>>> i got some more information:
>>>>
>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>> And it has nothing to do with the local configuration of the server,
>>>> since
>>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>>> from one job to another.
>>>> 2) the same happens even when i am using a completely different set of
>>>> servers.
>>>>
>>>> Who is capable to solve this unsolved mystery? :D
>>>>
>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>> Hi,
>>>>>
>>>>> is there anybody, who knows more about this issue: it has already been
>>>>> recently marked here:
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>
>>>>> I really want to do something about it, if i knew how... i tried so many
>>>>> different setup parameters and JVM options and nothing did the trick...
>>>>>
>>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>
>>>>> 2012-08-30 12:05:22,582 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay2 times
>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 2 times, canceled keys 70 times
>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:10:27,109 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:16:28,503 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay1 times
>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 1 times, canceled keys 69 times
>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>> Hi there,
>>>>>>
>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>
>>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>>> tuples for evry incoming key-value-pair.
>>>>>>
>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>>> into the tasktracker's logs saying:
>>>>>>
>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>>
>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>>> shell tells me it is 400000000).
>>>>>>
>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>>> naive math it should be:
>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>
>>>>>> Can anybody help me with this issue?
>>>>>>
>>>>>> Best regards,
>>>>>> Elmar
>>>>>>
>>>>>>
>>>>>
>>>>
>>>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Hi Vasco,

thank you for your help!

I can try to add the limit again (i currently have it turned off for all 
Java processes spawned by Hadoop). Also i do not have any persistent 
(member-) variables, that could store things that use alot of data: at 
the very moment i only got 2 local variables in the reduce-method, that 
could get a little larger - but those should be gc'ed easily, shouldnt they?

Thank you for your thoughts again. I will pay with the GC a little. If i 
know the reason, i'll let you know!
Best regards,
Elmar


Am 05.09.2012 14:38, schrieb Vasco Visser:
> Just a guess, but could it simply be memory issue on reducer?
>
> In your adjusted program, maybe try running without
> -UseGCOverheadLimit and see if you still got OOM errors.
>
>  From the sun website:
> The parallel collector will throw an OutOfMemoryError if too much time
> is being spent in garbage collection: if more than 98% of the total
> time is spent in garbage collection and less than 2% of the heap is
> recovered, an OutOfMemoryError will be thrown. This feature is
> designed to prevent applications from running for an extended period
> of time while making little or no progress because the heap is too
> small. If necessary, this feature can be disabled by adding the option
> -XX:-UseGCOverheadLimit to the command line.
>
> So basically if you get a GC overhead limit exceeded OOM error it
> means that your app is doing nothing but garbage collection, the vm is
> fighting against running out of memory. If that happens it might
> result in a timeout of the connection fetching the map data (i don't
> know if it can result in timeout, but I can imagine it could).
>
> Also, note that 63 GB on disk is probably going to be inflated in
> memory. So in general you cant say that 60 GB on disk will need 60GB
> of mem. Actually, some people us a rule of thumb to do x4 to get
> approx mem requirement.
>
> Just some ideas, not really a solution but maybe it helps you further.
>
> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
> <ma...@cs.uni-kassel.de> wrote:
>> Excuse me: in my last code section was some old code included. Here is it
>> again stripped of deprecated code:
>>
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>          public void map(Object key, Text value,
>>              Context context)
>>              throws IOException {
>>
>>              String line = value.toString();
>>
>>              Tweet aTweet = new Tweet(line);
>>              if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>>                  /*aTweet.setRT(0);
>>                  output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                  aTweet.setRT(1);
>>                  try {
>>                      context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                  } catch (InterruptedException e) {
>>                      // TODO Auto-generated catch block
>>                      e.printStackTrace();
>>                  }
>>              }
>>              aTweet.setRT(0);
>>              try {
>>                  context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>              } catch (InterruptedException e) {
>>                  // TODO Auto-generated catch block
>>                  e.printStackTrace();
>>              }
>>
>>          }
>>      }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
>> {
>>
>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>              Context context)
>>              throws IOException {
>>          List<Tweet> originals = new ArrayList<Tweet>();
>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>          while(tweets.hasNext()){
>>              Tweet aTweet = new Tweet(tweets.toString());
>>              originals.add(aTweet);
>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>          }
>>
>>          for(Tweet originalTweet : originals){
>>              for(Tweet aRetweet : retweets){
>>                  if(aRetweet.isRetweeted(originalTweet))
>>                  {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                      try {
>>                          context.write(
>>                                  atweet,
>>                                  new Text(aRetweet.toString()));
>>                      } catch (InterruptedException e) {
>>                          // TODO Auto-generated catch block
>>                          e.printStackTrace();
>>                      }
>>                  }
>>              }
>>          }
>>      }
>> }
>>
>>    public static void main(String[] args) throws Exception {
>>      Configuration conf = new Configuration();
>>      String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>      if (otherArgs.length != 2) {
>>        System.err.println("Usage: wordcount <in> <out>");
>>        System.exit(2);
>>      }
>>      Job job = new Job(conf, "word count");
>>
>>      job.setJarByClass(RetweetApplication.class);
>>      job.setMapperClass(RetweetMapper1.class);
>>      job.setReducerClass(RetweetReducer1.class);
>>      job.setOutputKeyClass(Text.class);
>>      job.setOutputValueClass(Text.class);
>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>    }
>> }
>> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>>
>>> Hello again,
>>>
>>> i just wanted to keep you updated, in case anybody reads this and is
>>> interested in the mortbay-issue:
>>> I applied the wordcount example to a big input file and everything worked
>>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>>> occurred. The only possible explanation for me seemed to be, that my code
>>> somehow produces the exceptions. So i started to keep my code as close to
>>> the wordcount example as possible:
>>>
>>> * first getting rid of any members within the reducer class, that may take
>>> lots of memory.
>>> * using the Context class instead of OutputCollector (in Mapper and
>>> Reducer)
>>> * removing any Comparator (and restructuring the code so it just
>>> calculates the same as with the sorting)
>>> * using standart HashPartitioner
>>>
>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>>> seems to complete all the work while the others are busy with their
>>> exceptions (at least from what i can see in the log).
>>>
>>> I REALLY dont know what i can do, to make this job run through like the
>>> wordcount example. I dont see any big difference between my program and the
>>> example anymore.
>>>
>>> I added the code here:
>>>
>>> *Please* help me here.
>>>
>>> Best regards,
>>> Elmar
>>>
>>>
>>> I added the code here:
>>> ########################################
>>> package uni.kassel.macek.rtprep;
>>>
>>> import java.util.Date;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.FileInputFormat;
>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>> import org.apache.hadoop.mapred.JobClient;
>>> import org.apache.hadoop.mapred.JobConf;
>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>> import org.apache.hadoop.util.Tool;
>>> import org.apache.hadoop.util.ToolRunner;
>>>
>>> public class RetweetApplication extends Configured implements Tool{
>>>
>>>      public static int DEBUG = -1;
>>>      private Configuration jobConf;
>>>
>>>      //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>>      //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>>> Date()).toString();
>>>
>>>      public static void main(String[] args) {
>>>           int res=-1;
>>>          try {
>>>              res = ToolRunner.run(new Configuration(), new
>>> RetweetApplication(), args);
>>>          } catch (Exception e) {
>>>              // TODO Auto-generated catch block
>>>              e.printStackTrace();
>>>          }
>>>          System.exit(res);
>>>
>>>      }
>>>
>>>      public void setConf(Configuration conf) {
>>>          jobConf = conf;
>>>
>>>      }
>>>
>>>      public Configuration getConf() {
>>>          return jobConf;
>>>      }
>>>
>>>      public int run(String[] args) throws Exception {
>>>          JobClient client = new JobClient();
>>>          JobConf conf = new JobConf(RetweetApplication.class);
>>>
>>>
>>>          if(DEBUG > 0){
>>>              conf.set("mapred.job.tracker", "local");
>>>              conf.set("fs.default.name", "file:///");
>>>              conf.set("dfs.replication", "1");
>>>          }
>>>
>>>          //conf.set("mapred.child.java.opts","-Xmx3000m
>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>>          conf.set("mapred.child.ulimit","100000000");
>>>
>>>          FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>          FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>
>>>
>>>          //conf.setOutputKeyClass(Text.class);
>>>          //conf.setOutputValueClass(Text.class);
>>>          conf.setMapOutputKeyClass(Text.class);
>>>          conf.setMapOutputValueClass(Text.class);
>>>
>>>          conf.setMapperClass(RetweetMapper.class);
>>>          //conf.setPartitionerClass(TweetPartitioner.class);
>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>>
>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>>          conf.setReducerClass(RetweetReducer.class);
>>>
>>>          conf.setOutputFormat(TextOutputFormat.class);
>>>
>>>          client.setConf(conf);
>>>          try {
>>>              JobClient.runJob(conf);
>>>          } catch (Exception e) {
>>>              e.printStackTrace();
>>>          }
>>>          return 0;
>>>      }
>>> }
>>> */
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>>> Text> {
>>>
>>>          public void map(Object key, Text value,
>>>              Context context)
>>>              throws IOException {
>>>
>>>              String line = value.toString();
>>>
>>>              Tweet aTweet = new Tweet(line);
>>>              if(aTweet.getMention()!=null &&
>>> aTweet.getMention().length()>0){
>>>                  /*aTweet.setRT(0);
>>>                  output.collect(new Text(aTweet.toString()), new
>>> Text(aTweet.toString()));*/
>>>                  aTweet.setRT(1);
>>>                  try {
>>>                      context.write(new Text(aTweet.getMention()), new
>>> Text(aTweet.toString()));
>>>                  } catch (InterruptedException e) {
>>>                      // TODO Auto-generated catch block
>>>                      e.printStackTrace();
>>>                  }
>>>              }
>>>              aTweet.setRT(0);
>>>              try {
>>>                  context.write(new Text(aTweet.getAuthor()), new
>>> Text(aTweet.toString()));
>>>              } catch (InterruptedException e) {
>>>                  // TODO Auto-generated catch block
>>>                  e.printStackTrace();
>>>              }
>>>
>>>          }
>>>      }
>>>
>>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>>> Text> {
>>>
>>>      public void reduce(Text atweet, Iterator<Text> tweets,
>>>              Context context)
>>>              throws IOException {
>>>          List<Tweet> originals = new ArrayList<Tweet>();
>>>          List<Tweet> retweets = new ArrayList<Tweet>();
>>>          while(tweets.hasNext()){
>>>              Tweet aTweet = new Tweet(tweets.toString());
>>>              originals.add(aTweet);
>>>              if(aTweet.getRT()>0) retweets.add(aTweet);
>>>          }
>>>
>>>          for(Tweet originalTweet : originals){
>>>              for(Tweet aRetweet : retweets){
>>>                  if(aRetweet.isRetweeted(originalTweet))
>>>                  {
>>> aRetweet.setReactionTime(originalTweet.getDate());
>>>                      try {
>>>                          context.write(
>>>                                  atweet,
>>>                                  new Text(aRetweet.toString()));
>>>                      } catch (InterruptedException e) {
>>>                          // TODO Auto-generated catch block
>>>                          e.printStackTrace();
>>>                      }
>>>                  }
>>>              }
>>>          }
>>>      }
>>> }
>>>
>>>    public static void main(String[] args) throws Exception {
>>>      Configuration conf = new Configuration();
>>>      String[] otherArgs = new GenericOptionsParser(conf,
>>> args).getRemainingArgs();
>>>      if (otherArgs.length != 2) {
>>>        System.err.println("Usage: wordcount <in> <out>");
>>>        System.exit(2);
>>>      }
>>>      Job job = new Job(conf, "tweets");
>>>      job.setJarByClass(RetweetApplication.class);
>>>      job.setMapperClass(RetweetMapper1.class);
>>>      job.setReducerClass(RetweetReducer1.class);
>>>      job.setOutputKeyClass(Text.class);
>>>      job.setOutputValueClass(Text.class);
>>>      FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>>      FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>>      System.exit(job.waitForCompletion(true) ? 0 : 1);
>>>    }
>>> }
>>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>> Hi all,
>>>>
>>>> i got some more information:
>>>>
>>>> It is always only one TT, where mortbay errors do not occur: all other
>>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>>> And it has nothing to do with the local configuration of the server,
>>>> since
>>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>>> from one job to another.
>>>> 2) the same happens even when i am using a completely different set of
>>>> servers.
>>>>
>>>> Who is capable to solve this unsolved mystery? :D
>>>>
>>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>> Hi,
>>>>>
>>>>> is there anybody, who knows more about this issue: it has already been
>>>>> recently marked here:
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>>
>>>>> I really want to do something about it, if i knew how... i tried so many
>>>>> different setup parameters and JVM options and nothing did the trick...
>>>>>
>>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>>
>>>>> 2012-08-30 12:05:22,582 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay2 times
>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 2 times, canceled keys 70 times
>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:08:39,236 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:10:27,109 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:14:29,740 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> 2012-08-30 12:16:28,503 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>>> delay1 times
>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>>> recreating selector 1 times, canceled keys 69 times
>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>>> org.mortbay.jetty.EofException
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>>      at
>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> Caused by: java.io.IOException: Broken pipe
>>>>>      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>>      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>>      at
>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>>      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>>      ... 27 more
>>>>>
>>>>> 2012-08-30 12:20:01,205 INFO
>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>      at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>      at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>      at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>      at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>      at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>      at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>      at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>      at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>      at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>      at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>> Hi there,
>>>>>>
>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>>
>>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>>> tuples for evry incoming key-value-pair.
>>>>>>
>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>>> into the tasktracker's logs saying:
>>>>>>
>>>>>> ERROR org.mortbay.log: /mapOutput
>>>>>> java.lang.IllegalStateException: Committed
>>>>>>      at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>>      at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>>      at
>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>>      at
>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>>      at
>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>>      at
>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>>      at
>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>>      at
>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>>      at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>>      at
>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>>      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>>      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>>      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>>      at
>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>>      at
>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>>
>>>>>>
>>>>>> And while it is no problem at the beginning of the reduce process,
>>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>>> shell tells me it is 400000000).
>>>>>>
>>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>>> naive math it should be:
>>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>>
>>>>>> Can anybody help me with this issue?
>>>>>>
>>>>>> Best regards,
>>>>>> Elmar
>>>>>>
>>>>>>
>>>>>
>>>>
>>>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Just a guess, but could it simply be memory issue on reducer?

In your adjusted program, maybe try running without
-UseGCOverheadLimit and see if you still got OOM errors.

>From the sun website:
The parallel collector will throw an OutOfMemoryError if too much time
is being spent in garbage collection: if more than 98% of the total
time is spent in garbage collection and less than 2% of the heap is
recovered, an OutOfMemoryError will be thrown. This feature is
designed to prevent applications from running for an extended period
of time while making little or no progress because the heap is too
small. If necessary, this feature can be disabled by adding the option
-XX:-UseGCOverheadLimit to the command line.

So basically if you get a GC overhead limit exceeded OOM error it
means that your app is doing nothing but garbage collection, the vm is
fighting against running out of memory. If that happens it might
result in a timeout of the connection fetching the map data (i don't
know if it can result in timeout, but I can imagine it could).

Also, note that 63 GB on disk is probably going to be inflated in
memory. So in general you cant say that 60 GB on disk will need 60GB
of mem. Actually, some people us a rule of thumb to do x4 to get
approx mem requirement.

Just some ideas, not really a solution but maybe it helps you further.

On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Excuse me: in my last code section was some old code included. Here is it
> again stripped of deprecated code:
>
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
> Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "word count");
>
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>
>> Hello again,
>>
>> i just wanted to keep you updated, in case anybody reads this and is
>> interested in the mortbay-issue:
>> I applied the wordcount example to a big input file and everything worked
>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>> occurred. The only possible explanation for me seemed to be, that my code
>> somehow produces the exceptions. So i started to keep my code as close to
>> the wordcount example as possible:
>>
>> * first getting rid of any members within the reducer class, that may take
>> lots of memory.
>> * using the Context class instead of OutputCollector (in Mapper and
>> Reducer)
>> * removing any Comparator (and restructuring the code so it just
>> calculates the same as with the sorting)
>> * using standart HashPartitioner
>>
>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>> seems to complete all the work while the others are busy with their
>> exceptions (at least from what i can see in the log).
>>
>> I REALLY dont know what i can do, to make this job run through like the
>> wordcount example. I dont see any big difference between my program and the
>> example anymore.
>>
>> I added the code here:
>>
>> *Please* help me here.
>>
>> Best regards,
>> Elmar
>>
>>
>> I added the code here:
>> ########################################
>> package uni.kassel.macek.rtprep;
>>
>> import java.util.Date;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.conf.Configured;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.FileInputFormat;
>> import org.apache.hadoop.mapred.FileOutputFormat;
>> import org.apache.hadoop.mapred.JobClient;
>> import org.apache.hadoop.mapred.JobConf;
>> import org.apache.hadoop.mapred.TextOutputFormat;
>> import org.apache.hadoop.util.Tool;
>> import org.apache.hadoop.util.ToolRunner;
>>
>> public class RetweetApplication extends Configured implements Tool{
>>
>>     public static int DEBUG = -1;
>>     private Configuration jobConf;
>>
>>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>> Date()).toString();
>>
>>     public static void main(String[] args) {
>>          int res=-1;
>>         try {
>>             res = ToolRunner.run(new Configuration(), new
>> RetweetApplication(), args);
>>         } catch (Exception e) {
>>             // TODO Auto-generated catch block
>>             e.printStackTrace();
>>         }
>>         System.exit(res);
>>
>>     }
>>
>>     public void setConf(Configuration conf) {
>>         jobConf = conf;
>>
>>     }
>>
>>     public Configuration getConf() {
>>         return jobConf;
>>     }
>>
>>     public int run(String[] args) throws Exception {
>>         JobClient client = new JobClient();
>>         JobConf conf = new JobConf(RetweetApplication.class);
>>
>>
>>         if(DEBUG > 0){
>>             conf.set("mapred.job.tracker", "local");
>>             conf.set("fs.default.name", "file:///");
>>             conf.set("dfs.replication", "1");
>>         }
>>
>>         //conf.set("mapred.child.java.opts","-Xmx3000m
>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>         conf.set("mapred.child.ulimit","100000000");
>>
>>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>
>>
>>         //conf.setOutputKeyClass(Text.class);
>>         //conf.setOutputValueClass(Text.class);
>>         conf.setMapOutputKeyClass(Text.class);
>>         conf.setMapOutputValueClass(Text.class);
>>
>>         conf.setMapperClass(RetweetMapper.class);
>>         //conf.setPartitionerClass(TweetPartitioner.class);
>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>
>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>         conf.setReducerClass(RetweetReducer.class);
>>
>>         conf.setOutputFormat(TextOutputFormat.class);
>>
>>         client.setConf(conf);
>>         try {
>>             JobClient.runJob(conf);
>>         } catch (Exception e) {
>>             e.printStackTrace();
>>         }
>>         return 0;
>>     }
>> }
>> */
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>         public void map(Object key, Text value,
>>             Context context)
>>             throws IOException {
>>
>>             String line = value.toString();
>>
>>             Tweet aTweet = new Tweet(line);
>>             if(aTweet.getMention()!=null &&
>> aTweet.getMention().length()>0){
>>                 /*aTweet.setRT(0);
>>                 output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                 aTweet.setRT(1);
>>                 try {
>>                     context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                 } catch (InterruptedException e) {
>>                     // TODO Auto-generated catch block
>>                     e.printStackTrace();
>>                 }
>>             }
>>             aTweet.setRT(0);
>>             try {
>>                 context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>             } catch (InterruptedException e) {
>>                 // TODO Auto-generated catch block
>>                 e.printStackTrace();
>>             }
>>
>>         }
>>     }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>> Text> {
>>
>>     public void reduce(Text atweet, Iterator<Text> tweets,
>>             Context context)
>>             throws IOException {
>>         List<Tweet> originals = new ArrayList<Tweet>();
>>         List<Tweet> retweets = new ArrayList<Tweet>();
>>         while(tweets.hasNext()){
>>             Tweet aTweet = new Tweet(tweets.toString());
>>             originals.add(aTweet);
>>             if(aTweet.getRT()>0) retweets.add(aTweet);
>>         }
>>
>>         for(Tweet originalTweet : originals){
>>             for(Tweet aRetweet : retweets){
>>                 if(aRetweet.isRetweeted(originalTweet))
>>                 {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                     try {
>>                         context.write(
>>                                 atweet,
>>                                 new Text(aRetweet.toString()));
>>                     } catch (InterruptedException e) {
>>                         // TODO Auto-generated catch block
>>                         e.printStackTrace();
>>                     }
>>                 }
>>             }
>>         }
>>     }
>> }
>>
>>   public static void main(String[] args) throws Exception {
>>     Configuration conf = new Configuration();
>>     String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>     if (otherArgs.length != 2) {
>>       System.err.println("Usage: wordcount <in> <out>");
>>       System.exit(2);
>>     }
>>     Job job = new Job(conf, "tweets");
>>     job.setJarByClass(RetweetApplication.class);
>>     job.setMapperClass(RetweetMapper1.class);
>>     job.setReducerClass(RetweetReducer1.class);
>>     job.setOutputKeyClass(Text.class);
>>     job.setOutputValueClass(Text.class);
>>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>>   }
>> }
>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>
>>> Hi all,
>>>
>>> i got some more information:
>>>
>>> It is always only one TT, where mortbay errors do not occur: all other
>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>> And it has nothing to do with the local configuration of the server,
>>> since
>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>> from one job to another.
>>> 2) the same happens even when i am using a completely different set of
>>> servers.
>>>
>>> Who is capable to solve this unsolved mystery? :D
>>>
>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>
>>>> Hi,
>>>>
>>>> is there anybody, who knows more about this issue: it has already been
>>>> recently marked here:
>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>
>>>> I really want to do something about it, if i knew how... i tried so many
>>>> different setup parameters and JVM options and nothing did the trick...
>>>>
>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>
>>>> 2012-08-30 12:05:22,582 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay2 times
>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 2 times, canceled keys 70 times
>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:10:27,109 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:16:28,503 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay1 times
>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 1 times, canceled keys 69 times
>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>
>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>> tuples for evry incoming key-value-pair.
>>>>>
>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>> into the tasktracker's logs saying:
>>>>>
>>>>> ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>     at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>     at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>     at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>     at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>     at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>     at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>     at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>>
>>>>> And while it is no problem at the beginning of the reduce process,
>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>> shell tells me it is 400000000).
>>>>>
>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>> naive math it should be:
>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>
>>>>> Can anybody help me with this issue?
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Just a guess, but could it simply be memory issue on reducer?

In your adjusted program, maybe try running without
-UseGCOverheadLimit and see if you still got OOM errors.

>From the sun website:
The parallel collector will throw an OutOfMemoryError if too much time
is being spent in garbage collection: if more than 98% of the total
time is spent in garbage collection and less than 2% of the heap is
recovered, an OutOfMemoryError will be thrown. This feature is
designed to prevent applications from running for an extended period
of time while making little or no progress because the heap is too
small. If necessary, this feature can be disabled by adding the option
-XX:-UseGCOverheadLimit to the command line.

So basically if you get a GC overhead limit exceeded OOM error it
means that your app is doing nothing but garbage collection, the vm is
fighting against running out of memory. If that happens it might
result in a timeout of the connection fetching the map data (i don't
know if it can result in timeout, but I can imagine it could).

Also, note that 63 GB on disk is probably going to be inflated in
memory. So in general you cant say that 60 GB on disk will need 60GB
of mem. Actually, some people us a rule of thumb to do x4 to get
approx mem requirement.

Just some ideas, not really a solution but maybe it helps you further.

On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Excuse me: in my last code section was some old code included. Here is it
> again stripped of deprecated code:
>
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
> Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "word count");
>
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>
>> Hello again,
>>
>> i just wanted to keep you updated, in case anybody reads this and is
>> interested in the mortbay-issue:
>> I applied the wordcount example to a big input file and everything worked
>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>> occurred. The only possible explanation for me seemed to be, that my code
>> somehow produces the exceptions. So i started to keep my code as close to
>> the wordcount example as possible:
>>
>> * first getting rid of any members within the reducer class, that may take
>> lots of memory.
>> * using the Context class instead of OutputCollector (in Mapper and
>> Reducer)
>> * removing any Comparator (and restructuring the code so it just
>> calculates the same as with the sorting)
>> * using standart HashPartitioner
>>
>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>> seems to complete all the work while the others are busy with their
>> exceptions (at least from what i can see in the log).
>>
>> I REALLY dont know what i can do, to make this job run through like the
>> wordcount example. I dont see any big difference between my program and the
>> example anymore.
>>
>> I added the code here:
>>
>> *Please* help me here.
>>
>> Best regards,
>> Elmar
>>
>>
>> I added the code here:
>> ########################################
>> package uni.kassel.macek.rtprep;
>>
>> import java.util.Date;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.conf.Configured;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.FileInputFormat;
>> import org.apache.hadoop.mapred.FileOutputFormat;
>> import org.apache.hadoop.mapred.JobClient;
>> import org.apache.hadoop.mapred.JobConf;
>> import org.apache.hadoop.mapred.TextOutputFormat;
>> import org.apache.hadoop.util.Tool;
>> import org.apache.hadoop.util.ToolRunner;
>>
>> public class RetweetApplication extends Configured implements Tool{
>>
>>     public static int DEBUG = -1;
>>     private Configuration jobConf;
>>
>>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>> Date()).toString();
>>
>>     public static void main(String[] args) {
>>          int res=-1;
>>         try {
>>             res = ToolRunner.run(new Configuration(), new
>> RetweetApplication(), args);
>>         } catch (Exception e) {
>>             // TODO Auto-generated catch block
>>             e.printStackTrace();
>>         }
>>         System.exit(res);
>>
>>     }
>>
>>     public void setConf(Configuration conf) {
>>         jobConf = conf;
>>
>>     }
>>
>>     public Configuration getConf() {
>>         return jobConf;
>>     }
>>
>>     public int run(String[] args) throws Exception {
>>         JobClient client = new JobClient();
>>         JobConf conf = new JobConf(RetweetApplication.class);
>>
>>
>>         if(DEBUG > 0){
>>             conf.set("mapred.job.tracker", "local");
>>             conf.set("fs.default.name", "file:///");
>>             conf.set("dfs.replication", "1");
>>         }
>>
>>         //conf.set("mapred.child.java.opts","-Xmx3000m
>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>         conf.set("mapred.child.ulimit","100000000");
>>
>>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>
>>
>>         //conf.setOutputKeyClass(Text.class);
>>         //conf.setOutputValueClass(Text.class);
>>         conf.setMapOutputKeyClass(Text.class);
>>         conf.setMapOutputValueClass(Text.class);
>>
>>         conf.setMapperClass(RetweetMapper.class);
>>         //conf.setPartitionerClass(TweetPartitioner.class);
>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>
>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>         conf.setReducerClass(RetweetReducer.class);
>>
>>         conf.setOutputFormat(TextOutputFormat.class);
>>
>>         client.setConf(conf);
>>         try {
>>             JobClient.runJob(conf);
>>         } catch (Exception e) {
>>             e.printStackTrace();
>>         }
>>         return 0;
>>     }
>> }
>> */
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>         public void map(Object key, Text value,
>>             Context context)
>>             throws IOException {
>>
>>             String line = value.toString();
>>
>>             Tweet aTweet = new Tweet(line);
>>             if(aTweet.getMention()!=null &&
>> aTweet.getMention().length()>0){
>>                 /*aTweet.setRT(0);
>>                 output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                 aTweet.setRT(1);
>>                 try {
>>                     context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                 } catch (InterruptedException e) {
>>                     // TODO Auto-generated catch block
>>                     e.printStackTrace();
>>                 }
>>             }
>>             aTweet.setRT(0);
>>             try {
>>                 context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>             } catch (InterruptedException e) {
>>                 // TODO Auto-generated catch block
>>                 e.printStackTrace();
>>             }
>>
>>         }
>>     }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>> Text> {
>>
>>     public void reduce(Text atweet, Iterator<Text> tweets,
>>             Context context)
>>             throws IOException {
>>         List<Tweet> originals = new ArrayList<Tweet>();
>>         List<Tweet> retweets = new ArrayList<Tweet>();
>>         while(tweets.hasNext()){
>>             Tweet aTweet = new Tweet(tweets.toString());
>>             originals.add(aTweet);
>>             if(aTweet.getRT()>0) retweets.add(aTweet);
>>         }
>>
>>         for(Tweet originalTweet : originals){
>>             for(Tweet aRetweet : retweets){
>>                 if(aRetweet.isRetweeted(originalTweet))
>>                 {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                     try {
>>                         context.write(
>>                                 atweet,
>>                                 new Text(aRetweet.toString()));
>>                     } catch (InterruptedException e) {
>>                         // TODO Auto-generated catch block
>>                         e.printStackTrace();
>>                     }
>>                 }
>>             }
>>         }
>>     }
>> }
>>
>>   public static void main(String[] args) throws Exception {
>>     Configuration conf = new Configuration();
>>     String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>     if (otherArgs.length != 2) {
>>       System.err.println("Usage: wordcount <in> <out>");
>>       System.exit(2);
>>     }
>>     Job job = new Job(conf, "tweets");
>>     job.setJarByClass(RetweetApplication.class);
>>     job.setMapperClass(RetweetMapper1.class);
>>     job.setReducerClass(RetweetReducer1.class);
>>     job.setOutputKeyClass(Text.class);
>>     job.setOutputValueClass(Text.class);
>>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>>   }
>> }
>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>
>>> Hi all,
>>>
>>> i got some more information:
>>>
>>> It is always only one TT, where mortbay errors do not occur: all other
>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>> And it has nothing to do with the local configuration of the server,
>>> since
>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>> from one job to another.
>>> 2) the same happens even when i am using a completely different set of
>>> servers.
>>>
>>> Who is capable to solve this unsolved mystery? :D
>>>
>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>
>>>> Hi,
>>>>
>>>> is there anybody, who knows more about this issue: it has already been
>>>> recently marked here:
>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>
>>>> I really want to do something about it, if i knew how... i tried so many
>>>> different setup parameters and JVM options and nothing did the trick...
>>>>
>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>
>>>> 2012-08-30 12:05:22,582 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay2 times
>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 2 times, canceled keys 70 times
>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:10:27,109 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:16:28,503 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay1 times
>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 1 times, canceled keys 69 times
>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>
>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>> tuples for evry incoming key-value-pair.
>>>>>
>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>> into the tasktracker's logs saying:
>>>>>
>>>>> ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>     at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>     at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>     at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>     at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>     at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>     at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>     at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>>
>>>>> And while it is no problem at the beginning of the reduce process,
>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>> shell tells me it is 400000000).
>>>>>
>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>> naive math it should be:
>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>
>>>>> Can anybody help me with this issue?
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Just a guess, but could it simply be memory issue on reducer?

In your adjusted program, maybe try running without
-UseGCOverheadLimit and see if you still got OOM errors.

>From the sun website:
The parallel collector will throw an OutOfMemoryError if too much time
is being spent in garbage collection: if more than 98% of the total
time is spent in garbage collection and less than 2% of the heap is
recovered, an OutOfMemoryError will be thrown. This feature is
designed to prevent applications from running for an extended period
of time while making little or no progress because the heap is too
small. If necessary, this feature can be disabled by adding the option
-XX:-UseGCOverheadLimit to the command line.

So basically if you get a GC overhead limit exceeded OOM error it
means that your app is doing nothing but garbage collection, the vm is
fighting against running out of memory. If that happens it might
result in a timeout of the connection fetching the map data (i don't
know if it can result in timeout, but I can imagine it could).

Also, note that 63 GB on disk is probably going to be inflated in
memory. So in general you cant say that 60 GB on disk will need 60GB
of mem. Actually, some people us a rule of thumb to do x4 to get
approx mem requirement.

Just some ideas, not really a solution but maybe it helps you further.

On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Excuse me: in my last code section was some old code included. Here is it
> again stripped of deprecated code:
>
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
> Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "word count");
>
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>
>> Hello again,
>>
>> i just wanted to keep you updated, in case anybody reads this and is
>> interested in the mortbay-issue:
>> I applied the wordcount example to a big input file and everything worked
>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>> occurred. The only possible explanation for me seemed to be, that my code
>> somehow produces the exceptions. So i started to keep my code as close to
>> the wordcount example as possible:
>>
>> * first getting rid of any members within the reducer class, that may take
>> lots of memory.
>> * using the Context class instead of OutputCollector (in Mapper and
>> Reducer)
>> * removing any Comparator (and restructuring the code so it just
>> calculates the same as with the sorting)
>> * using standart HashPartitioner
>>
>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>> seems to complete all the work while the others are busy with their
>> exceptions (at least from what i can see in the log).
>>
>> I REALLY dont know what i can do, to make this job run through like the
>> wordcount example. I dont see any big difference between my program and the
>> example anymore.
>>
>> I added the code here:
>>
>> *Please* help me here.
>>
>> Best regards,
>> Elmar
>>
>>
>> I added the code here:
>> ########################################
>> package uni.kassel.macek.rtprep;
>>
>> import java.util.Date;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.conf.Configured;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.FileInputFormat;
>> import org.apache.hadoop.mapred.FileOutputFormat;
>> import org.apache.hadoop.mapred.JobClient;
>> import org.apache.hadoop.mapred.JobConf;
>> import org.apache.hadoop.mapred.TextOutputFormat;
>> import org.apache.hadoop.util.Tool;
>> import org.apache.hadoop.util.ToolRunner;
>>
>> public class RetweetApplication extends Configured implements Tool{
>>
>>     public static int DEBUG = -1;
>>     private Configuration jobConf;
>>
>>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>> Date()).toString();
>>
>>     public static void main(String[] args) {
>>          int res=-1;
>>         try {
>>             res = ToolRunner.run(new Configuration(), new
>> RetweetApplication(), args);
>>         } catch (Exception e) {
>>             // TODO Auto-generated catch block
>>             e.printStackTrace();
>>         }
>>         System.exit(res);
>>
>>     }
>>
>>     public void setConf(Configuration conf) {
>>         jobConf = conf;
>>
>>     }
>>
>>     public Configuration getConf() {
>>         return jobConf;
>>     }
>>
>>     public int run(String[] args) throws Exception {
>>         JobClient client = new JobClient();
>>         JobConf conf = new JobConf(RetweetApplication.class);
>>
>>
>>         if(DEBUG > 0){
>>             conf.set("mapred.job.tracker", "local");
>>             conf.set("fs.default.name", "file:///");
>>             conf.set("dfs.replication", "1");
>>         }
>>
>>         //conf.set("mapred.child.java.opts","-Xmx3000m
>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>         conf.set("mapred.child.ulimit","100000000");
>>
>>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>
>>
>>         //conf.setOutputKeyClass(Text.class);
>>         //conf.setOutputValueClass(Text.class);
>>         conf.setMapOutputKeyClass(Text.class);
>>         conf.setMapOutputValueClass(Text.class);
>>
>>         conf.setMapperClass(RetweetMapper.class);
>>         //conf.setPartitionerClass(TweetPartitioner.class);
>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>
>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>         conf.setReducerClass(RetweetReducer.class);
>>
>>         conf.setOutputFormat(TextOutputFormat.class);
>>
>>         client.setConf(conf);
>>         try {
>>             JobClient.runJob(conf);
>>         } catch (Exception e) {
>>             e.printStackTrace();
>>         }
>>         return 0;
>>     }
>> }
>> */
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>         public void map(Object key, Text value,
>>             Context context)
>>             throws IOException {
>>
>>             String line = value.toString();
>>
>>             Tweet aTweet = new Tweet(line);
>>             if(aTweet.getMention()!=null &&
>> aTweet.getMention().length()>0){
>>                 /*aTweet.setRT(0);
>>                 output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                 aTweet.setRT(1);
>>                 try {
>>                     context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                 } catch (InterruptedException e) {
>>                     // TODO Auto-generated catch block
>>                     e.printStackTrace();
>>                 }
>>             }
>>             aTweet.setRT(0);
>>             try {
>>                 context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>             } catch (InterruptedException e) {
>>                 // TODO Auto-generated catch block
>>                 e.printStackTrace();
>>             }
>>
>>         }
>>     }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>> Text> {
>>
>>     public void reduce(Text atweet, Iterator<Text> tweets,
>>             Context context)
>>             throws IOException {
>>         List<Tweet> originals = new ArrayList<Tweet>();
>>         List<Tweet> retweets = new ArrayList<Tweet>();
>>         while(tweets.hasNext()){
>>             Tweet aTweet = new Tweet(tweets.toString());
>>             originals.add(aTweet);
>>             if(aTweet.getRT()>0) retweets.add(aTweet);
>>         }
>>
>>         for(Tweet originalTweet : originals){
>>             for(Tweet aRetweet : retweets){
>>                 if(aRetweet.isRetweeted(originalTweet))
>>                 {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                     try {
>>                         context.write(
>>                                 atweet,
>>                                 new Text(aRetweet.toString()));
>>                     } catch (InterruptedException e) {
>>                         // TODO Auto-generated catch block
>>                         e.printStackTrace();
>>                     }
>>                 }
>>             }
>>         }
>>     }
>> }
>>
>>   public static void main(String[] args) throws Exception {
>>     Configuration conf = new Configuration();
>>     String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>     if (otherArgs.length != 2) {
>>       System.err.println("Usage: wordcount <in> <out>");
>>       System.exit(2);
>>     }
>>     Job job = new Job(conf, "tweets");
>>     job.setJarByClass(RetweetApplication.class);
>>     job.setMapperClass(RetweetMapper1.class);
>>     job.setReducerClass(RetweetReducer1.class);
>>     job.setOutputKeyClass(Text.class);
>>     job.setOutputValueClass(Text.class);
>>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>>   }
>> }
>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>
>>> Hi all,
>>>
>>> i got some more information:
>>>
>>> It is always only one TT, where mortbay errors do not occur: all other
>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>> And it has nothing to do with the local configuration of the server,
>>> since
>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>> from one job to another.
>>> 2) the same happens even when i am using a completely different set of
>>> servers.
>>>
>>> Who is capable to solve this unsolved mystery? :D
>>>
>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>
>>>> Hi,
>>>>
>>>> is there anybody, who knows more about this issue: it has already been
>>>> recently marked here:
>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>
>>>> I really want to do something about it, if i knew how... i tried so many
>>>> different setup parameters and JVM options and nothing did the trick...
>>>>
>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>
>>>> 2012-08-30 12:05:22,582 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay2 times
>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 2 times, canceled keys 70 times
>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:10:27,109 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:16:28,503 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay1 times
>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 1 times, canceled keys 69 times
>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>
>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>> tuples for evry incoming key-value-pair.
>>>>>
>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>> into the tasktracker's logs saying:
>>>>>
>>>>> ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>     at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>     at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>     at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>     at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>     at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>     at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>     at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>>
>>>>> And while it is no problem at the beginning of the reduce process,
>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>> shell tells me it is 400000000).
>>>>>
>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>> naive math it should be:
>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>
>>>>> Can anybody help me with this issue?
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

Re: mortbay, huge files and the ulimit

Posted by Vasco Visser <va...@gmail.com>.

Just a guess, but could it simply be memory issue on reducer?

In your adjusted program, maybe try running without
-UseGCOverheadLimit and see if you still got OOM errors.

>From the sun website:
The parallel collector will throw an OutOfMemoryError if too much time
is being spent in garbage collection: if more than 98% of the total
time is spent in garbage collection and less than 2% of the heap is
recovered, an OutOfMemoryError will be thrown. This feature is
designed to prevent applications from running for an extended period
of time while making little or no progress because the heap is too
small. If necessary, this feature can be disabled by adding the option
-XX:-UseGCOverheadLimit to the command line.

So basically if you get a GC overhead limit exceeded OOM error it
means that your app is doing nothing but garbage collection, the vm is
fighting against running out of memory. If that happens it might
result in a timeout of the connection fetching the map data (i don't
know if it can result in timeout, but I can imagine it could).

Also, note that 63 GB on disk is probably going to be inflated in
memory. So in general you cant say that 60 GB on disk will need 60GB
of mem. Actually, some people us a rule of thumb to do x4 to get
approx mem requirement.

Just some ideas, not really a solution but maybe it helps you further.

On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
<ma...@cs.uni-kassel.de> wrote:
> Excuse me: in my last code section was some old code included. Here is it
> again stripped of deprecated code:
>
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
> Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, Text>
> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "word count");
>
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
>
>> Hello again,
>>
>> i just wanted to keep you updated, in case anybody reads this and is
>> interested in the mortbay-issue:
>> I applied the wordcount example to a big input file and everything worked
>> fine. I watched the TT logs all the time and no single mortbay-eof exception
>> occurred. The only possible explanation for me seemed to be, that my code
>> somehow produces the exceptions. So i started to keep my code as close to
>> the wordcount example as possible:
>>
>> * first getting rid of any members within the reducer class, that may take
>> lots of memory.
>> * using the Context class instead of OutputCollector (in Mapper and
>> Reducer)
>> * removing any Comparator (and restructuring the code so it just
>> calculates the same as with the sorting)
>> * using standart HashPartitioner
>>
>> Especially the first 2 points made my job succeed in the end, ALTHOUGH the
>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node
>> seems to complete all the work while the others are busy with their
>> exceptions (at least from what i can see in the log).
>>
>> I REALLY dont know what i can do, to make this job run through like the
>> wordcount example. I dont see any big difference between my program and the
>> example anymore.
>>
>> I added the code here:
>>
>> *Please* help me here.
>>
>> Best regards,
>> Elmar
>>
>>
>> I added the code here:
>> ########################################
>> package uni.kassel.macek.rtprep;
>>
>> import java.util.Date;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.conf.Configured;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.FileInputFormat;
>> import org.apache.hadoop.mapred.FileOutputFormat;
>> import org.apache.hadoop.mapred.JobClient;
>> import org.apache.hadoop.mapred.JobConf;
>> import org.apache.hadoop.mapred.TextOutputFormat;
>> import org.apache.hadoop.util.Tool;
>> import org.apache.hadoop.util.ToolRunner;
>>
>> public class RetweetApplication extends Configured implements Tool{
>>
>>     public static int DEBUG = -1;
>>     private Configuration jobConf;
>>
>>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
>> Date()).toString();
>>
>>     public static void main(String[] args) {
>>          int res=-1;
>>         try {
>>             res = ToolRunner.run(new Configuration(), new
>> RetweetApplication(), args);
>>         } catch (Exception e) {
>>             // TODO Auto-generated catch block
>>             e.printStackTrace();
>>         }
>>         System.exit(res);
>>
>>     }
>>
>>     public void setConf(Configuration conf) {
>>         jobConf = conf;
>>
>>     }
>>
>>     public Configuration getConf() {
>>         return jobConf;
>>     }
>>
>>     public int run(String[] args) throws Exception {
>>         JobClient client = new JobClient();
>>         JobConf conf = new JobConf(RetweetApplication.class);
>>
>>
>>         if(DEBUG > 0){
>>             conf.set("mapred.job.tracker", "local");
>>             conf.set("fs.default.name", "file:///");
>>             conf.set("dfs.replication", "1");
>>         }
>>
>>         //conf.set("mapred.child.java.opts","-Xmx3000m
>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>>         conf.set("mapred.child.ulimit","100000000");
>>
>>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>
>>
>>         //conf.setOutputKeyClass(Text.class);
>>         //conf.setOutputValueClass(Text.class);
>>         conf.setMapOutputKeyClass(Text.class);
>>         conf.setMapOutputValueClass(Text.class);
>>
>>         conf.setMapperClass(RetweetMapper.class);
>>         //conf.setPartitionerClass(TweetPartitioner.class);
>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
>>
>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
>>         conf.setReducerClass(RetweetReducer.class);
>>
>>         conf.setOutputFormat(TextOutputFormat.class);
>>
>>         client.setConf(conf);
>>         try {
>>             JobClient.runJob(conf);
>>         } catch (Exception e) {
>>             e.printStackTrace();
>>         }
>>         return 0;
>>     }
>> }
>> */
>>
>> package uni.kassel.macek.rtprep;
>>
>>
>>
>> import gnu.trove.iterator.TIntIterator;
>> import gnu.trove.map.hash.TIntObjectHashMap;
>> import gnu.trove.set.TIntSet;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.Calendar;
>> import java.util.Iterator;
>> import java.util.List;
>> import java.util.StringTokenizer;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IntWritable;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapred.MapReduceBase;
>> import org.apache.hadoop.mapred.OutputCollector;
>> import org.apache.hadoop.mapred.Reporter;
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.Mapper;
>> import org.apache.hadoop.mapreduce.Reducer;
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> import org.apache.hadoop.util.GenericOptionsParser;
>>
>> public class RetweetApplication {
>>
>>     public static class RetweetMapper1 extends Mapper<Object, Text, Text,
>> Text> {
>>
>>         public void map(Object key, Text value,
>>             Context context)
>>             throws IOException {
>>
>>             String line = value.toString();
>>
>>             Tweet aTweet = new Tweet(line);
>>             if(aTweet.getMention()!=null &&
>> aTweet.getMention().length()>0){
>>                 /*aTweet.setRT(0);
>>                 output.collect(new Text(aTweet.toString()), new
>> Text(aTweet.toString()));*/
>>                 aTweet.setRT(1);
>>                 try {
>>                     context.write(new Text(aTweet.getMention()), new
>> Text(aTweet.toString()));
>>                 } catch (InterruptedException e) {
>>                     // TODO Auto-generated catch block
>>                     e.printStackTrace();
>>                 }
>>             }
>>             aTweet.setRT(0);
>>             try {
>>                 context.write(new Text(aTweet.getAuthor()), new
>> Text(aTweet.toString()));
>>             } catch (InterruptedException e) {
>>                 // TODO Auto-generated catch block
>>                 e.printStackTrace();
>>             }
>>
>>         }
>>     }
>>
>> public static class RetweetReducer1 extends Reducer<Text, Text, Text,
>> Text> {
>>
>>     public void reduce(Text atweet, Iterator<Text> tweets,
>>             Context context)
>>             throws IOException {
>>         List<Tweet> originals = new ArrayList<Tweet>();
>>         List<Tweet> retweets = new ArrayList<Tweet>();
>>         while(tweets.hasNext()){
>>             Tweet aTweet = new Tweet(tweets.toString());
>>             originals.add(aTweet);
>>             if(aTweet.getRT()>0) retweets.add(aTweet);
>>         }
>>
>>         for(Tweet originalTweet : originals){
>>             for(Tweet aRetweet : retweets){
>>                 if(aRetweet.isRetweeted(originalTweet))
>>                 {
>> aRetweet.setReactionTime(originalTweet.getDate());
>>                     try {
>>                         context.write(
>>                                 atweet,
>>                                 new Text(aRetweet.toString()));
>>                     } catch (InterruptedException e) {
>>                         // TODO Auto-generated catch block
>>                         e.printStackTrace();
>>                     }
>>                 }
>>             }
>>         }
>>     }
>> }
>>
>>   public static void main(String[] args) throws Exception {
>>     Configuration conf = new Configuration();
>>     String[] otherArgs = new GenericOptionsParser(conf,
>> args).getRemainingArgs();
>>     if (otherArgs.length != 2) {
>>       System.err.println("Usage: wordcount <in> <out>");
>>       System.exit(2);
>>     }
>>     Job job = new Job(conf, "tweets");
>>     job.setJarByClass(RetweetApplication.class);
>>     job.setMapperClass(RetweetMapper1.class);
>>     job.setReducerClass(RetweetReducer1.class);
>>     job.setOutputKeyClass(Text.class);
>>     job.setOutputValueClass(Text.class);
>>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>>   }
>> }
>> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>>>
>>> Hi all,
>>>
>>> i got some more information:
>>>
>>> It is always only one TT, where mortbay errors do not occur: all other
>>> slaves fail relatively fast after the reduce process started: maybe it is a
>>> comparator issue (since the operation is MAPRED_SHUFFLE)
>>> And it has nothing to do with the local configuration of the server,
>>> since
>>> 1) if i rerun the job on the same set of servers, the ONE working differs
>>> from one job to another.
>>> 2) the same happens even when i am using a completely different set of
>>> servers.
>>>
>>> Who is capable to solve this unsolved mystery? :D
>>>
>>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>>>
>>>> Hi,
>>>>
>>>> is there anybody, who knows more about this issue: it has already been
>>>> recently marked here:
>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>>
>>>> I really want to do something about it, if i knew how... i tried so many
>>>> different setup parameters and JVM options and nothing did the trick...
>>>>
>>>> It was opened 2009 and gets still updated (2012): I mean - this does NOT
>>>> look OK, does it? The whole TT seems like a big time waster:
>>>>
>>>> 2012-08-30 12:05:22,582 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852
>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay2 times
>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 2 times, canceled keys 70 times
>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:08:39,236 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410
>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:10:27,109 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517
>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:14:29,740 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212
>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> 2012-08-30 12:16:28,503 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393
>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - injecting
>>>> delay1 times
>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log:
>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
>>>> recreating selector 1 times, canceled keys 69 times
>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker:
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410
>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>>> org.mortbay.jetty.EofException
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>>     at
>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>>     at
>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>>     ... 27 more
>>>>
>>>> 2012-08-30 12:20:01,205 INFO
>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:35497,
>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID:
>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031
>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>     at
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>     at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>     at
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv
>>>>> file on 20 physically distinct nodes with 15GB each:
>>>>>
>>>>> While the mapping works just fine and is low cost, the reducer does the
>>>>> main work: holding a hashmap with elements to join with and finding join
>>>>> tuples for evry incoming key-value-pair.
>>>>>
>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts to
>>>>> get "unstable" as the file size goes up: this becomes evident with a look
>>>>> into the tasktracker's logs saying:
>>>>>
>>>>> ERROR org.mortbay.log: /mapOutput
>>>>> java.lang.IllegalStateException: Committed
>>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>>     at
>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>>     at
>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>     at
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>>     at
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>     at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>>     at
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>>     at
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>     at
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>     at
>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>     at
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>>
>>>>> And while it is no problem at the beginning of the reduce process,
>>>>> where this happens only on a few nodes and rarely, it becomes crucial as the
>>>>> progress rises. The reason for this (afaik from reading articles), is that
>>>>> there are memory or file handle problems. I addressed the memory problem by
>>>>> conitiously purging the map of outdated elements evry 5 million processed
>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit in the
>>>>> shell tells me it is 400000000).
>>>>>
>>>>> Anyway i am still running into those mortbay errors and i start to
>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By pure
>>>>> naive math it should be:
>>>>> i explicily assigned 10GB memory to each JVM on each node and set
>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastructures
>>>>> cause the GC to throw exceptions). This would naively make 18 slave machines
>>>>> with 10GB each resulting in an overall memory of 180GB - three times as much
>>>>> as needed... i would think. So if the Partitioner distributes them just
>>>>> about equally to all nodes i should not run into any errors, do i?
>>>>>
>>>>> Can anybody help me with this issue?
>>>>>
>>>>> Best regards,
>>>>> Elmar
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Excuse me: in my last code section was some old code included. Here is 
it again stripped of deprecated code:

package uni.kassel.macek.rtprep;



import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text, 
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null && 
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new 
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new 
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor()), new 
Text(aTweet.toString()));
             } catch (InterruptedException e) {
                 // TODO Auto-generated catch block
                 e.printStackTrace();
             }

         }
     }

public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
Text> {

     public void reduce(Text atweet, Iterator<Text> tweets,
             Context context)
             throws IOException {
         List<Tweet> originals = new ArrayList<Tweet>();
         List<Tweet> retweets = new ArrayList<Tweet>();
         while(tweets.hasNext()){
             Tweet aTweet = new Tweet(tweets.toString());
             originals.add(aTweet);
             if(aTweet.getRT()>0) retweets.add(aTweet);
         }

         for(Tweet originalTweet : originals){
             for(Tweet aRetweet : retweets){
                 if(aRetweet.isRetweeted(originalTweet))
                 {
aRetweet.setReactionTime(originalTweet.getDate());
                     try {
                         context.write(
                                 atweet,
                                 new Text(aRetweet.toString()));
                     } catch (InterruptedException e) {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                     }
                 }
             }
         }
     }
}

   public static void main(String[] args) throws Exception {
     Configuration conf = new Configuration();
     String[] otherArgs = new GenericOptionsParser(conf, 
args).getRemainingArgs();
     if (otherArgs.length != 2) {
       System.err.println("Usage: wordcount <in> <out>");
       System.exit(2);
     }
     Job job = new Job(conf, "word count");
     job.setJarByClass(RetweetApplication.class);
     job.setMapperClass(RetweetMapper1.class);
     job.setReducerClass(RetweetReducer1.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);
     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
     System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
> Hello again,
>
> i just wanted to keep you updated, in case anybody reads this and is 
> interested in the mortbay-issue:
> I applied the wordcount example to a big input file and everything 
> worked fine. I watched the TT logs all the time and no single 
> mortbay-eof exception occurred. The only possible explanation for me 
> seemed to be, that my code somehow produces the exceptions. So i 
> started to keep my code as close to the wordcount example as possible:
>
> * first getting rid of any members within the reducer class, that may 
> take lots of memory.
> * using the Context class instead of OutputCollector (in Mapper and 
> Reducer)
> * removing any Comparator (and restructuring the code so it just 
> calculates the same as with the sorting)
> * using standart HashPartitioner
>
> Especially the first 2 points made my job succeed in the end, ALTHOUGH 
> the problem of "n-1 mortbay-exception-throwing TTs" still remains: one 
> node seems to complete all the work while the others are busy with 
> their exceptions (at least from what i can see in the log).
>
> I REALLY dont know what i can do, to make this job run through like 
> the wordcount example. I dont see any big difference between my 
> program and the example anymore.
>
> I added the code here:
>
> *Please* help me here.
>
> Best regards,
> Elmar
>
>
> I added the code here:
> ########################################
> package uni.kassel.macek.rtprep;
>
> import java.util.Date;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.FileInputFormat;
> import org.apache.hadoop.mapred.FileOutputFormat;
> import org.apache.hadoop.mapred.JobClient;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.TextOutputFormat;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
>
> public class RetweetApplication extends Configured implements Tool{
>
>     public static int DEBUG = -1;
>     private Configuration jobConf;
>
>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new 
> Date()).toString();
>
>     public static void main(String[] args) {
>          int res=-1;
>         try {
>             res = ToolRunner.run(new Configuration(), new 
> RetweetApplication(), args);
>         } catch (Exception e) {
>             // TODO Auto-generated catch block
>             e.printStackTrace();
>         }
>         System.exit(res);
>
>     }
>
>     public void setConf(Configuration conf) {
>         jobConf = conf;
>
>     }
>
>     public Configuration getConf() {
>         return jobConf;
>     }
>
>     public int run(String[] args) throws Exception {
>         JobClient client = new JobClient();
>         JobConf conf = new JobConf(RetweetApplication.class);
>
>
>         if(DEBUG > 0){
>             conf.set("mapred.job.tracker", "local");
>             conf.set("fs.default.name", "file:///");
>             conf.set("dfs.replication", "1");
>         }
>
>         //conf.set("mapred.child.java.opts","-Xmx3000m 
> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>         conf.set("mapred.child.ulimit","100000000");
>
>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
>
>         //conf.setOutputKeyClass(Text.class);
>         //conf.setOutputValueClass(Text.class);
>         conf.setMapOutputKeyClass(Text.class);
>         conf.setMapOutputValueClass(Text.class);
>
>         conf.setMapperClass(RetweetMapper.class);
>         //conf.setPartitionerClass(TweetPartitioner.class);
> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class); 
>
>         conf.setReducerClass(RetweetReducer.class);
>
>         conf.setOutputFormat(TextOutputFormat.class);
>
>         client.setConf(conf);
>         try {
>             JobClient.runJob(conf);
>         } catch (Exception e) {
>             e.printStackTrace();
>         }
>         return 0;
>     }
> }
> */
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, 
> Text, Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && 
> aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new 
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new 
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new 
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
> Text> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf, 
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "tweets");
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>> Hi all,
>>
>> i got some more information:
>>
>> It is always only one TT, where mortbay errors do not occur: all 
>> other slaves fail relatively fast after the reduce process started: 
>> maybe it is a comparator issue (since the operation is MAPRED_SHUFFLE)
>> And it has nothing to do with the local configuration of the server, 
>> since
>> 1) if i rerun the job on the same set of servers, the ONE working 
>> differs from one job to another.
>> 2) the same happens even when i am using a completely different set 
>> of servers.
>>
>> Who is capable to solve this unsolved mystery? :D
>>
>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>> Hi,
>>>
>>> is there anybody, who knows more about this issue: it has already 
>>> been recently marked here:
>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>
>>> I really want to do something about it, if i knew how... i tried so 
>>> many different setup parameters and JVM options and nothing did the 
>>> trick...
>>>
>>> It was opened 2009 and gets still updated (2012): I mean - this does 
>>> NOT look OK, does it? The whole TT seems like a big time waster:
>>>
>>> 2012-08-30 12:05:22,582 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, 
>>> duration: 5456122852
>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay2 times
>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 2 times, canceled keys 70 times
>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 3209410
>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:10:27,109 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 5686096517
>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 3746212
>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:16:28,503 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 7793391393
>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay1 times
>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 1 times, canceled keys 69 times
>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, 
>>> duration: 5405031
>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>> Hi there,
>>>>
>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv 
>>>> file on 20 physically distinct nodes with 15GB each:
>>>>
>>>> While the mapping works just fine and is low cost, the reducer does 
>>>> the main work: holding a hashmap with elements to join with and 
>>>> finding join tuples for evry incoming key-value-pair.
>>>>
>>>> The jobs works perfectly on small files with 2 gigabytes, but 
>>>> starts to get "unstable" as the file size goes up: this becomes 
>>>> evident with a look into the tasktracker's logs saying:
>>>>
>>>> ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at 
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at 
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at 
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at 
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>>
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at 
>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at 
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at 
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> And while it is no problem at the beginning of the reduce process, 
>>>> where this happens only on a few nodes and rarely, it becomes 
>>>> crucial as the progress rises. The reason for this (afaik from 
>>>> reading articles), is that there are memory or file handle 
>>>> problems. I addressed the memory problem by conitiously purging the 
>>>> map of outdated elements evry 5 million processed key-value-pairs. 
>>>> And i set mapred.child.ulimit to 100000000 (ulimit in the shell 
>>>> tells me it is 400000000).
>>>>
>>>> Anyway i am still running into those mortbay errors and i start to 
>>>> wonder, if hadoop can manage the job with this algorithmn anyways. 
>>>> By pure naive math it should be:
>>>> i explicily assigned 10GB memory to each JVM on each node and set 
>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
>>>> datastructures cause the GC to throw exceptions). This would 
>>>> naively make 18 slave machines with 10GB each resulting in an 
>>>> overall memory of 180GB - three times as much as needed... i would 
>>>> think. So if the Partitioner distributes them just about equally to 
>>>> all nodes i should not run into any errors, do i?
>>>>
>>>> Can anybody help me with this issue?
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Excuse me: in my last code section was some old code included. Here is 
it again stripped of deprecated code:

package uni.kassel.macek.rtprep;



import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text, 
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null && 
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new 
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new 
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor()), new 
Text(aTweet.toString()));
             } catch (InterruptedException e) {
                 // TODO Auto-generated catch block
                 e.printStackTrace();
             }

         }
     }

public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
Text> {

     public void reduce(Text atweet, Iterator<Text> tweets,
             Context context)
             throws IOException {
         List<Tweet> originals = new ArrayList<Tweet>();
         List<Tweet> retweets = new ArrayList<Tweet>();
         while(tweets.hasNext()){
             Tweet aTweet = new Tweet(tweets.toString());
             originals.add(aTweet);
             if(aTweet.getRT()>0) retweets.add(aTweet);
         }

         for(Tweet originalTweet : originals){
             for(Tweet aRetweet : retweets){
                 if(aRetweet.isRetweeted(originalTweet))
                 {
aRetweet.setReactionTime(originalTweet.getDate());
                     try {
                         context.write(
                                 atweet,
                                 new Text(aRetweet.toString()));
                     } catch (InterruptedException e) {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                     }
                 }
             }
         }
     }
}

   public static void main(String[] args) throws Exception {
     Configuration conf = new Configuration();
     String[] otherArgs = new GenericOptionsParser(conf, 
args).getRemainingArgs();
     if (otherArgs.length != 2) {
       System.err.println("Usage: wordcount <in> <out>");
       System.exit(2);
     }
     Job job = new Job(conf, "word count");
     job.setJarByClass(RetweetApplication.class);
     job.setMapperClass(RetweetMapper1.class);
     job.setReducerClass(RetweetReducer1.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);
     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
     System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
> Hello again,
>
> i just wanted to keep you updated, in case anybody reads this and is 
> interested in the mortbay-issue:
> I applied the wordcount example to a big input file and everything 
> worked fine. I watched the TT logs all the time and no single 
> mortbay-eof exception occurred. The only possible explanation for me 
> seemed to be, that my code somehow produces the exceptions. So i 
> started to keep my code as close to the wordcount example as possible:
>
> * first getting rid of any members within the reducer class, that may 
> take lots of memory.
> * using the Context class instead of OutputCollector (in Mapper and 
> Reducer)
> * removing any Comparator (and restructuring the code so it just 
> calculates the same as with the sorting)
> * using standart HashPartitioner
>
> Especially the first 2 points made my job succeed in the end, ALTHOUGH 
> the problem of "n-1 mortbay-exception-throwing TTs" still remains: one 
> node seems to complete all the work while the others are busy with 
> their exceptions (at least from what i can see in the log).
>
> I REALLY dont know what i can do, to make this job run through like 
> the wordcount example. I dont see any big difference between my 
> program and the example anymore.
>
> I added the code here:
>
> *Please* help me here.
>
> Best regards,
> Elmar
>
>
> I added the code here:
> ########################################
> package uni.kassel.macek.rtprep;
>
> import java.util.Date;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.FileInputFormat;
> import org.apache.hadoop.mapred.FileOutputFormat;
> import org.apache.hadoop.mapred.JobClient;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.TextOutputFormat;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
>
> public class RetweetApplication extends Configured implements Tool{
>
>     public static int DEBUG = -1;
>     private Configuration jobConf;
>
>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new 
> Date()).toString();
>
>     public static void main(String[] args) {
>          int res=-1;
>         try {
>             res = ToolRunner.run(new Configuration(), new 
> RetweetApplication(), args);
>         } catch (Exception e) {
>             // TODO Auto-generated catch block
>             e.printStackTrace();
>         }
>         System.exit(res);
>
>     }
>
>     public void setConf(Configuration conf) {
>         jobConf = conf;
>
>     }
>
>     public Configuration getConf() {
>         return jobConf;
>     }
>
>     public int run(String[] args) throws Exception {
>         JobClient client = new JobClient();
>         JobConf conf = new JobConf(RetweetApplication.class);
>
>
>         if(DEBUG > 0){
>             conf.set("mapred.job.tracker", "local");
>             conf.set("fs.default.name", "file:///");
>             conf.set("dfs.replication", "1");
>         }
>
>         //conf.set("mapred.child.java.opts","-Xmx3000m 
> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>         conf.set("mapred.child.ulimit","100000000");
>
>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
>
>         //conf.setOutputKeyClass(Text.class);
>         //conf.setOutputValueClass(Text.class);
>         conf.setMapOutputKeyClass(Text.class);
>         conf.setMapOutputValueClass(Text.class);
>
>         conf.setMapperClass(RetweetMapper.class);
>         //conf.setPartitionerClass(TweetPartitioner.class);
> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class); 
>
>         conf.setReducerClass(RetweetReducer.class);
>
>         conf.setOutputFormat(TextOutputFormat.class);
>
>         client.setConf(conf);
>         try {
>             JobClient.runJob(conf);
>         } catch (Exception e) {
>             e.printStackTrace();
>         }
>         return 0;
>     }
> }
> */
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, 
> Text, Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && 
> aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new 
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new 
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new 
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
> Text> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf, 
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "tweets");
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>> Hi all,
>>
>> i got some more information:
>>
>> It is always only one TT, where mortbay errors do not occur: all 
>> other slaves fail relatively fast after the reduce process started: 
>> maybe it is a comparator issue (since the operation is MAPRED_SHUFFLE)
>> And it has nothing to do with the local configuration of the server, 
>> since
>> 1) if i rerun the job on the same set of servers, the ONE working 
>> differs from one job to another.
>> 2) the same happens even when i am using a completely different set 
>> of servers.
>>
>> Who is capable to solve this unsolved mystery? :D
>>
>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>> Hi,
>>>
>>> is there anybody, who knows more about this issue: it has already 
>>> been recently marked here:
>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>
>>> I really want to do something about it, if i knew how... i tried so 
>>> many different setup parameters and JVM options and nothing did the 
>>> trick...
>>>
>>> It was opened 2009 and gets still updated (2012): I mean - this does 
>>> NOT look OK, does it? The whole TT seems like a big time waster:
>>>
>>> 2012-08-30 12:05:22,582 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, 
>>> duration: 5456122852
>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay2 times
>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 2 times, canceled keys 70 times
>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 3209410
>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:10:27,109 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 5686096517
>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 3746212
>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:16:28,503 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 7793391393
>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay1 times
>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 1 times, canceled keys 69 times
>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, 
>>> duration: 5405031
>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>> Hi there,
>>>>
>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv 
>>>> file on 20 physically distinct nodes with 15GB each:
>>>>
>>>> While the mapping works just fine and is low cost, the reducer does 
>>>> the main work: holding a hashmap with elements to join with and 
>>>> finding join tuples for evry incoming key-value-pair.
>>>>
>>>> The jobs works perfectly on small files with 2 gigabytes, but 
>>>> starts to get "unstable" as the file size goes up: this becomes 
>>>> evident with a look into the tasktracker's logs saying:
>>>>
>>>> ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at 
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at 
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at 
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at 
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>>
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at 
>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at 
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at 
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> And while it is no problem at the beginning of the reduce process, 
>>>> where this happens only on a few nodes and rarely, it becomes 
>>>> crucial as the progress rises. The reason for this (afaik from 
>>>> reading articles), is that there are memory or file handle 
>>>> problems. I addressed the memory problem by conitiously purging the 
>>>> map of outdated elements evry 5 million processed key-value-pairs. 
>>>> And i set mapred.child.ulimit to 100000000 (ulimit in the shell 
>>>> tells me it is 400000000).
>>>>
>>>> Anyway i am still running into those mortbay errors and i start to 
>>>> wonder, if hadoop can manage the job with this algorithmn anyways. 
>>>> By pure naive math it should be:
>>>> i explicily assigned 10GB memory to each JVM on each node and set 
>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
>>>> datastructures cause the GC to throw exceptions). This would 
>>>> naively make 18 slave machines with 10GB each resulting in an 
>>>> overall memory of 180GB - three times as much as needed... i would 
>>>> think. So if the Partitioner distributes them just about equally to 
>>>> all nodes i should not run into any errors, do i?
>>>>
>>>> Can anybody help me with this issue?
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Excuse me: in my last code section was some old code included. Here is 
it again stripped of deprecated code:

package uni.kassel.macek.rtprep;



import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text, 
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null && 
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new 
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new 
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor()), new 
Text(aTweet.toString()));
             } catch (InterruptedException e) {
                 // TODO Auto-generated catch block
                 e.printStackTrace();
             }

         }
     }

public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
Text> {

     public void reduce(Text atweet, Iterator<Text> tweets,
             Context context)
             throws IOException {
         List<Tweet> originals = new ArrayList<Tweet>();
         List<Tweet> retweets = new ArrayList<Tweet>();
         while(tweets.hasNext()){
             Tweet aTweet = new Tweet(tweets.toString());
             originals.add(aTweet);
             if(aTweet.getRT()>0) retweets.add(aTweet);
         }

         for(Tweet originalTweet : originals){
             for(Tweet aRetweet : retweets){
                 if(aRetweet.isRetweeted(originalTweet))
                 {
aRetweet.setReactionTime(originalTweet.getDate());
                     try {
                         context.write(
                                 atweet,
                                 new Text(aRetweet.toString()));
                     } catch (InterruptedException e) {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                     }
                 }
             }
         }
     }
}

   public static void main(String[] args) throws Exception {
     Configuration conf = new Configuration();
     String[] otherArgs = new GenericOptionsParser(conf, 
args).getRemainingArgs();
     if (otherArgs.length != 2) {
       System.err.println("Usage: wordcount <in> <out>");
       System.exit(2);
     }
     Job job = new Job(conf, "word count");
     job.setJarByClass(RetweetApplication.class);
     job.setMapperClass(RetweetMapper1.class);
     job.setReducerClass(RetweetReducer1.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);
     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
     System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
> Hello again,
>
> i just wanted to keep you updated, in case anybody reads this and is 
> interested in the mortbay-issue:
> I applied the wordcount example to a big input file and everything 
> worked fine. I watched the TT logs all the time and no single 
> mortbay-eof exception occurred. The only possible explanation for me 
> seemed to be, that my code somehow produces the exceptions. So i 
> started to keep my code as close to the wordcount example as possible:
>
> * first getting rid of any members within the reducer class, that may 
> take lots of memory.
> * using the Context class instead of OutputCollector (in Mapper and 
> Reducer)
> * removing any Comparator (and restructuring the code so it just 
> calculates the same as with the sorting)
> * using standart HashPartitioner
>
> Especially the first 2 points made my job succeed in the end, ALTHOUGH 
> the problem of "n-1 mortbay-exception-throwing TTs" still remains: one 
> node seems to complete all the work while the others are busy with 
> their exceptions (at least from what i can see in the log).
>
> I REALLY dont know what i can do, to make this job run through like 
> the wordcount example. I dont see any big difference between my 
> program and the example anymore.
>
> I added the code here:
>
> *Please* help me here.
>
> Best regards,
> Elmar
>
>
> I added the code here:
> ########################################
> package uni.kassel.macek.rtprep;
>
> import java.util.Date;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.FileInputFormat;
> import org.apache.hadoop.mapred.FileOutputFormat;
> import org.apache.hadoop.mapred.JobClient;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.TextOutputFormat;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
>
> public class RetweetApplication extends Configured implements Tool{
>
>     public static int DEBUG = -1;
>     private Configuration jobConf;
>
>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new 
> Date()).toString();
>
>     public static void main(String[] args) {
>          int res=-1;
>         try {
>             res = ToolRunner.run(new Configuration(), new 
> RetweetApplication(), args);
>         } catch (Exception e) {
>             // TODO Auto-generated catch block
>             e.printStackTrace();
>         }
>         System.exit(res);
>
>     }
>
>     public void setConf(Configuration conf) {
>         jobConf = conf;
>
>     }
>
>     public Configuration getConf() {
>         return jobConf;
>     }
>
>     public int run(String[] args) throws Exception {
>         JobClient client = new JobClient();
>         JobConf conf = new JobConf(RetweetApplication.class);
>
>
>         if(DEBUG > 0){
>             conf.set("mapred.job.tracker", "local");
>             conf.set("fs.default.name", "file:///");
>             conf.set("dfs.replication", "1");
>         }
>
>         //conf.set("mapred.child.java.opts","-Xmx3000m 
> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>         conf.set("mapred.child.ulimit","100000000");
>
>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
>
>         //conf.setOutputKeyClass(Text.class);
>         //conf.setOutputValueClass(Text.class);
>         conf.setMapOutputKeyClass(Text.class);
>         conf.setMapOutputValueClass(Text.class);
>
>         conf.setMapperClass(RetweetMapper.class);
>         //conf.setPartitionerClass(TweetPartitioner.class);
> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class); 
>
>         conf.setReducerClass(RetweetReducer.class);
>
>         conf.setOutputFormat(TextOutputFormat.class);
>
>         client.setConf(conf);
>         try {
>             JobClient.runJob(conf);
>         } catch (Exception e) {
>             e.printStackTrace();
>         }
>         return 0;
>     }
> }
> */
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, 
> Text, Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && 
> aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new 
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new 
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new 
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
> Text> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf, 
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "tweets");
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>> Hi all,
>>
>> i got some more information:
>>
>> It is always only one TT, where mortbay errors do not occur: all 
>> other slaves fail relatively fast after the reduce process started: 
>> maybe it is a comparator issue (since the operation is MAPRED_SHUFFLE)
>> And it has nothing to do with the local configuration of the server, 
>> since
>> 1) if i rerun the job on the same set of servers, the ONE working 
>> differs from one job to another.
>> 2) the same happens even when i am using a completely different set 
>> of servers.
>>
>> Who is capable to solve this unsolved mystery? :D
>>
>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>> Hi,
>>>
>>> is there anybody, who knows more about this issue: it has already 
>>> been recently marked here:
>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>
>>> I really want to do something about it, if i knew how... i tried so 
>>> many different setup parameters and JVM options and nothing did the 
>>> trick...
>>>
>>> It was opened 2009 and gets still updated (2012): I mean - this does 
>>> NOT look OK, does it? The whole TT seems like a big time waster:
>>>
>>> 2012-08-30 12:05:22,582 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, 
>>> duration: 5456122852
>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay2 times
>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 2 times, canceled keys 70 times
>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 3209410
>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:10:27,109 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 5686096517
>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 3746212
>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:16:28,503 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 7793391393
>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay1 times
>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 1 times, canceled keys 69 times
>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, 
>>> duration: 5405031
>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>> Hi there,
>>>>
>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv 
>>>> file on 20 physically distinct nodes with 15GB each:
>>>>
>>>> While the mapping works just fine and is low cost, the reducer does 
>>>> the main work: holding a hashmap with elements to join with and 
>>>> finding join tuples for evry incoming key-value-pair.
>>>>
>>>> The jobs works perfectly on small files with 2 gigabytes, but 
>>>> starts to get "unstable" as the file size goes up: this becomes 
>>>> evident with a look into the tasktracker's logs saying:
>>>>
>>>> ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at 
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at 
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at 
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at 
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>>
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at 
>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at 
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at 
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> And while it is no problem at the beginning of the reduce process, 
>>>> where this happens only on a few nodes and rarely, it becomes 
>>>> crucial as the progress rises. The reason for this (afaik from 
>>>> reading articles), is that there are memory or file handle 
>>>> problems. I addressed the memory problem by conitiously purging the 
>>>> map of outdated elements evry 5 million processed key-value-pairs. 
>>>> And i set mapred.child.ulimit to 100000000 (ulimit in the shell 
>>>> tells me it is 400000000).
>>>>
>>>> Anyway i am still running into those mortbay errors and i start to 
>>>> wonder, if hadoop can manage the job with this algorithmn anyways. 
>>>> By pure naive math it should be:
>>>> i explicily assigned 10GB memory to each JVM on each node and set 
>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
>>>> datastructures cause the GC to throw exceptions). This would 
>>>> naively make 18 slave machines with 10GB each resulting in an 
>>>> overall memory of 180GB - three times as much as needed... i would 
>>>> think. So if the Partitioner distributes them just about equally to 
>>>> all nodes i should not run into any errors, do i?
>>>>
>>>> Can anybody help me with this issue?
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: mortbay, huge files and the ulimit

Posted by Björn-Elmar Macek <ma...@cs.uni-kassel.de>.

Excuse me: in my last code section was some old code included. Here is 
it again stripped of deprecated code:

package uni.kassel.macek.rtprep;



import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text, 
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null && 
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new 
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new 
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor()), new 
Text(aTweet.toString()));
             } catch (InterruptedException e) {
                 // TODO Auto-generated catch block
                 e.printStackTrace();
             }

         }
     }

public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
Text> {

     public void reduce(Text atweet, Iterator<Text> tweets,
             Context context)
             throws IOException {
         List<Tweet> originals = new ArrayList<Tweet>();
         List<Tweet> retweets = new ArrayList<Tweet>();
         while(tweets.hasNext()){
             Tweet aTweet = new Tweet(tweets.toString());
             originals.add(aTweet);
             if(aTweet.getRT()>0) retweets.add(aTweet);
         }

         for(Tweet originalTweet : originals){
             for(Tweet aRetweet : retweets){
                 if(aRetweet.isRetweeted(originalTweet))
                 {
aRetweet.setReactionTime(originalTweet.getDate());
                     try {
                         context.write(
                                 atweet,
                                 new Text(aRetweet.toString()));
                     } catch (InterruptedException e) {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                     }
                 }
             }
         }
     }
}

   public static void main(String[] args) throws Exception {
     Configuration conf = new Configuration();
     String[] otherArgs = new GenericOptionsParser(conf, 
args).getRemainingArgs();
     if (otherArgs.length != 2) {
       System.err.println("Usage: wordcount <in> <out>");
       System.exit(2);
     }
     Job job = new Job(conf, "word count");
     job.setJarByClass(RetweetApplication.class);
     job.setMapperClass(RetweetMapper1.class);
     job.setReducerClass(RetweetReducer1.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);
     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
     System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
Am 05.09.2012 13:56, schrieb Björn-Elmar Macek:
> Hello again,
>
> i just wanted to keep you updated, in case anybody reads this and is 
> interested in the mortbay-issue:
> I applied the wordcount example to a big input file and everything 
> worked fine. I watched the TT logs all the time and no single 
> mortbay-eof exception occurred. The only possible explanation for me 
> seemed to be, that my code somehow produces the exceptions. So i 
> started to keep my code as close to the wordcount example as possible:
>
> * first getting rid of any members within the reducer class, that may 
> take lots of memory.
> * using the Context class instead of OutputCollector (in Mapper and 
> Reducer)
> * removing any Comparator (and restructuring the code so it just 
> calculates the same as with the sorting)
> * using standart HashPartitioner
>
> Especially the first 2 points made my job succeed in the end, ALTHOUGH 
> the problem of "n-1 mortbay-exception-throwing TTs" still remains: one 
> node seems to complete all the work while the others are busy with 
> their exceptions (at least from what i can see in the log).
>
> I REALLY dont know what i can do, to make this job run through like 
> the wordcount example. I dont see any big difference between my 
> program and the example anymore.
>
> I added the code here:
>
> *Please* help me here.
>
> Best regards,
> Elmar
>
>
> I added the code here:
> ########################################
> package uni.kassel.macek.rtprep;
>
> import java.util.Date;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.FileInputFormat;
> import org.apache.hadoop.mapred.FileOutputFormat;
> import org.apache.hadoop.mapred.JobClient;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.TextOutputFormat;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
>
> public class RetweetApplication extends Configured implements Tool{
>
>     public static int DEBUG = -1;
>     private Configuration jobConf;
>
>     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
>     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new 
> Date()).toString();
>
>     public static void main(String[] args) {
>          int res=-1;
>         try {
>             res = ToolRunner.run(new Configuration(), new 
> RetweetApplication(), args);
>         } catch (Exception e) {
>             // TODO Auto-generated catch block
>             e.printStackTrace();
>         }
>         System.exit(res);
>
>     }
>
>     public void setConf(Configuration conf) {
>         jobConf = conf;
>
>     }
>
>     public Configuration getConf() {
>         return jobConf;
>     }
>
>     public int run(String[] args) throws Exception {
>         JobClient client = new JobClient();
>         JobConf conf = new JobConf(RetweetApplication.class);
>
>
>         if(DEBUG > 0){
>             conf.set("mapred.job.tracker", "local");
>             conf.set("fs.default.name", "file:///");
>             conf.set("dfs.replication", "1");
>         }
>
>         //conf.set("mapred.child.java.opts","-Xmx3000m 
> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
>         conf.set("mapred.child.ulimit","100000000");
>
>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
>
>         //conf.setOutputKeyClass(Text.class);
>         //conf.setOutputValueClass(Text.class);
>         conf.setMapOutputKeyClass(Text.class);
>         conf.setMapOutputValueClass(Text.class);
>
>         conf.setMapperClass(RetweetMapper.class);
>         //conf.setPartitionerClass(TweetPartitioner.class);
> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class); 
>
>         conf.setReducerClass(RetweetReducer.class);
>
>         conf.setOutputFormat(TextOutputFormat.class);
>
>         client.setConf(conf);
>         try {
>             JobClient.runJob(conf);
>         } catch (Exception e) {
>             e.printStackTrace();
>         }
>         return 0;
>     }
> }
> */
>
> package uni.kassel.macek.rtprep;
>
>
>
> import gnu.trove.iterator.TIntIterator;
> import gnu.trove.map.hash.TIntObjectHashMap;
> import gnu.trove.set.TIntSet;
>
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.Calendar;
> import java.util.Iterator;
> import java.util.List;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.util.GenericOptionsParser;
>
> public class RetweetApplication {
>
>     public static class RetweetMapper1 extends Mapper<Object, Text, 
> Text, Text> {
>
>         public void map(Object key, Text value,
>             Context context)
>             throws IOException {
>
>             String line = value.toString();
>
>             Tweet aTweet = new Tweet(line);
>             if(aTweet.getMention()!=null && 
> aTweet.getMention().length()>0){
>                 /*aTweet.setRT(0);
>                 output.collect(new Text(aTweet.toString()), new 
> Text(aTweet.toString()));*/
>                 aTweet.setRT(1);
>                 try {
>                     context.write(new Text(aTweet.getMention()), new 
> Text(aTweet.toString()));
>                 } catch (InterruptedException e) {
>                     // TODO Auto-generated catch block
>                     e.printStackTrace();
>                 }
>             }
>             aTweet.setRT(0);
>             try {
>                 context.write(new Text(aTweet.getAuthor()), new 
> Text(aTweet.toString()));
>             } catch (InterruptedException e) {
>                 // TODO Auto-generated catch block
>                 e.printStackTrace();
>             }
>
>         }
>     }
>
> public static class RetweetReducer1 extends Reducer<Text, Text, Text, 
> Text> {
>
>     public void reduce(Text atweet, Iterator<Text> tweets,
>             Context context)
>             throws IOException {
>         List<Tweet> originals = new ArrayList<Tweet>();
>         List<Tweet> retweets = new ArrayList<Tweet>();
>         while(tweets.hasNext()){
>             Tweet aTweet = new Tweet(tweets.toString());
>             originals.add(aTweet);
>             if(aTweet.getRT()>0) retweets.add(aTweet);
>         }
>
>         for(Tweet originalTweet : originals){
>             for(Tweet aRetweet : retweets){
>                 if(aRetweet.isRetweeted(originalTweet))
>                 {
> aRetweet.setReactionTime(originalTweet.getDate());
>                     try {
>                         context.write(
>                                 atweet,
>                                 new Text(aRetweet.toString()));
>                     } catch (InterruptedException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     }
>                 }
>             }
>         }
>     }
> }
>
>   public static void main(String[] args) throws Exception {
>     Configuration conf = new Configuration();
>     String[] otherArgs = new GenericOptionsParser(conf, 
> args).getRemainingArgs();
>     if (otherArgs.length != 2) {
>       System.err.println("Usage: wordcount <in> <out>");
>       System.exit(2);
>     }
>     Job job = new Job(conf, "tweets");
>     job.setJarByClass(RetweetApplication.class);
>     job.setMapperClass(RetweetMapper1.class);
>     job.setReducerClass(RetweetReducer1.class);
>     job.setOutputKeyClass(Text.class);
>     job.setOutputValueClass(Text.class);
>     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>     System.exit(job.waitForCompletion(true) ? 0 : 1);
>   }
> }
> Am 31.08.2012 14:08, schrieb Björn-Elmar Macek:
>> Hi all,
>>
>> i got some more information:
>>
>> It is always only one TT, where mortbay errors do not occur: all 
>> other slaves fail relatively fast after the reduce process started: 
>> maybe it is a comparator issue (since the operation is MAPRED_SHUFFLE)
>> And it has nothing to do with the local configuration of the server, 
>> since
>> 1) if i rerun the job on the same set of servers, the ONE working 
>> differs from one job to another.
>> 2) the same happens even when i am using a completely different set 
>> of servers.
>>
>> Who is capable to solve this unsolved mystery? :D
>>
>> Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
>>> Hi,
>>>
>>> is there anybody, who knows more about this issue: it has already 
>>> been recently marked here:
>>> https://issues.apache.org/jira/browse/MAPREDUCE-5
>>>
>>> I really want to do something about it, if i knew how... i tried so 
>>> many different setup parameters and JVM options and nothing did the 
>>> trick...
>>>
>>> It was opened 2009 and gets still updated (2012): I mean - this does 
>>> NOT look OK, does it? The whole TT seems like a big time waster:
>>>
>>> 2012-08-30 12:05:22,582 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, 
>>> duration: 5456122852
>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay2 times
>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 2 times, canceled keys 70 times
>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:08:39,236 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 3209410
>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:10:27,109 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
>>> duration: 5686096517
>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:14:29,740 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 3746212
>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> 2012-08-30 12:16:28,503 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, 
>>> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
>>> duration: 7793391393
>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> injecting delay1 times
>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: 
>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
>>> recreating selector 1 times, canceled keys 69 times
>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
>>> org.mortbay.jetty.EofException
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>>>     at 
>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Caused by: java.io.IOException: Broken pipe
>>>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>>>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>>>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>>>     at 
>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>>>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>>>     ... 27 more
>>>
>>> 2012-08-30 12:20:01,205 INFO 
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
>>> 141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
>>> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, 
>>> duration: 5405031
>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
>>> java.lang.IllegalStateException: Committed
>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>     at 
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>     at 
>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>     at 
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>
>>>     at 
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>     at 
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>
>>>     at 
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>     at 
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>     at 
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>
>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>     at 
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>     at 
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>     at 
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>     at 
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>>>> Hi there,
>>>>
>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv 
>>>> file on 20 physically distinct nodes with 15GB each:
>>>>
>>>> While the mapping works just fine and is low cost, the reducer does 
>>>> the main work: holding a hashmap with elements to join with and 
>>>> finding join tuples for evry incoming key-value-pair.
>>>>
>>>> The jobs works perfectly on small files with 2 gigabytes, but 
>>>> starts to get "unstable" as the file size goes up: this becomes 
>>>> evident with a look into the tasktracker's logs saying:
>>>>
>>>> ERROR org.mortbay.log: /mapOutput
>>>> java.lang.IllegalStateException: Committed
>>>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>>>     at 
>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>     at 
>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>     at 
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>     at 
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>     at 
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>     at 
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
>>>>
>>>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 
>>>>
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>     at 
>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>     at 
>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>     at 
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>     at 
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> And while it is no problem at the beginning of the reduce process, 
>>>> where this happens only on a few nodes and rarely, it becomes 
>>>> crucial as the progress rises. The reason for this (afaik from 
>>>> reading articles), is that there are memory or file handle 
>>>> problems. I addressed the memory problem by conitiously purging the 
>>>> map of outdated elements evry 5 million processed key-value-pairs. 
>>>> And i set mapred.child.ulimit to 100000000 (ulimit in the shell 
>>>> tells me it is 400000000).
>>>>
>>>> Anyway i am still running into those mortbay errors and i start to 
>>>> wonder, if hadoop can manage the job with this algorithmn anyways. 
>>>> By pure naive math it should be:
>>>> i explicily assigned 10GB memory to each JVM on each node and set 
>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
>>>> datastructures cause the GC to throw exceptions). This would 
>>>> naively make 18 slave machines with 10GB each resulting in an 
>>>> overall memory of 180GB - three times as much as needed... i would 
>>>> think. So if the Partitioner distributes them just about equally to 
>>>> all nodes i should not run into any errors, do i?
>>>>
>>>> Can anybody help me with this issue?
>>>>
>>>> Best regards,
>>>> Elmar
>>>>
>>>>
>>>
>>>
>>
>>
>
>