org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 143

[Jakob Ericsson <ja...@gmail.com>]

Hello,

I have some questions about "Exit code from task is : 143". Why do we
get it and what is it?

Background.

We are running a cluster with about ~80 datanodes using CDH 4.1.1 on Debian.
Recently added ~20 new datanodes to our cluster. Both HDFS and
Mapreduce seems to be functional, tasks complete on all nodes
(jobtracker thinks it's a healthy node).
But one thing is different on the new machines and it is a new warning
in the mapreduce logs,

2014-02-04 22:50:39,679 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143

We have changed the OS-installation slighty (internal problems with old image).
I have checked quite a few of the old machines for the same log row
(exit code 143) but no luck. I found exit code 66 and 255 but as
frequent as I see 143.

How come we only see it on new hosts?
Could it be something that is limiting the map reduce java processes?
Or could it be that the newer hosts are behaving much better than the old ones?

Soo many questions... :-)


2014-02-04 22:49:58,817 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
        at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
        at org.mortbay.jetty.Response.sendError(Response.java:240)
        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:4249)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1056)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-02-04 22:50:03,920 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143
2014-02-04 22:50:04,000 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143
2014-02-04 22:50:05,914 WARN org.apache.hadoop.mapred.TaskTracker:
Unknown job job_201401281521_36345 being deleted.
2014-02-04 22:50:10,824 WARN org.apache.hadoop.conf.Configuration:
/data/3/mapred/local/ttprivate/taskTracker/hive/jobcache/job_201401281521_36346/job.xml:an
attempt to override final parameter: mapred.reduce.child.java.opts;
Ignoring.
2014-02-04 22:50:22,342 WARN org.apache.hadoop.mapred.TaskTracker:
Unknown job job_201401281521_36347 being deleted.
2014-02-04 22:50:27,143 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143
2014-02-04 22:50:34,903 WARN org.apache.hadoop.conf.Configuration:
/data/1/mapred/local/ttprivate/taskTracker/hive/jobcache/job_201401281521_36348/job.xml:an
attempt to override final parameter: mapred.reduce.child.java.opts;
Ignoring.
2014-02-04 22:50:39,408 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143
2014-02-04 22:50:39,411 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143
2014-02-04 22:50:39,679 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143
2014-02-04 22:50:51,079 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is
: 143



Thanks,
Jakob