You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "xieguiming (JIRA)" <ji...@apache.org> on 2012/05/27 08:56:24 UTC
[jira] [Commented] (MAPREDUCE-5) Shuffle's getMapOutput() fails
with EofException, followed by IllegalStateException
[ https://issues.apache.org/jira/browse/MAPREDUCE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284114#comment-13284114 ]
xieguiming commented on MAPREDUCE-5:
------------------------------------
I also met the same exception, and so many. and at last the TT port(50060), 83 connections are in CLOSE_WAIT state. and can not accept any connect.
2012-05-17 15:52:42,513 INFO TaskTracker.clienttrace (TaskTracker.java:doGet(3211)) - src: 172.16.4.7:10060, dest: 172.16.4.8:47138, bytes: 458752, op: MAPRED_SHUFFLE, cliID: attempt_201205061050_13074_m_000179_0
2012-05-17 15:52:42,514 ERROR mortbay.log (Slf4jLog.java:warn(87)) - /mapOutput
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
at org.mortbay.jetty.Response.sendError(Response.java:240)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3202)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-05-17 15:52:42,510 WARN mapred.TaskTracker (TaskTracker.java:doGet(3198)) - getMapOutput(attempt_201205061050_13076_m_000055_0,4) failed :
org.mortbay.jetty.EofException: timeout
at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3179)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
> -----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Sun Java 1.6.0_13, OpenSolaris, running on a SunFire 4150 (x64) 10 node cluster
> Reporter: George Porter
>
> During the shuffle phase, I'm seeing a large sequence of the following actions:
> 1) WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_200905181452_0002_m_000010_0,0) failed : org.mortbay.jetty.EofException
> 2) WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_200905181452_0002_m_000010_0,0) failed : org.mortbay.jetty.EofException
> 3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException: Committed
> The map phase completes with 100%, and then the reduce phase crawls along with the above errors in each of the TaskTracker logs. None of the tasktrackers get lost. When I run non-data jobs like the 'pi' test from the example jar, everything works fine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira