You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2007/04/05 14:08:33 UTC
[jira] Commented: (HADOOP-1176) Reduce hang on huge map output
[ https://issues.apache.org/jira/browse/HADOOP-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486934 ]
Devaraj Das commented on HADOOP-1176:
-------------------------------------
We should use standard HTTP header strings like "Content-Length" as opposed to HttpFields.__ContentLength. Secondly, the part of the patch meant for getFile will apply to only the deprecated getFile call.
> Reduce hang on huge map output
> ------------------------------
>
> Key: HADOOP-1176
> URL: https://issues.apache.org/jira/browse/HADOOP-1176
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1176_20070405_1.patch
>
>
> I ran a job with one map and one reduce. The map succeeded, producing 3G output. The reducer hang on fetching the map output. The log shows the following error message:
> WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(task_0401_m_000000_0,0) failed :
> java.net.SocketException: Connection reset
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at org.mortbay.http.ChunkingOutputStream.bypassWrite(ChunkingOutputStream.java:151)
> at org.mortbay.http.BufferedOutputStream.write(BufferedOutputStream.java:139)
> at org.mortbay.http.HttpOutputStream.write(HttpOutputStream.java:423)
> at org.mortbay.jetty.servlet.ServletOut.write(ServletOut.java:54)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1664)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> WARN /: /mapOutput?map=task_0401_m_000000_0&reduce=0:
> java.lang.IllegalStateException: Committed
> at org.mortbay.jetty.servlet.ServletHttpResponse.resetBuffer(ServletHttpResponse.java:212)
> at org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ServletHttpResponse.java:375)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1694)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.