You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2007/03/29 00:10:25 UTC
[jira] Created: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
task Tracker should be restarted if its jetty http server cannot serve get-map-output files
-------------------------------------------------------------------------------------------
Key: HADOOP-1179
URL: https://issues.apache.org/jira/browse/HADOOP-1179
Project: Hadoop
Issue Type: Bug
Reporter: Runping Qi
Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
2007-03-28 12:28:06,642 WARN org.apache.hadoop.mapred.TaskRunner: task_0334_r_00
0379_0 Intermediate Merge of the inmemory files threw an exception: java.lang.Ou
tOfMemoryError
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write
(RawLocalFileSystem.java:166)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
putStream.java:38)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.write(Checksum
FileSystem.java:391)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
putStream.java:38)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$CompressedBytes.writeCompressedByte
s(SequenceFile.java:492)
at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.appendRaw(Sequ
enceFile.java:903)
at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:
2227)
at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(Redu
ceTaskRunner.java:838)
In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
Moreover, all the reduces depending on those map output files are just stuck there.
If the task tracker reports fail to the job tracker, the map/reduce job can recover.
If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-1179:
--------------------------------
Attachment: 1179.patch
I have attached a patch containing the part to do with closing index file as soon as the read is done. Regarding the crc thing, I think what needs to change is the method ChecksumFileSystem.getSumBufferSize, but was not sure whether it would affect the other parts of the code. So didn't touch that part.
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-1179:
--------------------------------
Fix Version/s: 0.12.3
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Fix For: 0.12.3
>
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das reopened HADOOP-1179:
---------------------------------
Assignee: Devaraj Das
Yeah Owen, i totally missed the open-file issues.
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485086 ]
Owen O'Malley commented on HADOOP-1179:
---------------------------------------
Actually, it isn't really a duplicate of HADOOP-1158. Part of what is going on is that the MapOutputServlet is using a lot of memory. In particular, each thread is using:
1. A file to read the index file
2. A file to read the index file crcs
3. A file to read the map outputs
4. A file to read the map output crcs
All 4 files use the io.file.buffer.size, which we usually set to 64k. Since the servlet is lazy about closing them, all 4 files stay open during the entire call. Since we have ~40 threads, that is 10MB of file buffers in the TaskTracker's JVM.
A couple of steps would make this much better:
1. Use smaller buffers for reading crc files (io.file.buffer.size * sizeof(crc) / sizeof(crc block) ?)
2. Close the index file before opening the data files
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-1179.
---------------------------------
Resolution: Duplicate
This is a duplicate of HADOOP-1158
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485370 ]
Koji Noguchi commented on HADOOP-1179:
--------------------------------------
For all the nodes with org.mortbay.jetty.servlet.ServletHandler: java.lang.OutOfMemoryError, they each had one instance of OutOfMemoryError from inmemory file.
2007-03-28 16:30:58,204 WARN org.apache.hadoop.mapred.TaskRunner: task_0334_r_000606_0 Intermediate Merge of the inmemory files threw an exception: java.lang.OutOfMemoryError
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:166)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:38)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.write(ChecksumFileSystem.java:391)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:38)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$CompressedBytes.writeCompressedBytes(SequenceFile.java:492)
at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.appendRaw(SequenceFile.java:903)
at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:2227)
at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(ReduceTaskRunner.java:838)
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1179:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Devaraj!
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Fix For: 0.12.3
>
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485201 ]
Runping Qi commented on HADOOP-1179:
------------------------------------
This issue is related to HADOOP-1158 but not a dup.
Two things need to be investigated/fixed: why out of memory in Jetty (or in TT). Early comments and the attached patch may address that partially. However, I am not sure that accounts all.
The second thing is proper handling of outOfMemory. Normally, when this happens, it may be desirable to kill the TT and hope to be restarted.
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485083 ]
Devaraj Das commented on HADOOP-1179:
-------------------------------------
This is related to HADOOP-1158
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486922 ]
Hadoop QA commented on HADOOP-1179:
-----------------------------------
Integrated in Hadoop-Nightly #48 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/48/)
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Fix For: 0.12.3
>
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Runping Qi updated HADOOP-1179:
-------------------------------
Description:
Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
/mapOutput?map=task_0334_m_013127_0&reduce=591
2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
/mapOutput?map=task_0334_m_013127_0&reduce=591
2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
java.lang.OutOfMemoryError
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
awLocalFileSystem.java:119)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
Stream.java:41)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
cksumFileSystem.java:182)
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
ileSystem.java:167)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
Stream.java:41)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
er.java:1643)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
onHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
67)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
Context.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
Moreover, all the reduces depending on those map output files are just stuck there.
If the task tracker reports fail to the job tracker, the map/reduce job can recover.
If the task tracker restarted, it can continue to join the cluster as a new mamber.
was:
Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
2007-03-28 12:28:06,642 WARN org.apache.hadoop.mapred.TaskRunner: task_0334_r_00
0379_0 Intermediate Merge of the inmemory files threw an exception: java.lang.Ou
tOfMemoryError
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write
(RawLocalFileSystem.java:166)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
putStream.java:38)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.write(Checksum
FileSystem.java:391)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
putStream.java:38)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$CompressedBytes.writeCompressedByte
s(SequenceFile.java:492)
at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.appendRaw(Sequ
enceFile.java:903)
at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:
2227)
at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(Redu
ceTaskRunner.java:838)
In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
Moreover, all the reduces depending on those map output files are just stuck there.
If the task tracker reports fail to the job tracker, the map/reduce job can recover.
If the task tracker restarted, it can continue to join the cluster as a new mamber.
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1179:
----------------------------------
Status: Patch Available (was: Reopened)
+1
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Fix For: 0.12.3
>
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1179) task Tracker should be restarted if
its jetty http server cannot serve get-map-output files
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485262 ]
Doug Cutting commented on HADOOP-1179:
--------------------------------------
Note that running out of file handles can cause OutOfMemoryExceptions, even when there's lots of memory left. So this could just be a file-handle leak. It'd be great to log the heap size somehow (e.g. simply by turning on verbosegc, which has little performance impact) so we had an idea of whether this is really a memory issue or a file handle issue.
> task Tracker should be restarted if its jetty http server cannot serve get-map-output files
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1179
> URL: https://issues.apache.org/jira/browse/HADOOP-1179
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assigned To: Devaraj Das
> Attachments: 1179.patch
>
>
> Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:
> 2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> /mapOutput?map=task_0334_m_013127_0&reduce=591
> 2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
> java.lang.OutOfMemoryError
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
> awLocalFileSystem.java:119)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
> cksumFileSystem.java:182)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
> ileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
> Stream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
> er.java:1643)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
> )
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
> onHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
> 67)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
> Context.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
> 244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
> Moreover, all the reduces depending on those map output files are just stuck there.
> If the task tracker reports fail to the job tracker, the map/reduce job can recover.
> If the task tracker restarted, it can continue to join the cluster as a new mamber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.