You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Viswanathan J <ja...@gmail.com> on 2014/04/12 17:02:28 UTC
Exception in hadoop jobtracker OOM
Hi,
I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
any issues.
Suddenly I got the below error in Jobtracker and jobs are failed to run.
Is this issue in JT or TT or Jetty issue?
2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
org.mortbay.io.nio. SelectorManager$SelectSet.
doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
SelectorManager.doSelect( SelectorManager.java:192) at
org.mortbay.jetty.nio. SelectChannelConnector.accept(
SelectChannelConnector.java:124) at org.mortbay.jetty.
AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
org.mortbay.thread. QueuedThreadPool$PoolThread.
run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
org.apache.hadoop.security. UserGroupInformation:
PriviledgedActionException as:root cause:java.io.IOException:
java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
heap space at java.lang.String.toCharArray( String.java:2725) at
sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
loadClass(ClassLoader.java:306) at sun.misc.Launcher$
AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
org.mortbay.jetty.servlet. ServletHandler$CachedChain.
doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
org.mortbay.jetty.servlet. ServletHandler$CachedChain.
doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
ServletHandler.handle(ServletHandler.java:399) at
org.mortbay.jetty.security. SecurityHandler.handle(
SecurityHandler.java:216) at org.mortbay.jetty.servlet.
SessionHandler.handle( SessionHandler.java:182) at
org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
HandlerWrapper.handle( HandlerWrapper.java:152) at
org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
HttpConnection.handleRequest( HttpConnection.java:542) at
org.mortbay.jetty. HttpConnection$RequestHandler.
headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
parseAvailable(HttpParser.java:212) at
org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
org.mortbay.io.nio. SelectChannelEndPoint.run(
SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
canonical subfolder. java.lang.OutOfMemoryError: Java heap space
Please help as I'm facing in production.
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
Sorry for the error. Did not have a proper look at the logs. This seems to
be a JT issue. Ignore the previous email.
Thanks
Divye Sheth
On Apr 14, 2014 6:06 PM, "divye sheth" <di...@gmail.com> wrote:
> This usually occurs when the task takes more memory and exceeds its heap
> space. You can increase the memory of the tasks by setting a property in
> mapred-site.xml
>
> Mapred.child.opts specify the -Xmx to a higher value.
>
> P.S the property name might not be correct, you may look up the
> mapred-default.xml for the exact property name.
>
> Sent from my phone.
>
> Thanks
> Divye Sheth
> On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
>> any issues.
>>
>> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>>
>> Is this issue in JT or TT or Jetty issue?
>>
>> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
>> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
>> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
>> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
>> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
>> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
>> org.mortbay.io.nio. SelectorManager$SelectSet.
>> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
>> SelectorManager.doSelect( SelectorManager.java:192) at
>> org.mortbay.jetty.nio. SelectChannelConnector.accept(
>> SelectChannelConnector.java:124) at org.mortbay.jetty.
>> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
>> org.mortbay.thread. QueuedThreadPool$PoolThread.
>> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
>> org.apache.hadoop.security. UserGroupInformation:
>> PriviledgedActionException as:root cause:java.io.IOException:
>> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
>> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
>> heap space at java.lang.String.toCharArray( String.java:2725) at
>> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
>> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
>> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
>> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
>> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
>> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
>> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
>> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
>> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
>> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
>> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
>> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
>> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
>> ServletHandler.handle(ServletHandler.java:399) at
>> org.mortbay.jetty.security. SecurityHandler.handle(
>> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
>> SessionHandler.handle( SessionHandler.java:182) at
>> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
>> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
>> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
>> HandlerWrapper.handle( HandlerWrapper.java:152) at
>> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
>> HttpConnection.handleRequest( HttpConnection.java:542) at
>> org.mortbay.jetty. HttpConnection$RequestHandler.
>> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
>> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
>> parseAvailable(HttpParser.java:212) at
>> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
>> org.mortbay.io.nio. SelectChannelEndPoint.run(
>> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
>> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
>> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>>
>> Please help as I'm facing in production.
>>
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
Sorry for the error. Did not have a proper look at the logs. This seems to
be a JT issue. Ignore the previous email.
Thanks
Divye Sheth
On Apr 14, 2014 6:06 PM, "divye sheth" <di...@gmail.com> wrote:
> This usually occurs when the task takes more memory and exceeds its heap
> space. You can increase the memory of the tasks by setting a property in
> mapred-site.xml
>
> Mapred.child.opts specify the -Xmx to a higher value.
>
> P.S the property name might not be correct, you may look up the
> mapred-default.xml for the exact property name.
>
> Sent from my phone.
>
> Thanks
> Divye Sheth
> On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
>> any issues.
>>
>> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>>
>> Is this issue in JT or TT or Jetty issue?
>>
>> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
>> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
>> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
>> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
>> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
>> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
>> org.mortbay.io.nio. SelectorManager$SelectSet.
>> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
>> SelectorManager.doSelect( SelectorManager.java:192) at
>> org.mortbay.jetty.nio. SelectChannelConnector.accept(
>> SelectChannelConnector.java:124) at org.mortbay.jetty.
>> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
>> org.mortbay.thread. QueuedThreadPool$PoolThread.
>> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
>> org.apache.hadoop.security. UserGroupInformation:
>> PriviledgedActionException as:root cause:java.io.IOException:
>> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
>> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
>> heap space at java.lang.String.toCharArray( String.java:2725) at
>> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
>> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
>> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
>> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
>> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
>> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
>> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
>> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
>> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
>> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
>> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
>> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
>> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
>> ServletHandler.handle(ServletHandler.java:399) at
>> org.mortbay.jetty.security. SecurityHandler.handle(
>> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
>> SessionHandler.handle( SessionHandler.java:182) at
>> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
>> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
>> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
>> HandlerWrapper.handle( HandlerWrapper.java:152) at
>> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
>> HttpConnection.handleRequest( HttpConnection.java:542) at
>> org.mortbay.jetty. HttpConnection$RequestHandler.
>> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
>> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
>> parseAvailable(HttpParser.java:212) at
>> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
>> org.mortbay.io.nio. SelectChannelEndPoint.run(
>> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
>> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
>> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>>
>> Please help as I'm facing in production.
>>
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
Sorry for the error. Did not have a proper look at the logs. This seems to
be a JT issue. Ignore the previous email.
Thanks
Divye Sheth
On Apr 14, 2014 6:06 PM, "divye sheth" <di...@gmail.com> wrote:
> This usually occurs when the task takes more memory and exceeds its heap
> space. You can increase the memory of the tasks by setting a property in
> mapred-site.xml
>
> Mapred.child.opts specify the -Xmx to a higher value.
>
> P.S the property name might not be correct, you may look up the
> mapred-default.xml for the exact property name.
>
> Sent from my phone.
>
> Thanks
> Divye Sheth
> On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
>> any issues.
>>
>> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>>
>> Is this issue in JT or TT or Jetty issue?
>>
>> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
>> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
>> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
>> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
>> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
>> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
>> org.mortbay.io.nio. SelectorManager$SelectSet.
>> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
>> SelectorManager.doSelect( SelectorManager.java:192) at
>> org.mortbay.jetty.nio. SelectChannelConnector.accept(
>> SelectChannelConnector.java:124) at org.mortbay.jetty.
>> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
>> org.mortbay.thread. QueuedThreadPool$PoolThread.
>> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
>> org.apache.hadoop.security. UserGroupInformation:
>> PriviledgedActionException as:root cause:java.io.IOException:
>> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
>> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
>> heap space at java.lang.String.toCharArray( String.java:2725) at
>> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
>> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
>> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
>> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
>> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
>> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
>> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
>> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
>> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
>> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
>> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
>> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
>> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
>> ServletHandler.handle(ServletHandler.java:399) at
>> org.mortbay.jetty.security. SecurityHandler.handle(
>> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
>> SessionHandler.handle( SessionHandler.java:182) at
>> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
>> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
>> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
>> HandlerWrapper.handle( HandlerWrapper.java:152) at
>> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
>> HttpConnection.handleRequest( HttpConnection.java:542) at
>> org.mortbay.jetty. HttpConnection$RequestHandler.
>> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
>> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
>> parseAvailable(HttpParser.java:212) at
>> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
>> org.mortbay.io.nio. SelectChannelEndPoint.run(
>> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
>> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
>> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>>
>> Please help as I'm facing in production.
>>
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
Sorry for the error. Did not have a proper look at the logs. This seems to
be a JT issue. Ignore the previous email.
Thanks
Divye Sheth
On Apr 14, 2014 6:06 PM, "divye sheth" <di...@gmail.com> wrote:
> This usually occurs when the task takes more memory and exceeds its heap
> space. You can increase the memory of the tasks by setting a property in
> mapred-site.xml
>
> Mapred.child.opts specify the -Xmx to a higher value.
>
> P.S the property name might not be correct, you may look up the
> mapred-default.xml for the exact property name.
>
> Sent from my phone.
>
> Thanks
> Divye Sheth
> On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
>> any issues.
>>
>> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>>
>> Is this issue in JT or TT or Jetty issue?
>>
>> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
>> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
>> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
>> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
>> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
>> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
>> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
>> org.mortbay.io.nio. SelectorManager$SelectSet.
>> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
>> SelectorManager.doSelect( SelectorManager.java:192) at
>> org.mortbay.jetty.nio. SelectChannelConnector.accept(
>> SelectChannelConnector.java:124) at org.mortbay.jetty.
>> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
>> org.mortbay.thread. QueuedThreadPool$PoolThread.
>> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
>> org.apache.hadoop.security. UserGroupInformation:
>> PriviledgedActionException as:root cause:java.io.IOException:
>> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
>> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
>> heap space at java.lang.String.toCharArray( String.java:2725) at
>> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
>> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
>> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
>> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
>> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
>> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
>> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
>> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
>> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
>> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
>> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
>> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
>> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
>> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
>> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
>> ServletHandler.handle(ServletHandler.java:399) at
>> org.mortbay.jetty.security. SecurityHandler.handle(
>> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
>> SessionHandler.handle( SessionHandler.java:182) at
>> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
>> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
>> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
>> HandlerWrapper.handle( HandlerWrapper.java:152) at
>> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
>> HttpConnection.handleRequest( HttpConnection.java:542) at
>> org.mortbay.jetty. HttpConnection$RequestHandler.
>> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
>> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
>> parseAvailable(HttpParser.java:212) at
>> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
>> org.mortbay.io.nio. SelectChannelEndPoint.run(
>> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
>> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
>> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>>
>> Please help as I'm facing in production.
>>
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
This usually occurs when the task takes more memory and exceeds its heap
space. You can increase the memory of the tasks by setting a property in
mapred-site.xml
Mapred.child.opts specify the -Xmx to a higher value.
P.S the property name might not be correct, you may look up the
mapred-default.xml for the exact property name.
Sent from my phone.
Thanks
Divye Sheth
On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com> wrote:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by dingweiqiong <di...@gmail.com>.
unsubscribe
2014-04-12 23:02 GMT+08:00 Viswanathan J <ja...@gmail.com>:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
This usually occurs when the task takes more memory and exceeds its heap
space. You can increase the memory of the tasks by setting a property in
mapred-site.xml
Mapred.child.opts specify the -Xmx to a higher value.
P.S the property name might not be correct, you may look up the
mapred-default.xml for the exact property name.
Sent from my phone.
Thanks
Divye Sheth
On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com> wrote:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
This usually occurs when the task takes more memory and exceeds its heap
space. You can increase the memory of the tasks by setting a property in
mapred-site.xml
Mapred.child.opts specify the -Xmx to a higher value.
P.S the property name might not be correct, you may look up the
mapred-default.xml for the exact property name.
Sent from my phone.
Thanks
Divye Sheth
On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com> wrote:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by divye sheth <di...@gmail.com>.
This usually occurs when the task takes more memory and exceeds its heap
space. You can increase the memory of the tasks by setting a property in
mapred-site.xml
Mapred.child.opts specify the -Xmx to a higher value.
P.S the property name might not be correct, you may look up the
mapred-default.xml for the exact property name.
Sent from my phone.
Thanks
Divye Sheth
On Apr 12, 2014 8:33 PM, "Viswanathan J" <ja...@gmail.com> wrote:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by dingweiqiong <di...@gmail.com>.
unsubscribe
2014-04-12 23:02 GMT+08:00 Viswanathan J <ja...@gmail.com>:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by dingweiqiong <di...@gmail.com>.
unsubscribe
2014-04-12 23:02 GMT+08:00 Viswanathan J <ja...@gmail.com>:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>
Re: Exception in hadoop jobtracker OOM
Posted by dingweiqiong <di...@gmail.com>.
unsubscribe
2014-04-12 23:02 GMT+08:00 Viswanathan J <ja...@gmail.com>:
> Hi,
>
> I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
> any issues.
>
> Suddenly I got the below error in Jobtracker and jobs are failed to run.
>
> Is this issue in JT or TT or Jetty issue?
>
> 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
> newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
> iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
> at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
> at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
> sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
> org.mortbay.io.nio. SelectorManager$SelectSet.
> doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
> SelectorManager.doSelect( SelectorManager.java:192) at
> org.mortbay.jetty.nio. SelectChannelConnector.accept(
> SelectChannelConnector.java:124) at org.mortbay.jetty.
> AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
> org.mortbay.thread. QueuedThreadPool$PoolThread.
> run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
> org.apache.hadoop.security. UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
> org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
> heap space at java.lang.String.toCharArray( String.java:2725) at
> sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
> sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
> sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
> sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
> java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
> AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
> findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:306) at sun.misc.Launcher$
> AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
> loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
> jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
> org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
> javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
> org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
> HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
> org.mortbay.jetty.servlet. ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
> ServletHandler.handle(ServletHandler.java:399) at
> org.mortbay.jetty.security. SecurityHandler.handle(
> SecurityHandler.java:216) at org.mortbay.jetty.servlet.
> SessionHandler.handle( SessionHandler.java:182) at
> org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
> at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
> at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
> ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
> HandlerWrapper.handle( HandlerWrapper.java:152) at
> org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
> HttpConnection.handleRequest( HttpConnection.java:542) at
> org.mortbay.jetty. HttpConnection$RequestHandler.
> headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
> parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
> parseAvailable(HttpParser.java:212) at
> org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
> org.mortbay.io.nio. SelectChannelEndPoint.run(
> SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
> org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
> canonical subfolder. java.lang.OutOfMemoryError: Java heap space
>
> Please help as I'm facing in production.
>