You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2013/09/20 23:52:56 UTC
[jira] [Updated] (HBASE-9602) Cannot show the master's web UI when
splitting logs at start time
[ https://issues.apache.org/jira/browse/HBASE-9602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-9602:
--------------------------------------
Attachment: HBASE-9602.jstack.rtf
It's actually worse than that, everytime you refresh you occupy one more handler and eventually they all get full, and log splitting *stops*. I attached my full jstack.
> Cannot show the master's web UI when splitting logs at start time
> -----------------------------------------------------------------
>
> Key: HBASE-9602
> URL: https://issues.apache.org/jira/browse/HBASE-9602
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.96.0
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Priority: Critical
> Fix For: 0.98.0, 0.96.0
>
> Attachments: HBASE-9602.jstack.rtf
>
>
> It looks like we cannot show the master's web ui at start time when there are logs to split because we can't reach the namespace regions.
> So it means that you can't see how things are progressing without tailing the log while waiting on your cluster to boot up. This wasn't the case in 0.94
> See this jstack:
> {noformat}
> "606214580@qtp-2001431298-3" prio=10 tid=0x00007f6ac8040000 nid=0x7b1 in Object.wait() [0x00007f6aa82bf000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000bc0c1460> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1416)
> - locked <0x00000000bc0c1460> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
> at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
> at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
> at org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$BlockingStub.listTableDescriptorsByNamespace(MasterAdminProtos.java:35031)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.listTableDescriptorsByNamespace(HConnectionManager.java:2181)
> at org.apache.hadoop.hbase.client.HBaseAdmin$22.call(HBaseAdmin.java:2265)
> at org.apache.hadoop.hbase.client.HBaseAdmin$22.call(HBaseAdmin.java:2262)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
> - locked <0x00000000c09baf20> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
> - locked <0x00000000c09baf20> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3155)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptorsByNamespace(HBaseAdmin.java:2261)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__catalogTables(MasterStatusTmplImpl.java:461)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:270)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
> at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:95)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:850)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> ...
> "RpcServer.handler=28,port=60000" daemon prio=10 tid=0x00007f6ad08a5800 nid=0x77e waiting on condition [0x00007f6aa97d5000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:148)
> - locked <0x00000000c0909178> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:760)
> at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:171)
> at org.apache.hadoop.hbase.master.TableNamespaceManager.getNamespaceTable(TableNamespaceManager.java:119)
> - locked <0x00000000c1958890> (a org.apache.hadoop.hbase.master.TableNamespaceManager)
> at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:155)
> - locked <0x00000000c1958890> (a org.apache.hadoop.hbase.master.TableNamespaceManager)
> at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3088)
> at org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3102)
> at org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3012)
> at org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32908)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira