You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Michael Bieniosek (JIRA)" <ji...@apache.org> on 2008/09/03 02:49:44 UTC
[jira] Created: (HBASE-863) deadlock in hbase rest
deadlock in hbase rest
----------------------
Key: HBASE-863
URL: https://issues.apache.org/jira/browse/HBASE-863
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0
Environment: hadoop-0.17.2
Reporter: Michael Bieniosek
Whenever I make any request to the rest server, my http call hangs forever.
In the stacktrace, it looks like hbase waits on a lock it holds:
"SocketListener0-1" prio=10 tid=0x00007fc88c1e2000 nid=0x4860 in Object.wait() [0x0000000041bdd000..0x0000000041bdea00]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:552)
- locked <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:340)
at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:327)
at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:364)
at org.apache.hadoop.hbase.ipc.HbaseRPC.waitForProxy(HbaseRPC.java:302)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:771)
- locked <0x00007fc8ef067618> (a java.util.concurrent.ConcurrentHashMap)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:518)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:478)
- locked <0x00007fc8ef067600> (a java.lang.Integer)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127)
at org.apache.hadoop.hbase.rest.GenericHandler.getTable(GenericHandler.java:260)
at org.apache.hadoop.hbase.rest.TableHandler.doGet(TableHandler.java:74)
at org.apache.hadoop.hbase.rest.Dispatcher.doGet(Dispatcher.java:105)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-863) deadlock in hbase rest
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Bieniosek updated HBASE-863:
------------------------------------
Priority: Blocker (was: Major)
Blocker because it makes hbase rest unusable (hopefully I'm not just running with something misconfigured...)
> deadlock in hbase rest
> ----------------------
>
> Key: HBASE-863
> URL: https://issues.apache.org/jira/browse/HBASE-863
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: hadoop-0.17.2
> Reporter: Michael Bieniosek
> Priority: Blocker
>
> Whenever I make any request to the rest server, my http call hangs forever.
> In the stacktrace, it looks like hbase waits on a lock it holds:
> "SocketListener0-1" prio=10 tid=0x00007fc88c1e2000 nid=0x4860 in Object.wait() [0x0000000041bdd000..0x0000000041bdea00]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:552)
> - locked <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
> at $Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:340)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:327)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:364)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.waitForProxy(HbaseRPC.java:302)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:771)
> - locked <0x00007fc8ef067618> (a java.util.concurrent.ConcurrentHashMap)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:518)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:478)
> - locked <0x00007fc8ef067600> (a java.lang.Integer)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127)
> at org.apache.hadoop.hbase.rest.GenericHandler.getTable(GenericHandler.java:260)
> at org.apache.hadoop.hbase.rest.TableHandler.doGet(TableHandler.java:74)
> at org.apache.hadoop.hbase.rest.Dispatcher.doGet(Dispatcher.java:105)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-863) deadlock in hbase rest
Posted by "sishen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627905#action_12627905 ]
sishen commented on HBASE-863:
------------------------------
Hi, Michael. Possible to give more info?
> deadlock in hbase rest
> ----------------------
>
> Key: HBASE-863
> URL: https://issues.apache.org/jira/browse/HBASE-863
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: hadoop-0.17.2
> Reporter: Michael Bieniosek
> Priority: Blocker
>
> Whenever I make any request to the rest server, my http call hangs forever.
> In the stacktrace, it looks like hbase waits on a lock it holds:
> "SocketListener0-1" prio=10 tid=0x00007fc88c1e2000 nid=0x4860 in Object.wait() [0x0000000041bdd000..0x0000000041bdea00]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:552)
> - locked <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
> at $Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:340)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:327)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:364)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.waitForProxy(HbaseRPC.java:302)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:771)
> - locked <0x00007fc8ef067618> (a java.util.concurrent.ConcurrentHashMap)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:518)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:478)
> - locked <0x00007fc8ef067600> (a java.lang.Integer)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127)
> at org.apache.hadoop.hbase.rest.GenericHandler.getTable(GenericHandler.java:260)
> at org.apache.hadoop.hbase.rest.TableHandler.doGet(TableHandler.java:74)
> at org.apache.hadoop.hbase.rest.Dispatcher.doGet(Dispatcher.java:105)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-863) deadlock in hbase rest
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-863.
-------------------------
Resolution: Invalid
Working with Michael, real issue is HBASE-864.
> deadlock in hbase rest
> ----------------------
>
> Key: HBASE-863
> URL: https://issues.apache.org/jira/browse/HBASE-863
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: hadoop-0.17.2
> Reporter: Michael Bieniosek
> Priority: Blocker
>
> Whenever I make any request to the rest server, my http call hangs forever.
> In the stacktrace, it looks like hbase waits on a lock it holds:
> "SocketListener0-1" prio=10 tid=0x00007fc88c1e2000 nid=0x4860 in Object.wait() [0x0000000041bdd000..0x0000000041bdea00]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:552)
> - locked <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
> at $Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:340)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:327)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:364)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.waitForProxy(HbaseRPC.java:302)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:771)
> - locked <0x00007fc8ef067618> (a java.util.concurrent.ConcurrentHashMap)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:518)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:478)
> - locked <0x00007fc8ef067600> (a java.lang.Integer)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127)
> at org.apache.hadoop.hbase.rest.GenericHandler.getTable(GenericHandler.java:260)
> at org.apache.hadoop.hbase.rest.TableHandler.doGet(TableHandler.java:74)
> at org.apache.hadoop.hbase.rest.Dispatcher.doGet(Dispatcher.java:105)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-863) deadlock in hbase rest
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627902#action_12627902 ]
stack commented on HBASE-863:
-----------------------------
Versions match Michael? You can connect using hbase shell from the server running REST instance (using REST server install)?
> deadlock in hbase rest
> ----------------------
>
> Key: HBASE-863
> URL: https://issues.apache.org/jira/browse/HBASE-863
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: hadoop-0.17.2
> Reporter: Michael Bieniosek
> Priority: Blocker
>
> Whenever I make any request to the rest server, my http call hangs forever.
> In the stacktrace, it looks like hbase waits on a lock it holds:
> "SocketListener0-1" prio=10 tid=0x00007fc88c1e2000 nid=0x4860 in Object.wait() [0x0000000041bdd000..0x0000000041bdea00]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:552)
> - locked <0x00007fc8ef59d700> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
> at $Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:340)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:327)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:364)
> at org.apache.hadoop.hbase.ipc.HbaseRPC.waitForProxy(HbaseRPC.java:302)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:771)
> - locked <0x00007fc8ef067618> (a java.util.concurrent.ConcurrentHashMap)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:518)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:478)
> - locked <0x00007fc8ef067600> (a java.lang.Integer)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127)
> at org.apache.hadoop.hbase.rest.GenericHandler.getTable(GenericHandler.java:260)
> at org.apache.hadoop.hbase.rest.TableHandler.doGet(TableHandler.java:74)
> at org.apache.hadoop.hbase.rest.Dispatcher.doGet(Dispatcher.java:105)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.