You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Joel Halbert <jo...@su3analytics.com> on 2011/10/07 10:31:32 UTC

HBaseClient call stuck waiting

Hi,

Intermittently I am seeing my client threads stuck waiting on a call to
HBaseClient.call(HBaseClient.java:722)

This is a stand-along instance of HBase (no Hadoop).  The client
processes that use it are multi-threaded, and I am using a HTablePool to
get HTable instances.

Has anyone seen this before? Any ideas what could be the cause?

A thread dump of a client process shows:

e.g. 
UPDATE_CLIENT_SEEDS" prio=10 tid=0x09fbcc00 nid=0x34e0 in Object.wait()
[0x0ae5a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
        - locked <0x30bdd308> (a org.apache.hadoop.hbase.ipc.HBaseClient
$Call)
        at org.apache.hadoop.hbase.ipc.HBaseRPC
$Invoker.invoke(HBaseRPC.java:333)
        at $Proxy0.get(Unknown Source)
        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
        at org.apache.hadoop.hbase.client.HConnectionManager
$TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
    
    
"R-PPQ-6" daemon prio=10 tid=0x0738e000 nid=0x3510 in Object.wait()
[0x04ac4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
        - locked <0x155328e8> (a org.apache.hadoop.hbase.ipc.HBaseClient
$Call)
        at org.apache.hadoop.hbase.ipc.HBaseRPC
$Invoker.invoke(HBaseRPC.java:333)
        at $Proxy0.get(Unknown Source)
        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
        at org.apache.hadoop.hbase.client.HConnectionManager
$TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
        at
com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:111)
        at
com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:256)


"R-PPQ-9" daemon prio=10 tid=0x09ded400 nid=0x354a in Object.wait()
[0x019fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
        - locked <0x1576d880> (a org.apache.hadoop.hbase.ipc.HBaseClient
$Call)
        at org.apache.hadoop.hbase.ipc.HBaseRPC
$Invoker.invoke(HBaseRPC.java:333)
        at $Proxy0.get(Unknown Source)
        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
        at org.apache.hadoop.hbase.client.HConnectionManager
$TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)

Re: HBaseClient call stuck waiting

Posted by Joel Halbert <jo...@su3analytics.com>.

Thanks Vinod, I'll try as you suggest next time the problem occurs and
report back.

On Fri, 2011-10-07 at 01:36 -0700, Vinod Gupta Tankala wrote:
> i haven't seen this before but what i can suggest you is these 2 steps and
> you will probably get more insights -
> 1)
> # bin/hbase shell
> # list
> 
> see if the list commands returns back with your list of tables.
> 
> 2) take a look at the latest hbase logs and see if there are any FATAL,
> ERROR or WARN logs lately.
> 
> thanks
> 
> On Fri, Oct 7, 2011 at 1:31 AM, Joel Halbert <jo...@su3analytics.com> wrote:
> 
> > Hi,
> >
> > Intermittently I am seeing my client threads stuck waiting on a call to
> > HBaseClient.call(HBaseClient.java:722)
> >
> > This is a stand-along instance of HBase (no Hadoop).  The client
> > processes that use it are multi-threaded, and I am using a HTablePool to
> > get HTable instances.
> >
> >  Has anyone seen this before? Any ideas what could be the cause?
> >
> > A thread dump of a client process shows:
> >
> > e.g.
> > UPDATE_CLIENT_SEEDS" prio=10 tid=0x09fbcc00 nid=0x34e0 in Object.wait()
> > [0x0ae5a000]
> >   java.lang.Thread.State: WAITING (on object monitor)
> >        at java.lang.Object.wait(Native Method)
> >        at java.lang.Object.wait(Object.java:485)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
> >        - locked <0x30bdd308> (a org.apache.hadoop.hbase.ipc.HBaseClient
> > $Call)
> >        at org.apache.hadoop.hbase.ipc.HBaseRPC
> > $Invoker.invoke(HBaseRPC.java:333)
> >        at $Proxy0.get(Unknown Source)
> >        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
> >        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
> >        at org.apache.hadoop.hbase.client.HConnectionManager
> > $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
> >        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
> >
> >
> > "R-PPQ-6" daemon prio=10 tid=0x0738e000 nid=0x3510 in Object.wait()
> > [0x04ac4000]
> >   java.lang.Thread.State: WAITING (on object monitor)
> >        at java.lang.Object.wait(Native Method)
> >        at java.lang.Object.wait(Object.java:485)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
> >        - locked <0x155328e8> (a org.apache.hadoop.hbase.ipc.HBaseClient
> > $Call)
> >        at org.apache.hadoop.hbase.ipc.HBaseRPC
> > $Invoker.invoke(HBaseRPC.java:333)
> >        at $Proxy0.get(Unknown Source)
> >        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
> >        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
> >        at org.apache.hadoop.hbase.client.HConnectionManager
> > $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
> >        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
> >        at
> > com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:111)
> >        at
> > com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:256)
> >
> >
> > "R-PPQ-9" daemon prio=10 tid=0x09ded400 nid=0x354a in Object.wait()
> > [0x019fe000]
> >   java.lang.Thread.State: WAITING (on object monitor)
> >        at java.lang.Object.wait(Native Method)
> >        at java.lang.Object.wait(Object.java:485)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
> >        - locked <0x1576d880> (a org.apache.hadoop.hbase.ipc.HBaseClient
> > $Call)
> >        at org.apache.hadoop.hbase.ipc.HBaseRPC
> > $Invoker.invoke(HBaseRPC.java:333)
> >        at $Proxy0.get(Unknown Source)
> >        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
> >        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
> >        at org.apache.hadoop.hbase.client.HConnectionManager
> > $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
> >        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
> >
> >
> >
> >
> >

Re: HBaseClient call stuck waiting

Posted by Vinod Gupta Tankala <tv...@readypulse.com>.

i haven't seen this before but what i can suggest you is these 2 steps and
you will probably get more insights -
1)
# bin/hbase shell
# list

see if the list commands returns back with your list of tables.

2) take a look at the latest hbase logs and see if there are any FATAL,
ERROR or WARN logs lately.

thanks

On Fri, Oct 7, 2011 at 1:31 AM, Joel Halbert <jo...@su3analytics.com> wrote:

> Hi,
>
> Intermittently I am seeing my client threads stuck waiting on a call to
> HBaseClient.call(HBaseClient.java:722)
>
> This is a stand-along instance of HBase (no Hadoop).  The client
> processes that use it are multi-threaded, and I am using a HTablePool to
> get HTable instances.
>
>  Has anyone seen this before? Any ideas what could be the cause?
>
> A thread dump of a client process shows:
>
> e.g.
> UPDATE_CLIENT_SEEDS" prio=10 tid=0x09fbcc00 nid=0x34e0 in Object.wait()
> [0x0ae5a000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        at java.lang.Object.wait(Object.java:485)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
>        - locked <0x30bdd308> (a org.apache.hadoop.hbase.ipc.HBaseClient
> $Call)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC
> $Invoker.invoke(HBaseRPC.java:333)
>        at $Proxy0.get(Unknown Source)
>        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
>        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
>        at org.apache.hadoop.hbase.client.HConnectionManager
> $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
>        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
>
>
> "R-PPQ-6" daemon prio=10 tid=0x0738e000 nid=0x3510 in Object.wait()
> [0x04ac4000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        at java.lang.Object.wait(Object.java:485)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
>        - locked <0x155328e8> (a org.apache.hadoop.hbase.ipc.HBaseClient
> $Call)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC
> $Invoker.invoke(HBaseRPC.java:333)
>        at $Proxy0.get(Unknown Source)
>        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
>        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
>        at org.apache.hadoop.hbase.client.HConnectionManager
> $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
>        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
>        at
> com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:111)
>        at
> com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:256)
>
>
> "R-PPQ-9" daemon prio=10 tid=0x09ded400 nid=0x354a in Object.wait()
> [0x019fe000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        at java.lang.Object.wait(Object.java:485)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722)
>        - locked <0x1576d880> (a org.apache.hadoop.hbase.ipc.HBaseClient
> $Call)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC
> $Invoker.invoke(HBaseRPC.java:333)
>        at $Proxy0.get(Unknown Source)
>        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450)
>        at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448)
>        at org.apache.hadoop.hbase.client.HConnectionManager
> $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040)
>        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447)
>
>
>
>
>