You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ramkrishna S Vasudevan <ra...@huawei.com> on 2011/11/02 06:44:04 UTC

client thread waiting infinitely

Hi 

 

I was using 0.90.x version to insert data.  At one point suddenly the client
stopped pumping data.  The reason was the tableLocations SoftValueSortedMap
in the HConnectionManager got corrupted and the isEmpty check was running
infinitely.  But not sure how the internalMap got corrupted.  Any idea Ted?
Trying to debug this but not reproducing.    

 

Has any one faced this problem?


 


"WriterThread-2" prio=10 tid=0x00000000405cb000 nid=0x6a98 runnable
[0x00007fd0c18cf000]
  java.lang.Thread.State: RUNNABLE
                at
org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
                at
org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:10
9)
                at
org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:10
1)
                at
java.util.TreeMap.getEntryUsingComparator(TreeMap.java:351)
                at java.util.TreeMap.getEntry(TreeMap.java:322)
                at java.util.TreeMap.remove(TreeMap.java:580)
                at
org.apache.hadoop.hbase.util.SoftValueSortedMap.checkReferences(SoftValueSor
tedMap.java:76)
                at
org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.j
ava:150)
                - locked <0x00007fd106cd6870> (a
org.apache.hadoop.hbase.util.SoftValueSortedMap)
                at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
getCachedLocation(HConnectionManager.java:826)
                at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
locateRegionInMeta(HConnectionManager.java:668)
                at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
locateRegion(HConnectionManager.java:594)
                at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatch(HConnectionManager.java:1133)
                at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatchOfPuts(HConnectionManager.java:1253)
                at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:828)
                at
org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:684)
                at
org.apache.hadoop.hbase.client.HTable.put(HTable.java:674)
                at
PerRegionLoadData$WriterThread.writeBuffer(PerRegionLoadData.java:350)
                at
PerRegionLoadData$WriterThread.doRun(PerRegionLoadData.java:314)

 

Regards

Ram


Re: client thread waiting infinitely

Posted by Todd Lipcon <to...@cloudera.com>.
I've seen TreeMap infinite loop when it's concurrently modified
without locking. My bet is there's a threading bug.

-Todd

On Wed, Nov 2, 2011 at 7:12 AM, Ramkrishna S Vasudevan
<ra...@huawei.com> wrote:
> I said the Map could be corrupted because i dint view it as a JVM bug :).
>
> When i went through the code found like synchronization was taken care.  I
> will dig in more into this. Thanks Stack.
>
> Regards
> Ram
>
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Wednesday, November 02, 2011 7:30 PM
> To: dev@hbase.apache.org; ramakrishnas@huawei.com
> Subject: Re: client thread waiting infinitely
>
> On Tue, Nov 1, 2011 at 11:23 PM, Ramkrishna S Vasudevan
> <ra...@huawei.com> wrote:
>> JVM version is 1.6.  There was no profiling done.
>> In the TreeMap remove method we try to compare and then remove.  So may be
>> its not able to find the intended key as the map is corrupted and hence
> the
>> comparator is stuck and runs infinitely.
>>
>
> Is it a recent 1.6 JVM?  Perhaps a bug in JVM?
>
> Otherwise, can you reproduce?
>
> Why you think the Map 'corrupt'?
>
> St.Ack
>
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: client thread waiting infinitely

Posted by Stack <st...@duboce.net>.
On Tue, Nov 1, 2011 at 11:23 PM, Ramkrishna S Vasudevan
<ra...@huawei.com> wrote:
> JVM version is 1.6.  There was no profiling done.
> In the TreeMap remove method we try to compare and then remove.  So may be
> its not able to find the intended key as the map is corrupted and hence the
> comparator is stuck and runs infinitely.
>

Is it a recent 1.6 JVM?  Perhaps a bug in JVM?

Otherwise, can you reproduce?

Why you think the Map 'corrupt'?

St.Ack

Re: client thread waiting infinitely

Posted by Stack <st...@duboce.net>.
On Tue, Nov 1, 2011 at 10:44 PM, Ramkrishna S Vasudevan
<ra...@huawei.com> wrote:
> I was using 0.90.x version to insert data.  At one point suddenly the client
> stopped pumping data.  The reason was the tableLocations SoftValueSortedMap
> in the HConnectionManager got corrupted and the isEmpty check was running
> infinitely.  But not sure how the internalMap got corrupted.  Any idea Ted?
> Trying to debug this but not reproducing.
>
>
>
> Has any one faced this problem?
>

Is it for sure stuck there?  Maybe its stuck in comparator comparing?

What version of JVM?  Were you profiling at the time?
St.Ack