You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ramkrishna S Vasudevan <ra...@huawei.com> on 2011/11/02 06:44:04 UTC
client thread waiting infinitely
Hi
I was using 0.90.x version to insert data. At one point suddenly the client
stopped pumping data. The reason was the tableLocations SoftValueSortedMap
in the HConnectionManager got corrupted and the isEmpty check was running
infinitely. But not sure how the internalMap got corrupted. Any idea Ted?
Trying to debug this but not reproducing.
Has any one faced this problem?
"WriterThread-2" prio=10 tid=0x00000000405cb000 nid=0x6a98 runnable
[0x00007fd0c18cf000]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
at
org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:10
9)
at
org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:10
1)
at
java.util.TreeMap.getEntryUsingComparator(TreeMap.java:351)
at java.util.TreeMap.getEntry(TreeMap.java:322)
at java.util.TreeMap.remove(TreeMap.java:580)
at
org.apache.hadoop.hbase.util.SoftValueSortedMap.checkReferences(SoftValueSor
tedMap.java:76)
at
org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.j
ava:150)
- locked <0x00007fd106cd6870> (a
org.apache.hadoop.hbase.util.SoftValueSortedMap)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
getCachedLocation(HConnectionManager.java:826)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
locateRegionInMeta(HConnectionManager.java:668)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
locateRegion(HConnectionManager.java:594)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatch(HConnectionManager.java:1133)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatchOfPuts(HConnectionManager.java:1253)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:828)
at
org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:684)
at
org.apache.hadoop.hbase.client.HTable.put(HTable.java:674)
at
PerRegionLoadData$WriterThread.writeBuffer(PerRegionLoadData.java:350)
at
PerRegionLoadData$WriterThread.doRun(PerRegionLoadData.java:314)
Regards
Ram
Re: client thread waiting infinitely
Posted by Todd Lipcon <to...@cloudera.com>.
I've seen TreeMap infinite loop when it's concurrently modified
without locking. My bet is there's a threading bug.
-Todd
On Wed, Nov 2, 2011 at 7:12 AM, Ramkrishna S Vasudevan
<ra...@huawei.com> wrote:
> I said the Map could be corrupted because i dint view it as a JVM bug :).
>
> When i went through the code found like synchronization was taken care. I
> will dig in more into this. Thanks Stack.
>
> Regards
> Ram
>
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Wednesday, November 02, 2011 7:30 PM
> To: dev@hbase.apache.org; ramakrishnas@huawei.com
> Subject: Re: client thread waiting infinitely
>
> On Tue, Nov 1, 2011 at 11:23 PM, Ramkrishna S Vasudevan
> <ra...@huawei.com> wrote:
>> JVM version is 1.6. There was no profiling done.
>> In the TreeMap remove method we try to compare and then remove. So may be
>> its not able to find the intended key as the map is corrupted and hence
> the
>> comparator is stuck and runs infinitely.
>>
>
> Is it a recent 1.6 JVM? Perhaps a bug in JVM?
>
> Otherwise, can you reproduce?
>
> Why you think the Map 'corrupt'?
>
> St.Ack
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: client thread waiting infinitely
Posted by Stack <st...@duboce.net>.
On Tue, Nov 1, 2011 at 11:23 PM, Ramkrishna S Vasudevan
<ra...@huawei.com> wrote:
> JVM version is 1.6. There was no profiling done.
> In the TreeMap remove method we try to compare and then remove. So may be
> its not able to find the intended key as the map is corrupted and hence the
> comparator is stuck and runs infinitely.
>
Is it a recent 1.6 JVM? Perhaps a bug in JVM?
Otherwise, can you reproduce?
Why you think the Map 'corrupt'?
St.Ack
Re: client thread waiting infinitely
Posted by Stack <st...@duboce.net>.
On Tue, Nov 1, 2011 at 10:44 PM, Ramkrishna S Vasudevan
<ra...@huawei.com> wrote:
> I was using 0.90.x version to insert data. At one point suddenly the client
> stopped pumping data. The reason was the tableLocations SoftValueSortedMap
> in the HConnectionManager got corrupted and the isEmpty check was running
> infinitely. But not sure how the internalMap got corrupted. Any idea Ted?
> Trying to debug this but not reproducing.
>
>
>
> Has any one faced this problem?
>
Is it for sure stuck there? Maybe its stuck in comparator comparing?
What version of JVM? Were you profiling at the time?
St.Ack