You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by bijieshan <bi...@huawei.com> on 2011/12/21 11:48:17 UTC
A concurrency issue on SoftValueSortedMap?

Hi all,

We use thousand of threads doing the scan operations. One thread got blocked at the position of "java.util.TreeMap.fixAfterDeletion". It can't come out of the loop of "TreeMap.fixAfterDeletion":

So I think it maybe a concurrency issue. Has someone encountered this?
Thank you.

Jieshan.

Here's the thread dump:

"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable [0x00007f3d05c1b000]
   java.lang.Thread.State: RUNNABLE
                at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
                at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
                at java.util.TreeMap.remove(TreeMap.java:585)
                at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
                at org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
                - locked <0x00007f3d92937748> (a org.apache.hadoop.hbase.util.SoftValueSortedMap)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
                at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
                at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
                at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
                at com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
                at com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
                at framework.QueryThread.run(QueryThread.java:47)

"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable [0x00007f3d05c1a000]
   java.lang.Thread.State: RUNNABLE
                at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
                at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
                at java.util.TreeMap.remove(TreeMap.java:585)
                at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
                at org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
                - locked <0x00007f3d94f24f70> (a org.apache.hadoop.hbase.util.SoftValueSortedMap)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
                at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
                at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
                at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
                at com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
                at com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
                at framework.QueryThread.run(QueryThread.java:47)