[jira] [Created] (HBASE-21428) Performance issue due to userRegionLock in the ConnectionManager.

["koo (JIRA)" <ji...@apache.org>]

koo created HBASE-21428:
---------------------------

             Summary: Performance issue due to userRegionLock in the ConnectionManager.
                 Key: HBASE-21428
                 URL: https://issues.apache.org/jira/browse/HBASE-21428
             Project: HBase
          Issue Type: Bug
    Affects Versions: 1.2.7
            Reporter: koo


My service is that execute a lot of puts using HTableMultiplexer.
After the version change, most of the requests are rejected.

It works fine in 1.2.6.1, but there is a problem in 1.2.7.

This issue is related with the HBASE-19260.

Most of my threads are using a lot of time as below.

 
|"Worker-972" #2479 daemon prio=5 os_prio=0 tid=0x00007f8cea86b000 nid=0x4c8c waiting on condition [0x00007f8b78104000]
 java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x00000005dd703b78> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
 at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1274)
 at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1186)
 at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1170)
 at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1127)
 at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:962)
 at org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:206)
 at org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:150)|

 

When I looked at the issue(HBASE-19260), I recognized the dangerous of to allow accessessing multiple threads.
However, Already create many threads with the limitations
I think it is very inefficient to allow only one thread access.

 
| this.metaLookupPool = getThreadPool(
 conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128),
 conf.getInt("hbase.hconnection.meta.lookup.threads.core", 10),
 "-metaLookup-shared-", new LinkedBlockingQueue<Runnable>());|

 

I want to suggest changing it that allow to have multiple locks.(but not the entire thread)

The following is pseudocode.

 
|int lockSize = conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128) / 2;
BlockingQueue<ReentrantLock> userRegionLockQueue = new LinkedBlockingQueue<ReentrantLock>();
 for (int i=0; i <lockSize; i++) {
 userRegionLockQueue.put(new ReentrantLock());
 }|

 

thanks.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)