[jira] [Commented] (HBASE-15716) HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random read

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316153#comment-15316153 ] 

stack commented on HBASE-15716:
-------------------------------

Interesting observation was hacking out this lock, I ran then into my being blocked responding... 

{code}
"RpcServer.reader=1,bindAddress=ve0528.halxg.cloudera.com,port=16020" #34 daemon prio=5 os_prio=0 tid=0x00007fa76d886800 nid=0x59f0 runnable [0x00007f9f515e9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.NativeThread.current(Native Method)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
        - locked <0x00007fa41f096f40> (a java.lang.Object)
        - locked <0x00007fa41f096f28> (a java.lang.Object)
        at org.apache.hadoop.hbase.ipc.BufferChain.write(BufferChain.java:105)
        at org.apache.hadoop.hbase.ipc.RpcServer.channelWrite(RpcServer.java:2401)
        at org.apache.hadoop.hbase.ipc.RpcServer$Responder.processResponse(RpcServer.java:1072)
        at org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRespond(RpcServer.java:1136)
        at org.apache.hadoop.hbase.ipc.RpcServer$Call.sendResponseIfReady(RpcServer.java:570)
        - locked <0x00007f9fbf7652d0> (a org.apache.hadoop.hbase.ipc.RpcServer$Call)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
        at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:274)
        at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1871)
        at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1762)
        at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1608)
        at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1588)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:838)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:696)
        - locked <0x00007fa06a26acc0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:667)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}


Other notes on this synchronization are that as the throughput goes up, this synchronization becomes more of an obstacle. At rates of hundreds of ops a second, the churn in the CSLM shows... I should be able to do an array of volatiles or something sized by handlers/readers? I should also be able to do something with the fact that readpt is always incrementing... will be back.

> HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random read
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-15716
>                 URL: https://issues.apache.org/jira/browse/HBASE-15716
>             Project: HBase
>          Issue Type: Bug
>          Components: Performance
>            Reporter: stack
>            Assignee: stack
>         Attachments: 15716.prune.synchronizations.patch, 15716.prune.synchronizations.v3.patch, 15716.prune.synchronizations.v4.patch, 15716.prune.synchronizations.v4.patch, 15716.wip.more_to_be_done.patch, Screen Shot 2016-04-26 at 2.05.45 PM.png, Screen Shot 2016-04-26 at 2.06.14 PM.png, Screen Shot 2016-04-26 at 2.07.06 PM.png, Screen Shot 2016-04-26 at 2.25.26 PM.png, Screen Shot 2016-04-26 at 6.02.29 PM.png, Screen Shot 2016-04-27 at 9.49.35 AM.png, current-branch-1.vs.NoSynchronization.vs.Patch.png, hits.png, remove_cslm.patch
>
>
> Here is a [~lhofhansl] special.
> When we construct the region scanner, we get our read point and then store it with the scanner instance in a Region scoped CSLM. This is done under a synchronize on the CSLM.
> This synchronize on a region-scoped Map creating region scanners is the outstanding point of lock contention according to flight recorder (My work load is workload c, random reads).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)