[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

["Lars Hofhansl (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621508#comment-16621508 ] 

Lars Hofhansl commented on HBASE-13082:
---------------------------------------

Interestingly we just upgraded one of our clusters to 1.3.2 and now we see:

"Can't archive compacted file XXXX because of either isCompactedAway = true or file has reference, isReferencedInReads = true, skipping for now."

I looked through the code and it seems impossible to visually track where all we create StoreFileScanners and whether we always close them.

I don't like reference counting... In this case you forget to close just one StoreFileScanner, and BOOM compactions now can *never* succeed;
unless and until you move the region or bounce the region server. Not good!

 

> Coarsen StoreScanner locks to RegionScanner
> -------------------------------------------
>
>                 Key: HBASE-13082
>                 URL: https://issues.apache.org/jira/browse/HBASE-13082
>             Project: HBase
>          Issue Type: Bug
>          Components: Performance, Scanners
>            Reporter: Lars Hofhansl
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Major
>             Fix For: 1.3.0, 2.0.0
>
>         Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 13082-v4.txt, 13082.txt, 13082.txt, CountDownLatch-0.98.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)