[jira] [Comment Edited] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

["ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963502#comment-14963502 ] 

ramkrishna.s.vasudevan edited comment on HBASE-13082 at 10/19/15 3:58 PM:
--------------------------------------------------------------------------

bq.What about flush? Once the flush is finished, we will have to clear the snapshot of cells in Memstore no? Then we will have to open this newly added file for reading.. If we dont open it, we can not release the memstore snapshot. 
We do open the reader for sure. But one thing may be to check is that if we open but still the current scanner heap is not reset will it have any impact on the current scan is what needs to be checked?  Because during flush the reads can still be served from the snapshot. Only after flush is a point to be noted. 
bq.This status is in reader? It suits better in StoreFile no?
The reader is the common object here.  Hence was referencing it from there. Initially had it in storefile only but felt that StoreFile is more volatile. Let me check. Can be done.  Infact I was also not very sure of having the state in the reader. 
We can change the states no problem.
bq.This is a new Chore thread adding to the system.. Mention about it clearly. It will be one thread per RS. Also what is its interval? Is it configurable? How?
Forgot to add that config details. It is per store and the chore is started by the HStore and not the RS.  It can be configured may be once in 2 mins or so?  currently in patch it is once in 5 min.
bq.Hence it requires us to add new APIs which ensures that a split can go ahead even if reference files are present but they are in the COMPACTED state
This is bit of sticky area.  In case of merge I have tried to forcefully clear the compacted files. May be we need to do the same with split also. But in split do we explicitly call compact?  I was not pretty sure on that. But in merge we do. 




was (Author: ram_krish):
bq.What about flush? Once the flush is finished, we will have to clear the snapshot of cells in Memstore no? Then we will have to open this newly added file for reading.. If we dont open it, we can not release the memstore snapshot. 
We do open the reader for sure. But one thing may be to check is that if we open but still the current scanner heap is not reset will it have any impact on the current scan is what needs to be checked?  Because during flush the reads can still be served from the snapshot. Only after flush is a point to be noted. 
bq.This status is in reader? It suits better in StoreFile no?
The reader is the common object here.  Hence was referencing it from there. Initially had it in storefile only but felt that StoreFile is more volatile. Let me check. Can be done.  Infact I was also not very sure of having the state in the reader. 
We can change the states no problem.
bq.This is a new Chore thread adding to the system.. Mention about it clearly. It will be one thread per RS. Also what is its interval? Is it configurable? How?
For to add that config details. It is per store and the chore is started by the HStore and not the RS. 
bq.Hence it requires us to add new APIs which ensures that a split can go ahead even if reference files are present but they are in the COMPACTED state
This is bit of sticky area.  In case of merge I have tried to forcefully clear the compacted files. May be we need to do the same with split also. But in split do we explicitly call compact?  I was not pretty sure on that. But in merge we do. 



> Coarsen StoreScanner locks to RegionScanner
> -------------------------------------------
>
>                 Key: HBASE-13082
>                 URL: https://issues.apache.org/jira/browse/HBASE-13082
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: ramkrishna.s.vasudevan
>         Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)