[jira] Issue Comment Edited: (HBASE-823) Concurrent "open mapfile reader" limit

["Luo Ning (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671562#action_12671562 ] 

ln@webcate.net edited comment on HBASE-823 at 2/7/09 10:18 PM:
---------------------------------------------------------

i think we need archiving 2 goals for the 'open reader limit':
1. fit xceiverCount limitation of hadoop, see HBASE-24
2. control memory usage of mapfile indexes.

we may got 2 by control 'concurrent open readers', but it is not efficient. because mapfile index size is very different. for regionserver stable, we should make sure 'open file limit' * 'max  mapfile index size' < 'memory limit of regionserver'. this mean set the 'open file limit' low enough.

so besides 'open reader limit', there should a 'max index size limit' for concurrent mapfile reader controlling. the eviction policy need considering these 2 limitation together. 




      was (Author: ln@webcate.net):
    i think we need archiving 2 goals for the 'open reader limit':
1. fit xceiverCount limitation of hadoop, see HBASE-24
2. control memory usage of mapfile indexes.

we may got 2 by control 'concurrent open readers', but it is not efficient. because mapfile index size is very different. for regionserver stable, we should make sure 'open file limit' * 'max  mapfile index size' < 'memory limit of regionserver'. this mean we have to set the 'open file limit' low enough.

so besides 'open reader limit', there should a 'max index size limit' for concurrent mapfile reader controlling. the eviction policy should considering these 2 limitation together. 



  
> Concurrent "open mapfile reader" limit
> --------------------------------------
>
>                 Key: HBASE-823
>                 URL: https://issues.apache.org/jira/browse/HBASE-823
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.20.0
>
>
> Over in HBASE-745, Luo Ning profiling found that the number of open Readers has direct impact on memory used.  This issue is about putting an upper bound on the number of open Readers doing something like a bounded pool w/ a LRU eviction policy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.