[jira] [Commented] (HBASE-11263) Share the open/close store file thread pool for all store in a region

["Liu Shaohui (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016463#comment-14016463 ] 

Liu Shaohui commented on HBASE-11263:
-------------------------------------

[~andrew.purtell@gmail.com]
{quote}
Do we need storeFileThreadPoolCounter?
{quote}
Each store of the region will try to get and close the thread pool when opening and closing store files. StoreFileThreadPoolCounter represent the number of stores which are using this thread pool now. When StoreFileThreadPoolCounter is zero, we can shut down the thread pool actually.

If we only use a boolean flag, we must put the code of closing the thread pool in region level. It's a little odd because the thread pool is used in the construction of store and closed in region.

{quote}
What happens if you use one thread pool for opening and closing store files per RegionServer? If we have thousands of HRegions and they all have thread pools, surely we are not able to fully schedule that degree of parallelism?
{quote}
Agree! Per regionserver thread pool for for opening and closing store files will be better.

[~stack]
{quote}
Naming the thread pool for the first region that passes by seems a bit odd:
{quote}
In the patch, the thread pool opening and closing store files is per region, so we name the thread pool with region name.


> Share the open/close store file thread pool for all store in a region
> ---------------------------------------------------------------------
>
>                 Key: HBASE-11263
>                 URL: https://issues.apache.org/jira/browse/HBASE-11263
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>    Affects Versions: 0.99.0
>            Reporter: Liu Shaohui
>            Assignee: Liu Shaohui
>            Priority: Minor
>         Attachments: HBASE-11263-trunk-v1.diff
>
>
> Currently, the open/close store file thread pool is divided equally to all stores of a region. 
> {code}
>   protected ThreadPoolExecutor getStoreFileOpenAndCloseThreadPool(
>       final String threadNamePrefix) {
>     int numStores = Math.max(1, this.htableDescriptor.getFamilies().size());
>     int maxThreads = Math.max(1,
>         conf.getInt(HConstants.HSTORE_OPEN_AND_CLOSE_THREADS_MAX,
>             HConstants.DEFAULT_HSTORE_OPEN_AND_CLOSE_THREADS_MAX)
>             / numStores);
>     return getOpenAndCloseThreadPool(maxThreads, threadNamePrefix);
>   }
> {code}
> This is not very optimal in following scenarios:
> # The data of some column families are very large and there are many hfiles in those stores, and others may be very small and in-memory column families. 
> # Usually we preserve some column families for later needs. The thread pool for these column families are wasted。
> The simple way is to share a big thread pool for all stores to open/close hfiles.  
> Suggestions are welcomed. 
> Thanks. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)