You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "chunhui shen (JIRA)" <ji...@apache.org> on 2012/07/27 05:14:33 UTC
[jira] [Created] (HBASE-6466) Enable multi-thread for Flush
chunhui shen created HBASE-6466:
-----------------------------------
Summary: Enable multi-thread for Flush
Key: HBASE-6466
URL: https://issues.apache.org/jira/browse/HBASE-6466
Project: HBase
Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-6466.patch
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for Flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6466:
----------------------------------
Status: Patch Available (was: Open)
> Enable multi-thread for Flush
> -----------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424213#comment-13424213 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
CompactSplitThread.requestCompaction(final HRegion r, final Store s,
final String why, int priority) has this call (line 225):
{code}
pool.execute(cr);
{code}
which leads to CompactionRequest line 252:
{code}
boolean completed = r.compact(this);
{code}
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424750#comment-13424750 ]
ramkrishna.s.vasudevan commented on HBASE-6466:
-----------------------------------------------
I think i can comment only after seeing a profiled snapshots.. Because now more threads will acquire updateLock.writeLock where as previously only one was acqiuiring it.
Lets see what others have to say here.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Sergey Shelukhin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492065#comment-13492065 ]
Sergey Shelukhin commented on HBASE-6466:
-----------------------------------------
Ping?
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492469#comment-13492469 ]
Hadoop QA commented on HBASE-6466:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538303/HBASE-6466v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3250//console
This message is automatically generated.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493641#comment-13493641 ]
Hadoop QA commented on HBASE-6466:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12552728/HBASE-6466v3.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 87 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 16 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3277//console
This message is automatically generated.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.1.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424716#comment-13424716 ]
chunhui shen commented on HBASE-6466:
-------------------------------------
bq.So in startCacheFlush we will now be blocking the incoming writes more right?
As more flush threads will try getting the updatesLock.writeLock
We will blocking the incoming writes when doing HLog#startCacheFlush now, it is the same as before.
Flush threads will try getting the updatesLock.writeLock, it is also same as the before, but it's quite fast, so do you think it's a problem?
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424233#comment-13424233 ]
chunhui shen commented on HBASE-6466:
-------------------------------------
Sorry, I forgot removing lock race in HLog when making patch for trunk,
I'll update the patch later, outting now...
Sorry, it's my mistake...
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6466:
--------------------------------
Attachment: HBASE-6466v3.patch
In patch v3, removing lock race in HLog and skipping region who is flushing when flushOneForGlobalPressure
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424121#comment-13424121 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
{{cacheFlushLock}} is locked by HLog.startCacheFlush() which is called by HRegion.internalFlushcache()
In common scenario, this is in the path of region initialization.
MemStoreFlusher, on the other hand, operates on {{MemStoreFlusher.flushQueue}} which is a BlockingQueue.
So I think the patch is valid.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424127#comment-13424127 ]
Jean-Daniel Cryans commented on HBASE-6466:
-------------------------------------------
internalFlushcache is called by HRegion.flushcache which is called by MemStoreFlusher.flushRegion so it is called for every flush.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476596#comment-13476596 ]
Kannan Muthukkaruppan commented on HBASE-6466:
----------------------------------------------
+1 on the JIRA. I created HBASE-6980 only to later find out that this was already been worked on :).
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427716#comment-13427716 ]
Elliott Clark commented on HBASE-6466:
--------------------------------------
I tried the patch out on a smallish cluster(0.94 + 1.0.3) and things seemed a little weird. Every once in a while a flush would take 60s. Jstacks weren't very obvious about what was going on and I needed to use the cluster for other things, so I don't have too many more details. I hope to put the patch back up on a cluster later next week.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424412#comment-13424412 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
@J-D:
I was in a meeting yesterday.
Pardon me for not looking deep enough.
I came to the same conclusion as you did.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424699#comment-13424699 ]
chunhui shen commented on HBASE-6466:
-------------------------------------
Some performance test data for reference,
1.test environment ： random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 handler per client for writing
2.test results:
before, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver
after(2 cache flush handler), tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver
BTW, under above test, we could increase the tps if increase the client handlers.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424037#comment-13424037 ]
Jean-Daniel Cryans commented on HBASE-6466:
-------------------------------------------
How does this relate to HBASE-2832? It's a shame the patch of there was lost tho. Jon Gray might still have it somewhere.
Also if this patch doesn't modify the behavior of {{HLog.startCacheFlush}} and {{HLog.completeCacheFlush}} WRT the {{cacheFlushLock}} I can't see how it could make things any faster. In fact, it's even worse since the other flushing thread acquire a write lock just before getting the flush lock (meaning that while one thread flushes, the others are blocking the inserts and not flushing).
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424146#comment-13424146 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
MemStoreFlusher.flushRegion does the following after region.flushcache() returns:
{code}
boolean shouldSplit = region.checkSplit() != null;
if (shouldSplit) {
this.server.compactSplitThread.requestSplit(region);
} else if (shouldCompact) {
server.compactSplitThread.requestCompaction(region, Thread.currentThread().getName());
}
{code}
The above operation is region-wise. The performance boost in tps might have come from this.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424713#comment-13424713 ]
ramkrishna.s.vasudevan commented on HBASE-6466:
-----------------------------------------------
@Chunhui
bq.(meaning that while one thread flushes, the others are blocking the inserts and not flushing)
The above is one of JD's comments. Currently the patch tries to make more flushes concurrently. So in startCacheFlush we will now be blocking the incoming writes more right?
As more flush threads will try getting the updatesLock.writeLock. I may be wrong here. Pls correct me..Thanks..
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6466) Enable multi-thread for
memstore flush
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476596#comment-13476596 ]
Kannan Muthukkaruppan edited comment on HBASE-6466 at 10/16/12 12:15 AM:
-------------------------------------------------------------------------
+1 on the JIRA. I created HBASE-6980 only to later find out that this was already being worked on :).
was (Author: kannanm):
+1 on the JIRA. I created HBASE-6980 only to later find out that this was already been worked on :).
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424389#comment-13424389 ]
Jean-Daniel Cryans commented on HBASE-6466:
-------------------------------------------
bq. This doesn't explain the performance gain
So please explain to me where you're trying to get to with this pasted code... the flushes are still serialized, the compactions are done elsewhere, nothing changes as far as I can tell.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6466) Enable multi-thread for
memstore flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424213#comment-13424213 ]
Zhihong Ted Yu edited comment on HBASE-6466 at 7/27/12 11:21 PM:
-----------------------------------------------------------------
CompactSplitThread.requestCompaction(final HRegion r, final Store s,
final String why, int priority) has this call (line 225):
{code}
pool.execute(cr);
{code}
which leads to CompactionRequest line 252:
{code}
boolean completed = r.compact(this);
{code}
This doesn't explain the performance gain :-)
was (Author: zhihyu@ebaysf.com):
CompactSplitThread.requestCompaction(final HRegion r, final Store s,
final String why, int priority) has this call (line 225):
{code}
pool.execute(cr);
{code}
which leads to CompactionRequest line 252:
{code}
boolean completed = r.compact(this);
{code}
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423922#comment-13423922 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
Minor comment:
{code}
+ LOG.warn("Memstore is above high water mark and block " + took + "ms");
{code}
'is above' -> 'was above'
' block ' -> ' reclaimMemStoreMemory took '
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Sergey Shelukhin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Shelukhin updated HBASE-6466:
------------------------------------
Attachment: HBASE-6466v3.1.patch
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.1.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6466:
--------------------------------
Description:
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
Some performance test data for reference,
1.test environment ：
random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
2.test results:
one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
was:
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
Some performance test data for reference,
1.test environment ：
random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
2.test results:
one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many above global memstore blocking
two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6466:
----------------------------------
Summary: Enable multi-thread for memstore flush (was: Enable multi-thread for Flush)
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438360#comment-13438360 ]
Jimmy Xiang commented on HBASE-6466:
------------------------------------
Instead of having multiple flush threads, can we flush multiple regions each time when under memory pressure?
For flush thread, one is enough in most case. Multiple threads complicate things. If under global memory pressure,
we can figure out how many regions to flush based on a configurable ratio of memory to release.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Sergey Shelukhin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485263#comment-13485263 ]
Sergey Shelukhin commented on HBASE-6466:
-----------------------------------------
Hi. Is there consensus on the approach?
I think multiple threads make sense; as far as I see there's no bottleneck (e.g. several parallel writers to the same spindle or such).
It's as if we are starting the requisite number of flushes in async manner one after another...
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476597#comment-13476597 ]
Kannan Muthukkaruppan commented on HBASE-6466:
----------------------------------------------
Chunhui wrote: <<<I agree with one flush thread is enough in most case, however we have encountered the case that memstore always at the global limit in the online application, the small affect is writing will be blocked sometimes and the serious affect is jvm would always do CMS if your configuration is not very appropriate>>>. See description in HBASE-6980. I think for some use cases, the wins will be substantial. Also, if we do multiple commit logs per server, then too, this feature will be important to maximize throughput.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424756#comment-13424756 ]
chunhui shen commented on HBASE-6466:
-------------------------------------
updateLock is in the HRegion, and if isAboveHighWaterMark, it will block the whole regionserver
Other the way, set the flush handler as 1, so it is the same as the before
Another way, updateLock.writeLock's lock time quite rapid
On the whole, IMO, taking a easy testing is more than analysing a lot
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for Flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423668#comment-13423668 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
{code}
+ this.handlerCount = conf.getInt("hbase.regionserver.thread.flush", 1);
{code}
How about naming the new config param hbase.hstore.flusher.count ?
I would expect a default value greater than 1.
{code}
+ String threadName = "Cache Flush Handler " + this.instanceNumber;
{code}
Name the thread 'MemStore Flush Handler ' ?
{code}
+ // Wasn't able to flush any region, but we're above low water
+ // mark
{code}
There is no need to reformat comment.
{code}
+ this.cacheFlusher.setUncaughtExceptionHandler(handler);
+ this.cacheFlusher.start();
{code}
flushHandlers array is populated in start() method. Would the call to setUncaughtExceptionHandler() be effective ?
{code}
+ } catch (InterruptedException e) {
+ LOG.warn(flushHander.getName(), e);
+ }
{code}
Please restore interrupt status above.
If you can publish percentage of performance improvement, that would be great.
> Enable multi-thread for Flush
> -----------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438468#comment-13438468 ]
chunhui shen commented on HBASE-6466:
-------------------------------------
@Jimmy
Flushing multiple regions each time could only help a little, about increase 5% flush ability because we select a region to flush is very fast.
I agree with one flush thread is enough in most case, however we have encountered the case that memstore always at the global limit in the online application, the small affect is writing will be blocked sometimes and the serious affect is jvm would always do CMS if your configuration is not very appropriate
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for Flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423680#comment-13423680 ]
chunhui shen commented on HBASE-6466:
-------------------------------------
@Ted
In most cases, one flusher is enough, so I think we could keep One Flusher same with before.
In our random write testing, KeyLen=50/ValueLen=1k,under the version 0.94
Without the patch, tps is about 7.5k/s where found many following logs we added:
{quote}
2012-07-18 00:31:04,133 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and wait 1549ms
{quote}
After the patch, tps is about 10k/s
> Enable multi-thread for Flush
> -----------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6466:
--------------------------------
Description:
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
Some performance test data for reference,
1.test environment ：
random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
2.test results:
one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many above global memstore blocking
two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
was:
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many above global memstore blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492482#comment-13492482 ]
Hadoop QA commented on HBASE-6466:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538303/HBASE-6466v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3251//console
This message is automatically generated.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428821#comment-13428821 ]
ramkrishna.s.vasudevan commented on HBASE-6466:
-----------------------------------------------
@Ted
I will also try to check it out. We are not able to test this as some other things are going on.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424706#comment-13424706 ]
Hadoop QA commented on HBASE-6466:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538303/HBASE-6466v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//console
This message is automatically generated.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for Flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6466:
--------------------------------
Attachment: HBASE-6466v2.patch
Updating patch as Ted's comment.
Thanks for review again...
> Enable multi-thread for Flush
> -----------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6466:
--------------------------------
Description:
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
Some performance test data for reference,
1.test environment ：
random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
2.test results:
one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
was:
If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
So should we enable multi-thread for Memstore Flush?
Some performance test data for reference,
1.test environment ：
random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
2.test results:
one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for Flush
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6466:
--------------------------------
Attachment: HBASE-6466.patch
> Enable multi-thread for Flush
> -----------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492463#comment-13492463 ]
Andrew Purtell commented on HBASE-6466:
---------------------------------------
bq. Is there consensus on the approach?
[~sershe] Ack your ping. My read of this JIRA is that multithreading memstore flushing is seen as potentially useful, but experimental verification of improvement in trunk is lacking. See HBASE-6980 and especially the testing described in the commit message for http://svn.apache.org/viewvc?view=revision&revision=1403627.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430582#comment-13430582 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
@Elliot:
The flush taking 60s normally would take how many seconds to complete ?
Thanks
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424210#comment-13424210 ]
Jean-Daniel Cryans commented on HBASE-6466:
-------------------------------------------
bq. The above operation is region-wise. The performance boost in tps might have come from this.
Those are just queueing requests to the compact split thread, I don't think I follow you.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for Flush
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423784#comment-13423784 ]
Hadoop QA commented on HBASE-6466:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538126/HBASE-6466v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2446//console
This message is automatically generated.
> Enable multi-thread for Flush
> -----------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426327#comment-13426327 ]
Zhihong Ted Yu commented on HBASE-6466:
---------------------------------------
@Ram, @J-D:
Please share your comments.
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore
flush
Posted by "Sergey Shelukhin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493572#comment-13493572 ]
Sergey Shelukhin commented on HBASE-6466:
-----------------------------------------
Updating trunk patch. I will run some tests...
> Enable multi-thread for memstore flush
> --------------------------------------
>
> Key: HBASE-6466
> URL: https://issues.apache.org/jira/browse/HBASE-6466
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ：
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira